Changeset 6167 for trunk/doc/design/ippSSDD.tex
- Timestamp:
- Jan 21, 2006, 11:54:47 PM (20 years ago)
- File:
-
- 1 edited
-
trunk/doc/design/ippSSDD.tex (modified) (12 diffs)
Legend:
- Unmodified
- Added
- Removed
-
trunk/doc/design/ippSSDD.tex
r6055 r6167 1 %%% $Id: ippSSDD.tex,v 1. 6 2006-01-19 10:58:19eugene Exp $1 %%% $Id: ippSSDD.tex,v 1.7 2006-01-22 09:54:47 eugene Exp $ 2 2 \documentclass[panstarrs]{panstarrs} 3 3 … … 15 15 16 16 % allow paragraphs to be listed in TOC for now 17 \setcounter{tocdepth}{ 4}17 \setcounter{tocdepth}{3} 18 18 19 19 \begin{document} … … 777 777 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 778 778 779 \subsection{AP Database} 780 781 \tbd{this section needs to be updated with current implemetation; the 782 DVO SDD contains much of this information, but needs to be fleshed 783 out in places.} 779 \subsection{DVO : the AP Database} 784 780 785 781 \subsubsection{Corresponding Requirements} … … 787 783 The AP Database must meet the requirements specified in Section 3.4.3 788 784 of the Pan-STARRS PS-1 IPP SRS (PSDC-430-005). The specified design 789 is chosen to meet requirements 3.4.3.1 and 3.4.3.2. In order to meet 790 the throughput requirements, the AP Database will be distributed 791 across 10 Nodes independent of the Image Server Nodes. An alternative 792 organization of the database which will be studied will have the AP 793 Database co-located with the Image Server Phase 4 Nodes. 785 is chosen to meet requirements 3.4.3.1 and 3.4.3.2. The IPP is 786 modifying the Elixir program 'DVO' to perform the role of the IPP AP 787 Database. 794 788 795 789 \subsubsection{Overview} 796 790 797 The AP (Astrometry \& Photometry) Database is a CSCI which stores data 798 related to astronomical objects derived from various sources with a 799 variety of associations. The AP Database deals with two related 791 DVO, the Desktop Virtual Observatory, is a software system which 792 stores data related to astronomical objects derived from various 793 sources, and provides mechanisms to related multiple detections 794 together as astronomical objects. DVO deals with two related 800 795 concepts: {\em objects} and {\em detections}. The {\em objects} are 801 796 descriptions of astronomical objects while the {\em detections} are … … 809 804 must be accepted as they are reported. 810 805 811 The AP Database stores the collections of detections which were 812 derived from specific images from any of the analysis stages. It 813 provides a mechanism to determine the image from which a specific 814 detection was derived, and in conjunction with the Image Server locate 815 the corresponding data file. The AP Database also makes it possible 816 to extract all detections derived from a specific image and to817 determine quantities such as the pixel coordinates of the detection on 818 the image. 819 820 The AP Database also has the capability to associate multiple 821 detections of a specific object. Several major classes of objects 822 will be present, each of which must be handled correctly.823 824 First, the most distant stars, compact galaxies, and QSOs will have 825 nearly fixed locations relative to other distant stars, with only 826 small deviations for individual measurements. The association between 827 multiple detections of such objects is made on the basis of their 828 coincident positions. The AP Database determines the average position 829 of the object and the deviations of the individual detections from 830 that average on the basisof the ensemble of individual detection.831 832 Second, solar system objectsdo not have a fixed location. Detections806 DVO stores the collections of detections which were derived from 807 specific images. It provides a mechanism to determine the image from 808 which a specific detection was derived, and in conjunction with the 809 Image Server locate the corresponding data file. DVO also makes it 810 possible to extract all detections derived from a specific image and 811 to determine quantities such as the pixel coordinates of the detection 812 on the image. 813 814 DVO also has the capability to associate multiple detections of a 815 specific object. Several major classes of objects will be present, 816 each of which must be handled correctly. DVO distinguished the 817 following types of objects. 818 819 {\bf Stars, compact galaxies, and QSOs} will have nearly fixed 820 locations relative to other distant stars, with only small deviations 821 for individual measurements. The association between multiple 822 detections of such objects is made on the basis of their coincident 823 positions. DVO determines the average position of the object and the 824 deviations of the individual detections from that average on the basis 825 of the ensemble of individual detection. 826 827 {\bf Solar System Objects} do not have a fixed location. Detections 833 828 of such objects are linked by their orbits, and depend on both the 834 position and the time of the image. The AP Database does not attempt835 to make this link; this is the role of the MOPS system. However, it 836 has the ability to accept identifications made externally with 837 specifieddetections and to return the identifier of the moving object829 position and the time of the image. DVO does not attempt to make this 830 link; this is the role of the MOPS system. However, it has the 831 ability to accept identifications made externally with specified 832 detections and to return the identifier of the moving object 838 833 associated with the specific detections. These associations also 839 834 include descriptive information such as the offset of the detection 840 835 from the predicted location of the detection based on the orbit. This 841 functionality is required to allow the AP Database to ignore known 842 moving object detections from other types of queries. 843 844 Third, objects in the general vicinity of the solar system fall in 845 between these first two classes of objects. Their proper motion and 846 parallax response is significant enough ($>0.2$ arcsec in 1 year) that 847 they are not well-described by an average location and a collection of 848 offsets. These objects are described by a distance and a proper 849 motion vector. The AP Database provides the association between the 850 specific detections and an average object which includes finite 851 parallax and proper motion. 852 853 Fourth, many detections, especially in their initial states, will not 854 be associated with a specific astronomical object of any of the above 855 classes and are treated as orphans. Most of these will be spurious 856 (not representing real objects), some will be from solar system 857 objects for which orbits are not yet determined, some will be from 858 faint stars near the detection limits, and some will be from 859 short-term transients which have only been detected once. The AP 860 Database maintains these detections until they have been associated 861 with one of the objects above. The AP Database provides mechanisms by 862 which individual detections may be migrated back and forth between the 863 orphan state and association with an astronomical object. 864 865 For every object, and all orphaned detections, the AP Database also 866 provides the capability to determine the images containing the 867 location of the object but for which no detection was made. The 868 minimum set of information which must be carried for these 836 functionality is required to allow DVO to ignore known moving object 837 detections from other types of queries. 838 839 {\bf High-proper-motion objects} in the general vicinity of the solar 840 system fall in between these first two classes of objects. Their 841 proper motion and parallax response is significant enough ($>0.2$ 842 arcsec in 1 year) that they are not well-described by an average 843 location and a collection of offsets. These objects are better 844 described by a distance and a proper motion vector. DVO provides the 845 association between the specific detections and an average object 846 which includes finite parallax and proper motion. 847 848 {\bf Orphaned detections} are not associated with a specific 849 astronomical object of any of the above classes. Most of these will 850 be spurious (not representing real objects), some will be from solar 851 system objects for which orbits are not yet determined, some will be 852 from faint stars near the detection limits, and some will be from 853 short-term transients which have only been detected once. DVO 854 maintains these detections until they have been associated with one of 855 the objects above. DVO provides mechanisms by which individual 856 detections may be migrated back and forth between the orphan state and 857 association with an astronomical object. 858 859 DVO stores the information about the detection, the related objects, 860 and the images which provided the measurements. For every detection, 861 DVO provides the mechanisms to link the detection back to the image 862 which supplied it. DVO also provides the capability to determine the 863 images containing a specific location but for which no detection was 864 made. The minimum set of information which must be carried for these 869 865 non-detections is the image and the associated object or orphan. 870 866 871 The AP Databasealso stores the relationships between various867 DVO also stores the relationships between various 872 868 photometric systems and the evolution of that relationship. It 873 869 provides mechanisms to convert between the measured instrumental … … 880 876 various reference systems appropriate for those filters. 881 877 878 \subsubsection{Photometric systems and the DVO Photcodes} 879 880 One of the major roles of DVO is to relate different photometric 881 measurements made with different instruments and detectors together. 882 We may have observations made with the same basic filters, but using a 883 number of different detectors. We may have observations from 884 different telescopes in similar filters. We may have reference data 885 related to some filter, but obtained and published by other observers. 886 We would like to related these measurements together in optimal ways, 887 making use of whatever information we have available. DVO provides 888 several mechanisms to enable these relationships. 889 890 We identify three distinct types of photometry measurements within 891 DVO: 892 \begin{itemize} 893 \item {\bf reference photometry} These measurements are provided by 894 external observers. For reference photometry, we do not have access 895 to very must information used to determine the magnitudes of the 896 objects of interest. We have the reference magnitudes corresponding 897 to a type of filter, and presumably some information of the error on 898 the measurement. We might possibly know the epoch of the 899 observations, but not necessarily. 900 \item {\bf detection photometry} This is our primary measurement of 901 interest: the photometry of objects measured from images which we 902 have processed. More specifically, the detection photometry is an 903 instantaneous measurement from a specific image with well-known 904 properties, such as exposure time, airmass, instrument source, etc. 905 \item {\bf internal photometry} With the application of an appropriate 906 zero point and other calibration terms, any detection photometry can 907 be calibrated to represent a measurement in a well-known photometric 908 system. The internal photometry measurements are calibrated to be 909 on a photometric system which represents a consistent system for a 910 particular telescope or collection of data, minimizing the 911 calibration transformations necsessary. 912 \end{itemize} 913 914 Defining the relationships between the different types of measurements 915 is part of the process of photometric calibration. DVO uses the 916 concept of the 'photcode' to identify the source of the photometry, 917 and to define the relationships between different photometry sources. 918 A photcode identifies a photometric system: for the detection 919 photometry measurments, each combination of telescope, camera, filter, 920 and detector is associated with a unique photcode; there are also 921 unique photcodes for the internal photometry systems and any distinct 922 external reference source. 923 924 As a concrete example, consider the Pan-STARRS PS-1 system. There 925 will be three different cameras in use at different times: GPC-1, 926 TC-3, and the SkyProbe camera. There are at least 6 filter systems: 927 {\it grizy} and {\it w}. The SkyProbe camera has a single CCD, TC-3 928 has 16 different detectors, and GPC-1 has up to 64 different devices. 929 Each of these combinations is potentially a different photometric 930 system, so a different photcode is defined for each combination. 931 These photcodes would have names such as: GPC1.02.r (r filter with the 932 GPC1 camera and OTA 02) or SP1.00.g (SkyProbe 1, g filter). These 933 ($64 \times 6 + 16 \times 6 + 5 = 485$) photcodes are all identified 934 as 'detection' photcodes, specifying that detection photometry is 935 associated with them 936 937 There are also 6 different internal photometric systems of interest, 938 namely those associated with the 6 named filters, {\it grizy} and {\it 939 w}. Each of these 6 systems is identified with an internal photcode. 940 The internal photcodes are further distinguished as 'primary' or 941 'secondary', which specifies how the DVO system stores average 942 quantities related to these types of photcodes (see the discussion of 943 the tables below). 944 945 Finally, there may be multiple external photometric systems of 946 interest, some of which are related to the major internal photometry 947 systems, some of which are not. For example, the Pan-STARRS project 948 may refer to photometry from the SDSS secondary standards, the SDSS 949 data releases, Johnson photometry from Landolt (1992), observations 950 from 2MASS in $JHK$, USNO-B observations, and so forth. Each of these 951 photometric systems is assoiciated with a different photcode; only 952 some of these are relevant to the detection or internal photometry 953 system. 954 955 Within DVO, the detection and internal photcodes each define a 956 relationships as well as a specific photometric system. Associated 957 with each of these photcodes are the parameters of the photometry 958 transformation from the photometric system of the photcode to another 959 photometric system. For the detection photcodes, the parameters 960 define the transformation to the equivalent internal photcode system. 961 The currently-defined transformation parameters consist of the 962 following photometry equation: 963 % 964 \[ 965 M_\lambda = m_\lambda + C_\lambda + K_\lambda (\mbox{airmass} - 1) + \sum_{i = 1}^{i < N} 966 A_{\lambda,i} (\mbox{color}_\lambda - \mbox{color}_{o,\lambda})^i 967 \] 968 % 969 where $C_r$ represents the zero-point of the transformation, $K_r$ 970 represents the slope of the airmass trend, $\mbox{airmass}$ is the 971 airmass for a given measurement, $\mbox{color}$ is the color of the 972 source of interest (as identified below), $\mbox{color}_r$ is the 973 reference color for sources in this photometry system, and $A_{r,i}$ 974 is the coefficient of the $i$ power of the color difference. Up to 975 fourth order color terms are currently allowed. For any photcode, the 976 color is defined as the difference of the measurements in two other 977 photcodes, usually two 'internal' photcodes. The photcode information 978 also specified the equivalent photcode to which the transformation corresponds. 979 980 For the detection photcodes, the target of the transformation must be 981 an internal photcode. For the internal photcodes, the target of the 982 transformation is an external reference photcode system. This 983 restriction implies that the internal photometry may only be 984 transformed (and thus compared with) a single external reference. 985 This is in fact the best practice as far as photometric calibration is 986 concerned: the 'standard' observations from different references 987 should always be treated as different photometric systems. To allow 988 for the relationship of the internal photometry to multiple sources of 989 reference photometry, an additional set of photcodes are defined which 990 identify 'alternative' transformations for the internal photcodes. 991 992 It is important to note that not all of the photometry transformation 993 parameters identified above are relevant for each of the three major 994 types of photcode. The detection photcodes will in general make use 995 of all of these elements, though the order of the color transformation 996 will hopefully be limited if the different devices are sufficiently 997 similar. For the transformation from the internal photcodes, which 998 are derivative in some way of the detection photcodes, the airmass 999 component is invalid: for a single measurement, the 1000 detection-to-internal transformation has already removed the airmass 1001 trend; for an averaged internal photometric measurement, no single 1002 airmass corresponds to the observations. Finally, no transformation 1003 parameters are defined for the reference photcodes at this time. 1004 1005 DVO provides methods by which these photometry transforamtions are 1006 automatically applied. The specific measurements (detection 1007 photometry) are stored in the database tables as instrumental 1008 magnitudes, and any operation which examines these measurements must 1009 make use of the APIs to convert to an appropriate common system. A 1010 further complication to note is that the photcodes defined above are 1011 static; they do not include any information about changes to the 1012 system sensitivity. This information is carried externally to the 1013 photcode calibration information; the transformations defined by the 1014 photcodes must be considered the {\em starting point} for any 1015 photometric analysis. An additional adjusment can be applied. 1016 1017 The detections from a specific image may all have a 'calibration' 1018 offset applied which bring the measured photometry into a common 1019 relative system. This calibration offset is associated with the image 1020 and may be a function of position on the detector. The tables which 1021 carry the individual measurements also include the calibration 1022 magnitude appropriate for each measurement to speed up the application 1023 of this offset. In a well-calibrated collection of photometry, all of 1024 the detection measurements will have a measured calibration magnitude, 1025 yielding a collection of internal photometry measurements which are 1026 all consistent. An additional piece of information is the zero-point 1027 history, which tracks the system-wide variations in the average 1028 sensitivity. The zero-point history can be used to predict the 1029 calibration magnitudes for any observation which is not tied directly 1030 via relative photometry to the rest of the photometric observations. 1031 1032 Putting all of these pieces together, the photometry APIs in DVO can 1033 be used to return any of the following types of photometric 1034 measurements: 1035 \begin{itemize} 1036 \item raw instrumental magnitudes for any detection 1037 1038 \item 'catalog' magnitudes, applying only the airmass and static 1039 zero-point calibrations to a detection magnitude; this is useful to 1040 test the detector-color transformation. 1041 1042 \item 'system' measurements, applying the complete static 1043 transformation for a detection magnitude to the internal photometry 1044 system; for photometric weather and no zero-point variations, this 1045 would be a measurement in the internal photometry system. 1046 1047 \item 'relative' magnitudes, applying the measured calibration offset 1048 to the calibrated detection magnitude determined above; in a 1049 well-calibrated system, this represents a consistent internal 1050 photometry measurement. 1051 1052 \item 'calibrated' magnitudes, correcting the measure detection 1053 photometry by applying the transformation from the internal 1054 magnitude system to the external reference magntiude system. 1055 1056 \item 'average' magntiudes, the raw internal photometry magnitudes 1057 (note the distinction between the 'average' quantities, which are 1058 derived from a collection of detections an the 'relative' quantities 1059 which represent an instantenous measurement in the same system). 1060 1061 \item 'reference' magnitudes, in which the 'average' internal 1062 photometry values are transformed to the refernce magnitude system. 1063 \end{itemize} 1064 The complexity of these transformations is necessary to allow the 1065 examination of the trends of actual measurements with external 1066 parameters. 1067 1068 \subsubsection{DVO Database Tables} 1069 882 1070 \begin{figure} 883 1071 \begin{center} 884 \resizebox{4.5in}{!}{\includegraphics{pics/APDB}} 885 \caption{AP DB components} 886 \label{fig:APDBComponents} 1072 \resizebox{4.5in}{!}{\includegraphics{pics/dvo.01.ps}} 1073 \caption{\label{fig:DVOtables} \small Data types managed by DVO} 887 1074 \end{center} 888 1075 \end{figure} 889 1076 890 The AP Database provides interfaces to extract lists of objects and 891 detections based on various query parameters. It provides the 892 capability to extract all detections associated with a specific 893 object, all non-detections of that object, all non-detections of an 894 orphan, and summary statistics from these collections. It will also 895 return all objects or detections within specified spatial regions 896 including regions bounded by great circles (RA,DEC; GLAT,GLON; 897 ELAT,ELON) and regions described by a location and a search radius. 898 It will also return the image parameters associated with a specific 899 detection including image coordinates of the detection, exposure time, 900 time and date of the detection, etc. 901 902 As shown in Figure~\ref{fig:APDBComponents}, the IPP AP Database 903 consists of the following components: 904 905 \begin{itemize} 906 \item AP Database database tables 907 \item AP Database database engine 908 \item AP Database servers 909 \item AP Database client APIs 910 \end{itemize} 911 912 \subsubsection{AP Database Tables} 913 914 Table~\ref{tab:APDBTables} lists the tables used by the AP Database. The 915 contents of these tables are outlined in 916 Appendix~\ref{sec:APDBTableContents}. Below, the use of these tables by 917 the AP Database software is discussed below. Several of the tables 918 are not just simple tables in the database but are instead table 919 groups divided into many subtables, each of which represents a portion 920 of the sky (a {\tt region}). These subtables may also be distributed 921 across different computers to distribute the processing load. 922 923 \paragraph{Images Table Group} 924 925 The {\tt Images} table group lists all of the images which provided 926 the data in the AP Database. These tables are subdivided by region on 927 the sky. In general, the images listed in this table correspond to 928 the Chips. This group of tables includes sufficient astrometric 929 parameters to represent the coordinates of the detections to a 930 sufficient accuracy. Parallel to the Images table is the Mosaic 931 table. This table is very similar to the Images table, but defines 932 the Mosaic which corresponds to a group of Images. The parameters 933 include the astrometric information needed to define the camera 934 distortion. 935 936 \paragraph{Image Overlaps Table Group} 937 938 The specific subtable of {\tt Images} which contains a given image is 939 the one which contains the center pixel of that image. An additional 940 table group, {\tt Image Overlaps} (with the same subtable organization 941 as the {\tt Images} subtables), lists images which overlap that 942 specific subtable. Thus, given a particular coordinate, in order to 943 find that images which overlap that coordinate, it is necessary to 944 search the images in the {\tt Images} subtable which includes that 945 coordinate, and all images in the {\tt ImageOverlaps} subtable for 946 that coordinate. 947 948 \begin{table}[hb] 949 \begin{center} 950 \caption{AP Database Tables\label{tab:APDBTables}} 951 \begin{tabular}{ll} 952 \hline 953 \hline 954 {\bf Table Name} & {\bf Description} \\ 955 \hline 956 Images & The images that have objects in the DB. \\ 957 Image Overlaps & Image regions which are touched by specific images. \\ 958 Objects & The objects --- average properties of multiple detections of the same object. \\ 959 Average Magnitudes & Average photometry in multiple filters \\ 960 Solar System Objects & Identification of solar system objects \\ 961 Matched Detections & Detections of sources in an image identified with an Object. \\ 962 Orphaned Detections & Detections of sources in an image not identified with an Object. \\ 963 Non-detections & Non-detections of objects in an image. \\ 964 Regions & spatial distribution of tables \\ 965 Filters & Filters understood by the system. \\ 966 Photcodes & Transformations between different photometric systems \\ 967 Zero Points & History of Zero-point \& Airmass terms \\ 968 Distortion Models & History of Optical Distortion terms \\ 969 Database Hosts & computers used to store the tables \\ 970 \hline 971 \end{tabular} 972 \end{center} 973 \end{table} 974 975 \paragraph{Objects Table Group} 976 977 The {\tt Objects} table group (also divided by region) stores the 978 average parameters for each astronomical object. Certain details of 979 this table have not yet been specified. In particular, objects with 980 significant parallax and/or proper motion may potentially be stored in 981 a distinct table. Solar system object identifications, to the extent 982 average properties are maintained in the AP Database, will certainly 983 be stored in a separate table. 984 985 \paragraph{Average Magnitudes Table Group} 986 987 A related table, also divided into the same regions, is the {\tt 988 Average Magnitudes} table. In this table, there are multiple rows per 989 object, one for each of the primary filters of interest for which 990 photometric averaging is performed. This organization makes the 991 number of primary (averaged) filters a configurable value. 992 993 \paragraph{Matched Detections Table Group} 994 995 The {\tt Matched Detections} table stores all of the measurements of 996 astronomical objects on specific images. This table includes all 997 detections associated with the average {\tt Objects}. As discussed 998 below, bright objects (above a configuration-specified signal-to-noise 999 level) are defined object even if only one detection has been found at 1000 that position. Faint orphaned objects are not added to this list or 1001 the list of objects. The different types of detections (P2, 1002 P4$\Delta$, P4$\Sigma$) are distinguished by their photometry codes. 1003 (This is only valid if the AP Database does not store different 1004 quantities for these types of detections.) 1005 1006 \paragraph{Orphaned Detections Table Group} 1007 1008 The {\tt Orphaned Detections} table stores the detections which have 1009 not been correlated with an existing object. This table is only 1010 populated for objects below a configuration-specified signal-to-noise 1011 limit (e.g., 5$\sigma$). Bright orphaned detections are assigned an 1012 object and added to the {\tt Matched Detections} table. 1013 1014 \paragraph{Non-detections Table Group} 1015 1016 The {\tt Non-detections} table stores information about detection 1017 failures for each object. If an image is added to the database which 1018 overlaps an object but the object is not detected, an entry is made in 1019 this table. In practice, this table may store only the most recent 1020 non-detection and the total number, or a similar reduced set of 1021 non-detection statistics. 1022 1023 \paragraph{Regions Table} 1077 Figure~\ref{fig:DVOtables} illustrates the data managed by DVO, and 1078 Table~\ref{tab:DVOtables} provides a complete listing. The contents 1079 of these tables are outlined in Appendix~\ref{sec:DVOTableContents}. 1080 Below, the use of these tables by DVO software is discussed below. 1081 Several of the tables are not just simple tables in the database but 1082 are instead table groups divided into many subtables, each of which 1083 represents a portion of the sky (a {\tt region}). These subtables may 1084 also be distributed across different computers to distribute the 1085 processing load. 1086 1087 \paragraph{Sky Regions Table} 1024 1088 1025 1089 The {\tt Regions} table is used to subdivide the tables of images, 1026 objects, and detections, etc, as discussed above. The AP Database1090 objects, and detections, etc, as discussed above. DVO 1027 1091 divides the sky into a hierarchy of regions (portions of the sky) each 1028 1092 of which is in turn subdivided into smaller portions. Since nearly 1029 all interactions with the AP Databaseperformed by the IPP are limited1093 all interactions with DVO performed by the IPP are limited 1030 1094 in spatial coverage, subdividing the tables allows a specific 1031 1095 interaction to search only a small subset of the data. The table of … … 1049 1113 \begin{figure} 1050 1114 \begin{center} 1051 \resizebox{6in}{!}{\includegraphics{pics/ APDBRegions}}1052 \caption{ AP DBRegions and Image / Object tables}1053 \label{fig: APDBRegions}1115 \resizebox{6in}{!}{\includegraphics{pics/dvo.02.ps}} 1116 \caption{DVO Regions and Image / Object tables} 1117 \label{fig:DVOskyregions} 1054 1118 \end{center} 1055 1119 \end{figure} 1120 1121 \paragraph{Images Table Group} 1122 1123 The {\tt Images} table group lists all of the images which provided 1124 the data in DVO. These tables are subdivided by region on 1125 the sky. In general, the images listed in this table correspond to 1126 the Chips. This group of tables includes sufficient astrometric 1127 parameters to represent the coordinates of the detections to a 1128 sufficient accuracy. Parallel to the Images table is the Mosaic 1129 table. This table is very similar to the Images table, but defines 1130 the Mosaic which corresponds to a group of Images. The parameters 1131 include the astrometric information needed to define the camera 1132 distortion. 1133 1134 \paragraph{Image Overlaps Table Group} 1135 1136 The specific subtable of {\tt Images} which contains a given image is 1137 the one which contains the center pixel of that image. An additional 1138 table group, {\tt Image Overlaps} (with the same subtable organization 1139 as the {\tt Images} subtables), lists images which overlap that 1140 specific subtable. Thus, given a particular coordinate, in order to 1141 find that images which overlap that coordinate, it is necessary to 1142 search the images in the {\tt Images} subtable which includes that 1143 coordinate, and all images in the {\tt ImageOverlaps} subtable for 1144 that coordinate. 1145 1146 \begin{table}[hb] 1147 \begin{center} 1148 \caption{DVO Database Tables\label{tab:DVOtables}} 1149 \begin{tabular}{ll} 1150 \hline 1151 \hline 1152 {\bf Table Name} & {\bf Description} \\ 1153 \hline 1154 Images & The images that have objects in the DB. \\ 1155 Image Overlaps & Image regions which are touched by specific images. \\ 1156 Objects & The objects --- average properties of multiple detections of the same object. \\ 1157 Average Magnitudes & Average photometry in multiple filters \\ 1158 Solar System Objects & Identification of solar system objects \\ 1159 Matched Detections & Detections of sources in an image identified with an Object. \\ 1160 Orphaned Detections & Detections of sources in an image not identified with an Object. \\ 1161 Non-detections & Non-detections of objects in an image. \\ 1162 SkyRegions & spatial distribution of tables \\ 1163 Filters & Filters understood by the system. \\ 1164 Photcodes & Transformations between different photometric systems \\ 1165 Zero Points & History of Zero-point \& Airmass terms \\ 1166 Distortion Models & History of Optical Distortion terms \\ 1167 Database Hosts & computers used to store the tables \\ 1168 \hline 1169 \end{tabular} 1170 \end{center} 1171 \end{table} 1172 1173 \subsection{Objects Table Group} 1174 1175 \begin{table} 1176 \begin{center} 1177 \caption{DBO Detection Classes \& Object Parameters\label{tab:APdetections}} 1178 \begin{tabular}{lrrrr} 1179 \hline 1180 \hline 1181 Object Parameter & P2 & P4S & P4D & SS \\ 1182 \hline 1183 PSF x,y, covar, $\alpha,\delta$ & + & + & + & + \\ 1184 PSF mag, $\sigma_{\rm mag}$ & + & + & + & + \\ 1185 star/gal sep & + & + & + & + \\ 1186 $\sigma_x$, $\sigma_y$, $\theta$ & + & + & + & + \\ 1187 local sky data & + & + & + & + \\ 1188 Petrosian R, M, $R_{50}$, $R_{90}$ & - & + & - & + \\ 1189 S\'ersic R, M, AB, $\phi$, $\nu$ & - & + & - & + \\ 1190 W.L. $\gamma_1$, $\gamma_2$, pol. terms & - & - & - & + \\ 1191 exp. spaced aps., Poisson noise, variance & - & - & - & + \\ 1192 \hline 1193 \end{tabular} 1194 \end{center} 1195 \end{table} 1196 1197 The {\tt Objects} table group (also divided by region) stores the 1198 average parameters for each astronomical object. Certain details of 1199 this table have not yet been specified. In particular, objects with 1200 significant parallax and/or proper motion may potentially be stored in 1201 a distinct table. Solar system object identifications, to the extent 1202 average properties are maintained in DVO, will certainly 1203 be stored in a separate table. 1204 1205 \paragraph{Average Magnitudes Table Group} 1206 1207 A related table, also divided into the same regions, is the {\tt 1208 Average Magnitudes} table. In this table, there are multiple rows per 1209 object, one for each of the primary filters of interest for which 1210 photometric averaging is performed. This organization makes the 1211 number of primary (averaged) filters a configurable value. 1212 1213 \paragraph{Matched Detections Table Group} 1214 1215 The {\tt Matched Detections} table stores all of the measurements of 1216 astronomical objects on specific images. This table includes all 1217 detections associated with the average {\tt Objects}. As discussed 1218 below, bright objects (above a configuration-specified signal-to-noise 1219 level) are defined object even if only one detection has been found at 1220 that position. Faint orphaned objects are not added to this list or 1221 the list of objects. The different types of detections (P2, 1222 P4$\Delta$, P4$\Sigma$) are distinguished by their photometry codes. 1223 (This is only valid if DVO does not store different 1224 quantities for these types of detections.) 1225 1226 \paragraph{Orphaned Detections Table Group} 1227 1228 The {\tt Orphaned Detections} table stores the detections which have 1229 not been correlated with an existing object. This table is only 1230 populated for objects below a configuration-specified signal-to-noise 1231 limit (e.g., 5$\sigma$). Bright orphaned detections are assigned an 1232 object and added to the {\tt Matched Detections} table. 1233 1234 \paragraph{Non-detections Table Group} 1235 1236 The {\tt Non-detections} table stores information about detection 1237 failures for each object. If an image is added to the database which 1238 overlaps an object but the object is not detected, an entry is made in 1239 this table. In practice, this table may store only the most recent 1240 non-detection and the total number, or a similar reduced set of 1241 non-detection statistics. 1056 1242 1057 1243 \paragraph{Other Reference Tables} … … 1062 1248 photometry system may consist of a detector, telescope, and specific 1063 1249 filter, or it may be a derived photometry system. The {\tt Database 1064 Machines} table identifies all of the computers available to the AP 1065 Database. 1066 1067 \subsubsection{AP Database servers} 1068 1069 The AP Database functions on a group of computers, with portions of 1070 the tables stored on separate machines, as described above. The 1071 association between a machine and the corresponding table or part of 1072 the sky is defined by the Region table. Each machine has a 1073 corresponding AP Database server which runs on that machine to 1074 interact with the tables available on that machine. Two possible 1075 interaction models are considered. 1076 1077 {\bf Option A:} A client chooses one of the machines and sends its 1078 query or data to that machine. The server then uses the region table 1079 to determine which machines contain the relevant portion of the sky. 1080 Data to be added to the database is divided into corresponding region 1081 chunks and sent to the appropriate servers. Queries are redirected to 1082 the appropriate server(s). The original server may collect the 1083 results and return them to the original client. 1084 1085 {\bf Option B:} The client downloads the region table and performs the 1086 division of the data into appropriate subsets. The subsets are then 1087 sent to the corresponding servers by the client. 1088 1089 The differences between these models is small. The first option may 1090 make the code more testable, placing all of the logic in the servers 1091 and making each server symmetric. The smaller tables (ie, Region, 1092 Filters, etc) could either be downloaded from a single server or 1093 replicated to all AP DB servers. For these reasons, Option A will be 1094 used for the PS-1 IPP. \tbd{update this in light of the addstar 1095 client / server implementation} 1096 1097 \subsubsection{AP Database engine} 1098 1099 The backend database engine for the AP Database stores the tables and 1100 provides them to the servers on demand. The AP Database will use a 1101 \code{mysql} database engine for this function. 1102 1103 \subsubsection{AP DB Client operations} 1104 1105 The AP Database client interactions consist of a collection of basic 1106 queries of the database, along with more complex operations to perform 1107 particular tasks. The complex operations are listed below. 1108 1109 \paragraph{Insert Image \& Detection Set (addstar)} 1110 1111 One of the most basic operations needed by the AP Database is to 1112 insert a collection of detections derived from a specific image, and 1113 add the definition of that image to the database. This operation is 1114 critical in terms of the processing throughput. After the detections 1115 have been assigned to the appropriate regions, they are matched 1116 against all objects in the {\tt Objects} table. Matches are performed 1117 only on the basis of positional coincidence, using a matching radius 1118 which may depend on the image astrometry errors, or may be a fixed 1119 distance. Any matched detections are added to the {\tt Matched 1120 Detections} table. Any unmatched detections brighter than the Faint 1121 Detection cut-off are specified as a new {\tt Object} and also added 1122 to the {\tt Matched Detections} table. Any faint unmatched detections 1123 are added to the {\tt Orphaned Detections} table. This division is 1124 important because it allows the automatic association of new 1125 detections with existing bright objects while limiting the I/O volume 1126 required to make the detections. In general, there will be many fewer 1127 {\tt Objects} than {\tt Detections}, and there will be fewer bright 1128 orphans than faint orphans. 1129 1130 \paragraph{Insert Reference Objects (addrefs)} 1131 1132 This operation is very similar to the previous one. A collection of 1133 reference objects are added to the database as a collection of 1134 detections. The reference photometry should in general be given its 1135 own photometry code. The reference data is different from the image 1136 detection set because the associated image information is not 1137 included. Thus, no corresponding images are added to the database. 1138 1139 \paragraph{Determine Relative Photometry in region (relphot)} 1250 Machines} table identifies all of the computers available to DVO. 1251 1252 \subsubsection{Database Table I/O} 1253 1254 \begin{figure} 1255 \begin{center} 1256 \resizebox{4.5in}{!}{\includegraphics{pics/dvo.03.ps}} 1257 \caption{\label{fig:DVOformats} \small DVO Table I/O } 1258 \end{center} 1259 \end{figure} 1260 1261 DVO allows for a flexible representation of its data on disk. Data 1262 may be written to disk in one four possible mode: RAW, FITS MEF, FITS 1263 SPLIT, and MYSQL. These modes define the overall organization of the 1264 data on disk. In the RAW mode, the data is written to disk in a 1265 pseudo-FITS table format which consists of a simple FITS header 1266 describing the layout followed by the binary data in a block. This 1267 storage mode is maintained for historical reasons. There are also two 1268 types of FITS modes in which the data tables are written as valid FITS 1269 Binary Tables. In the SPLIT format, every data table is written as a 1270 separate file, while in the MEF format, the object and detection 1271 tables are bundled together into a single FITS file with multiple 1272 table extensions. The MEF format has the advantage of minimize the 1273 proliferation of files, while the SPLIT format is required to make use 1274 of the fastest read/write capabilities of DVO. DVO makes use of these 1275 raw data formats as a throughput risk mitigation strategy. As 1276 discussed below, this strategy has proven very successful. 1277 1278 There are also multiple formats in which the data may be stored. The 1279 different formats define which specific database table columns are 1280 stored and with what numerical format and precision. 1281 Figure~\ref{fig:DVOformat} illustrates the conversion process which 1282 DVO performs when loading in the data. When DVO loads data from a 1283 file-based table (FITS or RAW), it first loads from the disk file into 1284 a data structure representing the external format in use. The 1285 external structure is then converted into the internal format. The 1286 internal structure is always specified to be the superset of all 1287 external data formats. This capability allows DVO to maintain 1288 backwards compatibility with data tables written with early versions. 1289 As DVO is extended and new elements are added to the tables, it is 1290 only necessary to define the methods to convert the new internal table 1291 into the external table. In addition, DVO makes use of autocoded 1292 table manipulation and I/O APIs which are generated for each data 1293 structure based on a descriptive table. This makes it easy to add new 1294 data types and input/output methods without significant re-coding. 1295 1296 \tbd{DVO mysql table storage is not yet implemented} 1297 1298 \subsubsection{addstar : Insert Image \& Detection Set} 1299 1300 \begin{figure} 1301 \begin{center} 1302 \resizebox{4.5in}{!}{\includegraphics{pics/dvo.04.ps}} 1303 \caption{\label{catalog} \small a figure } 1304 \end{center} 1305 \end{figure} 1306 1307 One of the most basic operations needed by DVO is to insert a 1308 collection of detections derived from a specific image, and add the 1309 definition of that image to the database. This operation is critical 1310 in terms of the processing throughput. After the detections have been 1311 assigned to the appropriate regions, they are matched against all 1312 objects in the {\tt Objects} table. Matches are performed only on the 1313 basis of positional coincidence, using a matching radius which may 1314 depend on the image astrometry errors, or may be a fixed distance. 1315 Any matched detections are added to the {\tt Matched Detections} 1316 table. Any unmatched detections brighter than the Faint Detection 1317 cut-off are specified as a new {\tt Object} and also added to the {\tt 1318 Matched Detections} table. Any faint unmatched detections are added 1319 to the {\tt Orphaned Detections} table. This division is important 1320 because it allows the automatic association of new detections with 1321 existing bright objects while limiting the I/O volume required to make 1322 the detections. In general, there will be many fewer {\tt Objects} 1323 than {\tt Detections}, and there will be fewer bright orphans than 1324 faint orphans. 1325 1326 A wide range of options are available to addstar. These can be used 1327 to modify the object matching rules, to reduce the number of tables 1328 which are updated, to specify the output data format, and so forth. A 1329 few options modify the behavoir in substantial ways, as discussed in 1330 the two sections below. 1331 1332 \tbd{flesh out discussion of the options} 1333 1334 \paragraph{Insert Reference Objects} 1335 1336 \code{addstar -ref (filename)} 1337 1338 This mode of addstar reads a text file and adds the listed objects to 1339 the database as a reference photcode type. A collection of reference 1340 objects are added to the database as a collection of detections. The 1341 reference photometry should in general be given its own photometry 1342 code. The reference data is different from the image detection set 1343 because the associated image information is not included. Thus, no 1344 corresponding images are added to the database. 1345 1346 \paragraph{Insert Catalog Objects} 1347 1348 \code{addstar -cat (name) -region ra ra dec dec} 1349 1350 In this mode, any of several all-sky or large-scale reference catalogs 1351 are used for the input sources. The catalog objects are added to the 1352 database as reference objects. The valid catalogs consist of 2MASS, 1353 USNO, GSC. Tycho and USNO-B will be added shortly. Specific 1354 photcode names are defined for each of these catalogs, and must be 1355 appropriately requested and defined in the photcode table. The 1356 optional region restriction limits the insert to a subset of the sky. 1357 The user does not always want to add 50GB of 2MASS detections to any 1358 DVO database... 1359 1360 \paragraph{Addstar Client/Server Interactions} 1361 1362 DVO currently uses stand-alone programs which are run from the command 1363 line (like addstar, or the programs listed below), or it works with 1364 the interactive DVO shell, which allows the user to query portions of 1365 the database. These programs all interact with the database tables 1366 directly, making use of file locking to prevent conflicts. 1367 1368 Unlike the other DVO programs (currently), it is possible to run 1369 addstar as a client/server system. In this configuration, the program 1370 \code{addstard} is launched to run in the background as a server. It 1371 monitors a socket waiting for clients to contact it. The client 1372 program, \code{addstarc} appears to the user identical to the 1373 stand-alone addstar. However, rather than directly insert data into 1374 the database, \code{addstarc} contacts the addstar server and sends it 1375 the detections and associated image data (along with the information 1376 about the user options). The daemons accepts the incoming data and 1377 then loads this data into the database, just as the stand-alone 1378 addstar does. 1379 1380 The purpose of the addstar client/server design is three-fold. First, 1381 the client can be used by processes to send data to the DVO database 1382 and then immediately exit. The addstar loading process is one of the 1383 more time-critical functions within the IPP. However, unlike the 1384 other portions of the IPP, the addstar processes must operate in 1385 serial, at least when they are updating the same portion of the sky 1386 (or the image table). If the IPP analysis routines all needed to run 1387 the stand-alone addstar program, they would eventually block waiting 1388 for each addstar to complete, preventing other processing from 1389 continuing. The addstar client / server model allows the processing 1390 node to invoke the addstar client, sending the data to the addstar 1391 server. The addstar server will then be the entity that manages the 1392 serialization of the incoming data stream. The addstar server has two 1393 threads which run in parallel. One thread monitors the socket and 1394 accepts new data sets from addstar clients, adding the data to an 1395 internal queue. The other thread pulls data off of the queue and 1396 updates the database with the data. 1397 1398 A second advantage of the client/server interaction is that only the 1399 new detections need to be sent across the network. To update the 1400 database, addstar must load the average objects for the region from 1401 the database tables. In the stand-alone mode, the addstar program 1402 loads this data via NFS across the network from whatever device stores 1403 the addstar tables. In the client/server model, the addstar server 1404 always runs locally on the machine which holds the database tables. 1405 Thus, for the server, all database access is local disk access. 1406 1407 The final advantage of the client/server model is that it enables the 1408 parallel database model, which is not yet implemented as of Jan 1409 2006. In this model, there are multiple addstar servers. Each one has 1410 a fraction of the sky in the local tables. The identification of 1411 which table is managed by this host/addstar server is stored in the 1412 SkyRegion table. The addstar server simply accepts incoming 1413 detections from the addstar clients. Any detections which it receives 1414 which fall within the boundaries of tables that it manages are updated 1415 as normal. The server then identifies the other addstar servers which 1416 are responsible for the other detections. It then sends these 1417 detections to those servers using the same socket communication used 1418 by the addstar clients. The addstar server must also be ready accept 1419 detections from other addstar servers. This relationship is 1420 completely parallel, and any addstar client may send its data to any 1421 addstar server, letting the servers hash out who owns what. The only 1422 difficulty with this model is in handling sources near the boundaries 1423 of the tables. Note that this issues exists whether those tables are 1424 distributed across multiple machines or not. 1425 1426 Addstar uses the following strategy to handle detections on the table 1427 boundaries. Detections are first added to each table completely 1428 ignoring the neighboring tables. A detection which is close to the 1429 boundary may either be associated with an average object contained 1430 within the table, or not. If it is, the detection is associated with 1431 that average object. If not, a new average object is created at the 1432 location of the detection. So far, this process is identical to the 1433 behavoir in the middle of the table. One a longer time-scale, a 1434 process is run which mediates the table boundaries. In this analysis, 1435 the two neighboring tables are simultaneously examined. The border 1436 region, in a strip wider than the correlation radius, is examined in 1437 detail. If two objects within the border region fall within 2x the 1438 correlation radius of each other, their individual detections are 1439 re-examined. These detections are re-added to a temporary table which 1440 encompases the overlap. the resulting objects will in general have 1441 detections from either side of the boundary. The average objects are 1442 kept within the table as normal, but the detections are allowed to 1443 migrate between the tables to stay with their object. \tbd{this 1444 boundary cleanup process is not implemented to date}. 1445 1446 \subsubsection{Relphot : Relative Photometry Analysis} 1140 1447 1141 1448 This operation uses the overlaps of images and multiple observations 1142 1449 of the same objects to determine the relative photometry zero-points 1143 for a collection of images. This is a task that wil berun much more1450 for a collection of images. This is a task that is run much more 1144 1451 infrequently than the object insertion tasks. 1145 1452 1146 \paragraph{Determine Consistent Photometry Zero Points (uniphot)} 1453 The relphot analysis is currently performed with a single Sky region 1454 as the starting point. All images (or all chips from all mosaic 1455 iamges) which overlap the sky region are identified in the image 1456 table. This set of images are considered set A. Next, all skyregions 1457 which are overlapped by all of these images are selected. Finally, 1458 all additional images which overlapped the new regions only are 1459 selected. These are considered as image set B. The image selections 1460 are also restricted to images of a single, user-selected photcode. 1461 1462 All of the objects and detections which are contributed by the images 1463 in sample A are extracted from the average and measure tables. Only a 1464 subset of the detections for which the S/N is greater than a 1465 user-selected limit are kept. Other restrictions, such as time range 1466 or instrumental magnitude ranges may also be specified. The 1467 collection of average objects, their detections, and the images from 1468 which they were derived now define a system of photometry equations. 1469 In this system, every image has a calibration offset magnitude 1470 ($M_{cal}$), every object has an average magnitude in a relative 1471 system ($M_{rel}$), and every detection of that object has a magnitude 1472 defined by the equation $M = M_{rel} + M_{cal}$. The goal is to solve 1473 for the values of $M_{ref}$ and $M_{cal}$. 1474 1475 There are two points to note about this operation. First, the system 1476 of equations is generally much too large to solve directly; we must 1477 use an iterative technique to converge on a solution. Second, it is 1478 important in the analysis to use robust averaging and identify 1479 detections, stars, or images which are deviant in some way. These 1480 should be marked and given set weight in the solution. These cases 1481 may represent poorly measured objects (perhaps detections on or near a 1482 bad column), variable stars, and images obtained in poor weather 1483 conditions. 1484 1485 Relphot can also be used to determine the mosaic grid used to generate 1486 photometrically corrected flats (-grid option). 1487 1488 \subsubsection{Uniphot : Zero Point Analysis} 1147 1489 1148 1490 This operation uses the time history of relative photometry zero 1149 1491 points for images and the spatial overlap information to determine a 1150 1492 best set of image zero points which have a specific time scale for the 1151 atmospheric stability. 1152 1153 \paragraph{Determine Distortion and Static Astrometry Model (mosastro)} 1493 atmospheric stability. This analysis would be used after relative 1494 photometry has been determined for data in DVO. This analysis 1495 currently is defined to unify the zero points of a collection of 1496 disjoint regions; additional modifications will be needed to 1497 simultaneously determine consistent zero points and relative 1498 photometry corrections for a collection of images distributed over a 1499 large range in space and time, but still with significant 1500 overlap. distritions with subustanailaccount for the c 1501 1502 \subsubsection{Global Astrometry Analysis} 1154 1503 1155 1504 This operation uses the reference and image detections to determine an … … 1164 1513 ideal flat focal plane. . 1165 1514 1166 \begin{table} 1167 \begin{center} 1168 \caption{AP Detection Classes \& Object Parameters\label{tab:APdetections}} 1169 \begin{tabular}{lrrrr} 1170 \hline 1171 \hline 1172 Object Parameter & P2 & P4S & P4D & SS \\ 1173 \hline 1174 PSF x,y, covar, $\alpha,\delta$ & + & + & + & + \\ 1175 PSF mag, $\sigma_{\rm mag}$ & + & + & + & + \\ 1176 star/gal sep & + & + & + & + \\ 1177 $\sigma_x$, $\sigma_y$, $\theta$ & + & + & + & + \\ 1178 local sky data & + & + & + & + \\ 1179 Petrosian R, M, $R_{50}$, $R_{90}$ & - & + & - & + \\ 1180 S\'ersic R, M, AB, $\phi$, $\nu$ & - & + & - & + \\ 1181 W.L. $\gamma_1$, $\gamma_2$, pol. terms & - & - & - & + \\ 1182 exp. spaced aps., Poisson noise, variance & - & - & - & + \\ 1183 \hline 1184 \end{tabular} 1185 \end{center} 1186 \end{table} 1187 1188 \subsubsection{Throughput} 1189 1190 The AP Database design partly driven by the need to make the 1191 detection-object associations quickly and to processes the incoming 1192 detections at a sufficiently high rate to meet the throughput 1193 requirements. For each upload of the object detections from a 1194 complete FPA, the AP Database must match roughly $1.4 \times 10^{6}$ 1195 detections from an FPA with roughly $6.4 \times 10^{6}$ objects, 1196 including orphaned bright detections. This corresponds to roughly 640 1197 MB, if each object uses 100 bytes for its descriptive informations 1198 (more than is currently specified in the Object table). With a 1199 throughput of 100 MB/s for reads from a RAID, the AP Database can 1200 perform the data read in a fraction of a second if the data is 1201 distributed across 10 computers. 1515 \subsubsection{DVO shell} 1516 1517 The DVO Shell is a user-tool for examining the visualizing data stored 1518 by DVO. The DVO Shell uses the Opihi shell language structure (see 1519 also PanTasks, Section~\ref{pantasks}), which provides a rich data 1520 analysis language. The shell language provides the user with 1521 capabilities to define new commands, set and manipulate scalar, 1522 vector, and image structures, and plot 2D graphics, including 1523 projections of the sky. In addition, the DVO shell is aware of the 1524 DVO data tables and provides access mechanisms to these tables. The 1525 following is a brief overview of these table access features. 1526 1527 DVO provides several ways to access the information stored in the 1528 database. Several simple commands allow the user to extract 1-D 1529 information directly from one of the primary database tables. The 1530 fundamental such commands are: 1531 \begin{itemize} 1532 \item imextract 1533 \item avextract 1534 \item mextract 1535 \end{itemize} 1536 1537 These commands allow the user to extract data from one of the columns 1538 represented by the image table(s), the average object tables, or the 1539 measurement tables. The extraction places the resulting data into a 1540 vector data elements, which may be used to make plots or perform 1541 analysis. The user may constrain the query with spatial selection, by 1542 photcode, by time ranges, and so forth. Some examples: 1543 \begin{verbatim} 1544 avextract all ra : select ra for all objects in displayed region 1545 avextract all g : select g magnitudes 1546 avextract all mag -photcode r : select r magnitudes 1547 avextract all Xm -photcode r : select chisq values for r average mags 1548 \end{verbatim} 1549 1550 Beyond these basic vector extractions, the user may perform more 1551 complex extract operations such as color-based selections. For 1552 example, color-color diagrams can be easily made by extracting the 1553 colors from the average or measurement tables and plotting the 1554 resulting vectors. The \code{ccd} commands extracts a specified pair 1555 of colors for all objects with that color pair from the specified data 1556 region. Similarly, the command \code{cmd} extracts a color and a 1557 magnitude into a pair of vectors. Both commands may specify any of 1558 the different types of magnitudes (relative, calibrated, etc) 1559 discussed above. 1560 1561 An additional class of DVO Shell commands perform more complex 1562 graphical operation. For example, the command \code{images}, plots 1563 the images which the specified region on the plotting too. Other 1564 commands allow the user to extract the images or the database tables 1565 which overlap specified locations. 1566 1567 \begin{figure} 1568 %\resizebox{4.5in}{!}{\includegraphics{pics/polar}} 1569 \caption{\label{polar} \small Map of 1570 the sky in polar project, and images added to database. } 1571 \end{figure} 1572 1573 \begin{figure} 1574 %\resizebox{4.5in}{!}{\includegraphics{pics/fullsky}} 1575 \caption{\label{allsky} \small Map of the entire sky, and images added to database. } 1576 \end{figure} 1577 1578 Some examples of using the DVO shell to perform visualization are 1579 given in Figures~\ref{polor} and \ref{allsky}. 1202 1580 1203 1581 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% … … 1924 2302 up the analysis stages are touched upon in Section~\ref{sec:PanTasks}, 1925 2303 which discusses the IPP Scheduler program, PanTasks. They are 1926 discussed in more detail in the document 'ippTools'.2304 discussed in more detail in Section~\ref{ippTools}. 1927 2305 1928 2306 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 1929 2307 1930 2308 \subsection{Phase 1: image processing preparation} 1931 1932 \tbd{need to add a discussion of Phase 0}1933 1934 \tbd{need to incorporate a discussion of ppImage, etc as distinct from1935 the ``phases''}1936 2309 1937 2310 The Phase 1 analysis stage is performed on each science exposure (each … … 2794 3167 year, or an average rate of $\sim$2 Mpix per second, or $< 1$\% of the 2795 3168 object analysis in the other analysis stages. 2796 2797 \section{IPPtools}2798 2799 Above, we discussed PanTasks, the IPP scheduler which determines the2800 new jobs to run and distributes them to computers across the network.2801 PanTasks is a general tool; by it self it does not define the specific2802 analysis tasks that the IPP requires. The previous few sections2803 discussed in detail the analysis which is performed by the IPP2804 analysis stages. IPPtools is the collection of PanTasks scripts,2805 Metadata Database interaction programs, and other tools used to2806 definet the specific analysis stages of the IPP.2807 2808 \tbd{this section needs to be fleshed out with a summary of the2809 ippTools functions. The stand-alone IPPTools document gives a2810 detailed discussion of these issues}.2811 3169 2812 3170 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% … … 3254 3612 stage and the image combination stage with robust outlier rejection. 3255 3613 \tbd{Paul: flesh this out!} 3614 3615 \subsection{Command Sequences} 3616 3617 It is useful in order to understand the analysis sequence to examine 3618 the complete series of processing steps involved in the analysis 3619 stages discussed above. We first illustrate the Phase 1-3 sequence, 3620 giving the commands which start with a raw image available on disk and 3621 results in a collection of detrended chip images, a high-quality 3622 astrometric calibration, and a collection of object detections. 3623 3624 \subsubsection{Phase 1-3 Analysis Commands} 3625 3626 In the example below, we imagine a GPC image available on disk with 3627 the exposure ID 654321.obj and chip IDs 00 through 88. The IPP design 3628 does not mandate specific naming convensions for the exposure IDs and 3629 the chip IDs; these values are opaque strings supplied by the data 3630 source (eg, GPC). Below, in Section~\ref{ipptools}, we discuss how 3631 the names and inputs are constructed, and how the relationships are 3632 tracked between an exposure and the data containers which make up the 3633 exposure. Also, the details of directory naming and organization are 3634 just examples, though some nightly folder scheme is a likely option. 3635 For the moment, these are assumed to be known by the system 3636 3637 Also discussed in Section~\ref{ipptools} it the concept of multiple 3638 analysis passes for a data element. Within the IPP, any data may be 3639 processed multiple times; the system tracks each attempt to process a 3640 particular set of data, tracking the analysis versions numbers which 3641 increment sequentially for each new attempt. In the sequence below, 3642 we are performing the first analysis attempt on the data, so the 3643 version numbers are all 0. 3644 3645 \begin{verbatim} 3646 Phase 1: 3647 ppImage -recipe PHASE1 3648 -inglob file:/data/2006.11.01/7654321o/7654321o.??.fits 3649 -out_astro file:/data/2006.11.01/7654321o/7654321o.XX.P1.00.ast.fits 3650 3651 Phase 2: 3652 ppImage -recipe PHASE2 3653 -infile file:/data/2006.11.01/7654321o/7654321o.24.fits 3654 -in_astro file:/data/2006.11.01/7654321o/7654321o.XX.P1.00.ast.fits 3655 -outfile file:/data/2006.11.01/7654321o/7654321o.24.P2.00.img.fits 3656 -outmask file:/data/2006.11.01/7654321o/7654321o.24.P2.00.msk.fits 3657 -outvar file:/data/2006.11.01/7654321o/7654321o.24.P2.00.var.fits 3658 -objects file:/data/2006.11.01/7654321o/7654321o.24.P2.00.cmf.fits 3659 -thumb file:/data/2006.11.01/7654321o/7654321o.24.P2.00.thm.fits 3660 -binned file:/data/2006.11.01/7654321o/7654321o.24.P2.00.bin.fits 3661 3662 Phase 3: 3663 ppImage -recipe PHASE3 3664 -inglob file:/data/2006.11.01/7654321o/7654321o.??.P2.00.cmf.fits 3665 -out_astro file:/data/2006.11.01/7654321o/7654321o.XX.P3.00.ast.fits 3666 -objects file:/data/2006.11.01/7654321o/7654321o.XX.P3.00.cmf.fits 3667 3668 ppImage -recipe MOSAIC 3669 -inglob file:/data/2006.11.01/7654321o/7654321o.??.P2.00.thm.fits 3670 -outjpeg file:/data/2006.11.01/7654321o/7654321o.XX.P3.00.thm.jpeg 3671 3672 ppImage -recipe MOSAIC 3673 -inglob file:/data/2006.11.01/7654321o/7654321o.??.P2.00.bin.fits 3674 -outjpeg file:/data/2006.11.01/7654321o/7654321o.XX.P3.00.bin.jpeg 3675 \end{verbatim} 3676 3677 In this example, the data are supplied by their file names in the UNIX 3678 file system. We are invoking the capability of ppImage to accept a 3679 glob to supply a list of files. In the Phase 1 stage, ppImage is used 3680 with the PHASE1 recipe to load data from a full mosaic (the files 3681 specified by the glob) and produce a single astrometry calibration 3682 file. The \code{XX} is supplied for the full-mosaic output files just 3683 to make the output data products appear in a more easily readable 3684 fashion is a directory listing. Also, note that we attach the 3685 \code{.fits} extension to all of these output files to make the data 3686 type more explicit to the reader. Again, none of these convensions 3687 are required by the analysis programs. 3688 3689 The Phase 2 analysis uses the Phase 1 astrometry to improve the 3690 astrometric starting guess. The analysis for Phase 2 is illustrated 3691 for just a single chip (24), though presumably equivalent commands are 3692 executed for the other 63 chips. The six output files selected in 3693 this example include the detrended image (\code{*.img.fits}), along 3694 with the corresponding mask (\code{*.msk.fits}) and variance images 3695 (\code{*.var.fits}) and the photometry results file 3696 (\code{*.cmf.fits}). Note that the output object file contains the 3697 astrometric solution parameters from this analysis stage. This 3698 process also constructs two smaller version output images: the binned 3699 image (\code{*.bin.fits}) and the thumbnail (\code{*.thm.fits}). The 3700 binning scale for these images is specified in the recipe for the 3701 camera; the first of these for GPC would likely be binned 32x32, while 3702 the second would probably be binned 320x320. 3703 3704 For the Phase 3 analysis stage, three actual analyses are illustrated. 3705 In the first, the photometry results files are identified by file glob 3706 and the result is an improved astrometric model for the camera and 3707 optics in our astrometry parameter table format. The object files are 3708 also grouped into a single multi-extension file along with the 3709 astrometric calibration. 3710 3711 In the second and third analysis examples, the collection of chip 3712 binned and thumbnail images are loaded by file glob, mosaic-ed 3713 together into a single image, and written to disk as a JPEG. These 3714 images are used by the pipeline tracking tool, ippMonitor. The binned 3715 image results in a full GPC image represented by 1200x1200 pixels; the 3716 thumbnail yields a 120x120 representation of the full GPC. 3717 3718 The example above would be sufficient if we were processing a small 3719 number of images by hand or for test purposes. However, the IPP is 3720 designed to be more flexible about the physical location of the data 3721 files that this illustration permits. The use of nebulous allows us 3722 to use a similar naming scheme and yet place the actual data files on 3723 different hardware depending on the chip ID (among other 3724 possibilities). To convert the filename version above to a version in 3725 which the files are stored on Nebulous simply requires changing the 3726 \code{file:} tag to \code{neb:}. The analysis programs recognize this 3727 tag to indicate a file available from Nebulous, and make a request to 3728 Nebulous for the actual file names. Nebulous can supply files based 3729 on a name match much like the file glob. Nebulous also allows the 3730 storage object ID to include path-like elements, allowing a structured 3731 organization of the files within Nebulous (which does not reflect a 3732 {\em physical location} relationship). 3733 3734 \subsubsection{Basic Detrend Creation Commands} 3735 3736 In the following example, we examine the steps to produce master 3737 detrend images. First, a few important points to note about this 3738 process. The construction of a master detrend frame (bias, flat, etc) 3739 involves combining a number of individual frames of an appropriate 3740 type of exposure, possibly after some preparatory processing. For 3741 example, in building a twilight flat-field image, 5 or 10 (or however 3742 many) raw flat-field images are first masked and bias corrected before 3743 being combined. To build a night-time fringe-frame image, a 3744 collection of raw night-time images are bias, dark, and flat-corrected 3745 before they are combined. In the combination, it may be necessary to 3746 apply some scaling and/or offset correction to the images to place 3747 them on a common footing. For example, in the construction of a 3748 master flat-field image, the individual images must be normalized in a 3749 consistent fashion; in building a master fringe frame, the fringe 3750 amplitude must be used as part of the scaling applied to the input 3751 images. In a mosaic camera, if individual chips are analysed 3752 independently, the resulting master chip images may require 3753 re-normalized to place the results on a common, consistent footing. 3754 3755 Beyond the details of the analysis steps, there is the question of the 3756 choice of input images. This choice is extremely dependent on the 3757 implementation for a particular camera, telescope, type of detrend 3758 image, etc. The analysis {\em process} should not be designed to make 3759 strong assumptions about the selection of the input data. In the IPP, 3760 the definition of the selection rules is part of the input 3761 configuration information and the scheduling rules, and can be 3762 considered outside of the discussion of the analysis commands. The 3763 IPP provides a tool, part of the \code{dettools} suite, which examines 3764 the Metadata Database tables for raw images of the appropriate type to 3765 select input images based on selection options such as time range, 3766 filter, camera, chip, exposure type, airmass, exposure time, etc. In 3767 the discussion below, we assume that some selection is made with 3768 \code{dettools}, resulting in a collection of input exposures and 3769 their corresponding chips. These lists are placed in tables which are 3770 then provided as part of the input to the analysis programs below; the 3771 corresponding images used as part of these inputs are also saved in 3772 Metadata Database tables as discussed in Section~\ref{ipptools}. In 3773 practice, the database tables provide the primary source; the list 3774 files are constructed from these tables and are simply intermediate 3775 data sources for the analysis programs. 3776 3777 Another important distinction to clarify in the detrend processing is 3778 between the detrend {\em run}, the {\em version}, and the {\em 3779 iteration}. These issues are discussed further in 3780 Section~\ref{ipptools}. Briefly, though, there are the following 3781 concepts to keep in mind: The detrend {\em run} is a particular 3782 attempt to construct a master detrend image. One run defines a 3783 collection of selection criteria for the initial set of input images. 3784 The resulting master detrend image is given an identifier, equivalent 3785 to the exposure ID. If the same selection criteria are used multiple 3786 times (eg, for multiple experiments on the analysis recipe used to 3787 construct the image), the same detrend ID may be used for multiple 3788 detrend runs. In this case, each new detrend run is given a different 3789 {\em version} number, equivalent to the version numbers used to track 3790 the science analysis passes. For the detrend image construction, this 3791 concept must go one level further, however. In order to produce a 3792 single validated master detrend image, it is necessary in general to 3793 produce multiple intermediate attempts. The intermediate master 3794 frames are applied to the input images; the statistics of the residual 3795 images are then used to select a subset of the input images, rejecting 3796 poor quality or deviant images. This processing is a form of 3797 image-level outlier rejection, and is particularly necessary for input 3798 images which result from observations of the sky (eg, twilight flats 3799 or night-time fringe frames); images obtained using stable calibration 3800 sources may not require this level of iterative processing. This type 3801 of analysis can also be used to determine if a new master frame is 3802 needed (all input images internally consistent, but deviant from the 3803 current best master) or if conditions are unacceptable to produce a 3804 new master (all input images mutually inconsistent). In order to 3805 track these multiple analysis passes, the IPP infrastructure assigns 3806 iteration numbers to the data products associated with a particular 3807 detrend run and version. 3808 3809 One final point to address is the issue of the validity domain of a 3810 detrend image. The end result of the detrend run is a master detrend 3811 image of a particular type, e.g., a master r' flat-field image for 3812 GPC-1. As a result of the input selection criteria, the resulting 3813 master detrend frame will have a primary domain of validity, which 3814 consists of a particular camera, telescope, set of chips, and which 3815 may include a time range, filter, airmass range, etc. The primary 3816 domain of validity defines those images which would be best processed 3817 with the particular master detrend image. Beyond this primary domain 3818 of validity is a wider, relaxed domain of partial validity. 3819 3820 Clearly, a SkyProbe flat-field image would be inapporiate in all 3821 context to be applied to a GPC image. Likewise, a GPC-1 $r'$ flat would 3822 be inappropriate for a GPC-1 $g'$ science image. However, in some 3823 circumstances, it is appropriate or desireable to apply a detrend 3824 frame to an image outside of its primary domain of validity. For 3825 example, if a master flat-field image was produced using input images 3826 from a certain week, an image from a different week may be viewed as 3827 outside the primary domain; however, for some experiments or because a 3828 flat-field in the appropriate time range could not be produced, it may 3829 be acceptable to apply the out-of-date flat-field image to the science 3830 image. In general, any image of the appropriate type, camera, filter 3831 (if a valid construct) and detector is in the partial domain of 3832 validity as a detrend image with those same values. The detrend 3833 creation system assigns a primary domain of validity to the masters 3834 which it creates; it is the choice of the analysis routines to apply 3835 images from a more relaxed domain, if necessary or desired, or to 3836 require the primary domain and yield an error if it is not available. 3837 3838 Below we give the series of analysis commands used to construct a 3839 master detrend frame. In this example, we construct a master $r'$ 3840 flat-field image for GPC1, using the detrend images for week 050. In 3841 practice, we will likely build detrend frames on a nightly basis, but 3842 the choice of timescale will depend to some extent on the observing 3843 process and the stability of the system. In this example, we 3844 construct a detrend ID using the camera, type, filter, and week 3845 number, though this choice is completely arbitrary. We also 3846 illustrate the example for one of the input images with exposure ID 3847 7654321f and for chip 24. In this example, this is the third time 3848 this image has been used for the analysis, thus the processing results 3849 for this frame/chip are given a version number of 02. 3850 3851 \begin{verbatim} 3852 dettools -define [selection criteria] -detID GPC1.flat.r.w050.00 3853 3854 ppImage -recipe MKDET.PROCESS 3855 -infile file:/data/2006.11.01/7654321f/7654321f.24.fits 3856 -output file:/data/2006.11.01/7654321f/7654321f.24.PC.02.img.fits 3857 3858 ppMerge -recipe MKDET.STACK 3859 -inlist file:/data/detrend/w050/GPC1.flat.r.w050.v00.n00.c24.list 3860 -output file:/data/detrend/w050/GPC1.flat.r.w050.v00.n00.c24.fits 3861 3862 ppNorm -recipe MKDET.NORM 3863 -inglob file:/data/detrend/w050/GPC1.flat.r.w050.v00.n00.c??.fits 3864 3865 ppImage -recipe MKDET.RESID 3866 -infile file:/data/2006.11.01/7654321f/7654321f.24.PC.02.img.fits 3867 -output file:/data/2006.11.01/7654321f/7654321f.24.RS.06.img.fits 3868 -thumb file:/data/2006.11.01/7654321f/7654321f.24.RS.06.thm.fits 3869 -binned file:/data/2006.11.01/7654321f/7654321f.24.RS.06.bin.fits 3870 3871 ppImage -recipe MOSAIC 3872 -inglob file:/data/2006.11.01/7654321o/7654321o.??.RS.06.bin.fits 3873 -outjpeg file:/data/2006.11.01/7654321o/7654321o.XX.RS.06.bin.jpeg 3874 3875 ppImage -recipe MOSAIC 3876 -inglob file:/data/2006.11.01/7654321o/7654321o.??.RS.06.thm.fits 3877 -outjpeg file:/data/2006.11.01/7654321o/7654321o.XX.RS.06.thm.jpeg 3878 3879 ppImage -recipe MKDET.MOSAIC. 3880 -inglob file:/data/detrend/w050/GPC1.flat.r.w050.v00.n00.c??.fits 3881 -outjpgt file:/data/detrend/w050/GPC1.flat.r.w050.v00.n00.thm.jpeg 3882 -outjpgb file:/data/detrend/w050/GPC1.flat.r.w050.v00.n00.bin.jpeg 3883 3884 \end{verbatim} 3885 3886 Again, this data illustrates the use of files in the UNIX file system; 3887 the substitution of \code{neb:} for \code{file:} will inform the 3888 programs to retrieve the file names via Nebulous. In this example, we 3889 use output names for the intermediate images which are equivalent to 3890 those used for the science processing; the version numbers used for 3891 these data products are sequential over all detrend runs which use 3892 those input frames. Also, for the detrend image products, we have 3893 added the 'v', 'n', 'c' tags to clarify which of the two-digit numbers 3894 represents a version (v), an iteration (n), and a chip ID (c). Note 3895 that this sequence of analysis steps makes heavy use of 3896 \code{ppImage}, with different choices of the recipe and the output 3897 options to change the behavior somewhat. However, all of the uses of 3898 \code{ppImage} represented here are consistent with the primary 3899 responsibilities of \code{ppImage}: read in the file or mosaic image 3900 into the correct level of the image hierarchy, perform a detrend 3901 analysis (including rebinning in this category), re-structure the 3902 collection of image arrays as described by the recipe, write out the 3903 image in the desired format. 3904 3905 \section{IPPTools} 3906 3907 PanTasks is the IPP tool which manages the sequencing of data analysis 3908 steps and, with the related tool `PControl', distributes the data 3909 processing across a cluster of computers. However, by itself, PanTasks 3910 does not determine the organization of data or the analysis sequencing 3911 for a particular pipeline. This level of information is contained 3912 within specific PanTasks scripts. To use the tasks defined by a set 3913 of PanTasks scripts, additional helper programs are needed. This 3914 section discusses these programs and the PanTasks scripts used by the 3915 IPP. 3916 3917 IPPTools is a collection of programs, Metadata Database table 3918 definitions, and PanTasks scripts used to define the actual data 3919 organization and the sequencing of operations by the IPP. Within the 3920 IPP, the Metadata Database is used to store the analysis state, as 3921 well as result processing data points. This section discusses the 3922 tasks needed to define each of the IPP analysis stages (Phase 1-4, 3923 detrend creation, etc) and examines the relevant MDDB tables. 3924 3925 \subsection{Persistent vs Ephemeral State in PanTasks} 3926 3927 \begin{figure} 3928 \begin{center} 3929 \includegraphics[scale=0.85]{pics/ipptools.01.ps} 3930 \caption{\label{queues} PanTasks queues and MDDB tables} 3931 \end{center} 3932 \end{figure} 3933 3934 The IPP, a fairly complex analysis system, uses PanTasks to select 3935 jobs, distribute them to the cluster, and harvest the results. It 3936 uses the Metadata Database to record the results of a given analysis 3937 step, and to determine which jobs must be performed when. 3938 3939 There are some subtleties in the interaction between PanTasks, the 3940 Metadata Database tables which store the system state, and the jobs 3941 which are currently being performed. There is a choice to be made 3942 between rigorously maintaining the system state in the Metadata DB at 3943 all times or keeping an intermediate set of state tables. Keeping the 3944 exact system state in the Metadata DB tables would require many extra 3945 queries to/from the database and may introduce additional latencies 3946 which are undesirable. This is because any attempt by PanTasks to 3947 initiate a new job would require PanTasks to mark the corresponding 3948 data item in the Metadata DB (the item which acts as the trigger) with 3949 a `pending' state, and then mark it again as `done' when the job 3950 actually completes. This also has the drawback that, if the system 3951 crashes (eg, hardware failure), some initial process would be required 3952 on start up to find all Metadata DB items which are in the `pending' 3953 state (examining all possible items which can be in such a state) and 3954 reset them to the `new' state. 3955 3956 We implement an alternative in which PanTasks maintains an internal, 3957 ephemeral stack of the pending jobs, and only updates the system state 3958 entries in the Metadata DB when jobs are actually completed. In this 3959 scenario, as far as the Metadata DB tables are concerned, data items 3960 transition only between a `new' and a `done' state. Any jobs which 3961 are pending when the system crashes or the power is lost are simply 3962 dropped, and will be automatically re-constructed when the system 3963 restarts. In this paradigm, no intermediate operation state is saved, 3964 and no partially completed job can be recovered. Since the IPP is 3965 defined in terms of a fine granularity, with jobs lasting no more than 3966 30 - 120 seconds, crashes under this model will not have a large 3967 impact on the data processing. 3968 3969 Figure~\ref{queues} illustrates this ephemeral vs persistent state 3970 information and the interrelation between the metadata tables and 3971 PanTasks. The left-hand portion of the diagram illustrates the 3972 recommended interaction between the metadata database tables and 3973 PanTasks' internal queues. Some table in the metadata database 3974 defines a list of data items which are to be processed by some 3975 analysis job. PanTasks uses a two-step approach to define the 3976 analysis jobs based on this list. First, one task queries the MDDB 3977 for a list of pending items, adds the returned items to an internal 3978 PanTasks queue. The process of adding the elements to the queue is 3979 defined so that only unique items are added: already existing items 3980 are skipped. The entries in the queue consist of the data items of 3981 interest and an internal temporary state. At first, this would be 3982 `pending'. A second tasks pops `pending' entries one-by-one from this 3983 internal queue, submits a job based on the entry, and sets the 3984 temporary state in the internal queue to `running'. The internal 3985 state is needed to prevent PanTasks from re-submitting a job for the 3986 same data item before the first job is done or assessed. Since the 3987 job make take an arbitrary amount of time, PanTasks requires a 3988 mechanism to remember which data items it has already submitted. When 3989 the job eventually completes, the metadata database table is updated 3990 noting the completion. This may be done either by the job itself or 3991 by PanTasks as part of the job exit rules. In addition, the state of 3992 the entry in the queue can be set to either `done' or the entry can be 3993 simply removed from the queue. 3994 3995 3996 The purpose of this interaction is to maintain the temporary state 3997 information within non-persistent elements of PanTasks rather than 3998 using the metadata database tables to store this information. This 3999 concept has two advantages. First, PanTasks internal queues are in 4000 memory and relatively small, thus interfacing with them is quite fast 4001 for PanTasks -- this should reduce the system latency. Second, by 4002 keeping this information non-persistent, the system responds correctly 4003 to stopping and restarting PanTasks. Any jobs which have not been 4004 completed will not be marked in the database, and will be restarted 4005 naturally by PanTasks. The alternative, of writing a temporary state 4006 marker in the database would require PanTasks, on startup, to 4007 initially clean all database tables of these temporary state markers. 4008 4009 The right-hand portion of the diagram illustrates this process using 4010 the process of copying the images from the summit as an example. The 4011 metadata database table of interest in this case is the list of 4012 pending images, with entries supplied by a job which queries the 4013 summit data systems. The job which is actually performed is a remote 4014 copy of the image file from the location specified by the summit data 4015 system to the appropriate location within the IPP Image Server 4016 (Nebulous). (As an alternative to the above, the `pending images' 4017 table may be part of the summit database system, and the `get images' 4018 command may query the summit directly. In this scenario, the `copy 4019 image' command reports to the summit data system that an individual 4020 image file has been copied.) 4021 4022 In the rest of this document, the use of PanTasks internal queues to 4023 manage the temporary data states is glossed over and assumed part of 4024 the tasks defined in the process. 4025 4026 4027 \subsection{IPP Pipelines Overview} 4028 4029 The IPP as a whole performs all of the image analysis functions 4030 required by the Pan-STARRS telescopes, including images from the full 4031 Gigapixel camera (or cameras), the test camera TC-3, and the SkyProbe 4032 camera. The IPP is designed to be very flexible, with instrument 4033 specific details isolated in configuration files associated with the 4034 different cameras known to the system. As a result, the organization 4035 of the top level analysis infrastructure must be sufficiently general 4036 that a wide range of cameras can be accomodated. We have a few 4037 general principles regarding constraints on the data to be processed 4038 which are used to guide the IPP design and developement: 4039 4040 \begin{itemize} 4041 \item {\bf Camera Focal Plane Hierarchy} The IPP analysis programs 4042 assume that the images to be processed are obtained by a camera 4043 which can be represented by our Camera Focal-Plane Hierarchy of data 4044 structures. This hierarchy is discussed in detail in the Modules 4045 SDRS, and defines a top-level {\em Focal-Plane Array (FPA)}, which 4046 may contain 1 or more {\em Chips}, each of which may contain one or 4047 more {\em Cells}. An {\em FPA} is identified as having a single 4048 optical system feeding photons to the detectors. A {\em Chip} is 4049 identified as a unit of data all deriving from a single detector 4050 (piece of silicon), while a {\em Cell} is identified as a collection 4051 of pixels read out as a continuous cartesian grid. Finally, a 4052 single collection of data from an {\em FPA} may include multiple 4053 {\em Readouts} from any or all of the {\em Cells}. 4054 4055 \item {\bf Exposures vs Groups} The processing presumes that the data 4056 is organized into {\em exposures} and exposure {\em groups}. An 4057 exposure represents the data from a single FPA, with the possible 4058 subdivision of the exposure into multiple readouts for some or all 4059 of the cells. Exposure {\em Groups} are any group of exposures 4060 which are related together in some way; the definition of the {\em 4061 Groups} may be provided by the observers, or they may be derived 4062 from the characteristics of the exposures. The use of a particular 4063 {\em group} depends on the context of that group. A few examples of 4064 exposure groups: 4065 4066 \begin{itemize} 4067 \item a dithered sequence of exposures to be stacked for cosmetics 4068 and improved signal-to-noise. 4069 \item a twilight flat-field sequence. 4070 \item all images of the same filter within a 10 degree region to be 4071 used to construct an sample astrometric reference. 4072 \end{itemize} 4073 4074 \item {\bf Image Files (imfiles) vs Exposures} Any single exposure 4075 may consist of a number of different data files. The number of {\em 4076 imfiles} for a given exposure will depend on the camera, as will the 4077 data organization within those image files. Also, a particular 4078 camera will supply files corresponding to one of the particular 4079 Focal-Plane Hierarchy elements. The IPP analysis must be able to 4080 interpret the incoming data correctly. 4081 \end{itemize} 4082 4083 As discussed elsewhere, there are several major types of analysis 4084 performed by the IPP. For the purposes of data organization and 4085 parallel processing efficiencies, we have identified the following 4086 divisions of the analysis tasks. These will be discuss in much more 4087 detail below. 4088 4089 \begin{itemize} 4090 \item {\bf Science Image Analysis} : This represents the analysis 4091 performed on the images obtained by the telescope, and generally 4092 performed in real-time, night-by-night. The science image analysis 4093 tasks are further subdivided as follows: 4094 4095 \begin{itemize} 4096 \item {\bf Phase 1} : The full focal-plane array is examined quickly 4097 to determine an initial astrometric calibration. In this step, the 4098 OTA guide stars may be used as the astrometric reference; if none 4099 are available, predicted bright star positions are examined. This 4100 step is only used for mosaic images, and may be skipped if no guide 4101 stars are available {\em and} the astrometric calibration for the 4102 telescope / camera is reliable (better than 10 arcseconds). 4103 4104 \item {\bf Phase 2} : Each image file is analysed independently: the 4105 image is detrended (bias, dark, flat, fringe, etc), sources are then 4106 detected to a modest level, improved astrometric calibration is 4107 performed. 4108 4109 \item {\bf Phase 3} : The collection of sources measured from all of 4110 the image files for the camera are used to determine a global 4111 astrometric, and possibly photometric, solution for the exposure. 4112 This step is only required for mosaic cameras. 4113 4114 \item {\bf Phase 4.1} : An exposure group consisting of images 4115 obtained in a specific region of the sky are merged together. In 4116 this step, the images are first warped to a common pixel grid, defined by 4117 the static sky images. The collection of images are then used to 4118 construct a single, cleaned image by rejecting the outliers from the 4119 source images in the stack. The corresponding static sky pixels are 4120 then used to construct a difference image from the resulting stack. 4121 4122 \item {\bf Magic} : In this step, the difference images are examined 4123 to find the trailed images introduced by artificial satelites. 4124 These so-called {\em streaks} are excised from the difference 4125 images, as well as all of the source images which were used to 4126 generate the difference images; the public data sources are updated 4127 with the precise, correct time. Note that this step requires that 4128 separate difference images be generated for each of the input 4129 images, a step which would be skipped if {\em magic} were avoided. 4130 Also note that, until {\em magic} is performed, the publically 4131 available time has a limited precision (probably $\sim 1$ minute 4132 errors). This step is only necessary in the operational IPP system 4133 given the restrictions from the Air Force. 4134 4135 \item {\bf Phase 4.2} : After {\em magic} the final difference and 4136 the final cleaned stacked image are produced and objects in both 4137 images are detected. The difference sources are used to mask the 4138 extreme outliers in the cleaned stack, which is then used to update 4139 the Static Sky images. 4140 \end{itemize} 4141 4142 \item {\bf Static Sky Image Analysis} : While the science image 4143 analysis is performed as images are availablef, the static sky image 4144 analysi occurs on a very different timescale. In steady state, the 4145 full static sky analysis will take place over the course of a full 4146 year. At any given time, the portion of the sky corresponding to the 4147 location of the sun will be under-going the analysis. In practice, 4148 for PS-1, the static sky is produced in a somewhat different fashion 4149 than in the steady-state model. In PS-1, the different survey 4150 strategies introduce very different update rates for the static sky. 4151 At one extreme, the AP Survey will not have enough data for a 4152 complete static sky analysis until nearly 22 months after the survey 4153 begins. At the other extreme, the deep survey, which observes a much 4154 smaller portion of the sky, may best be analysed quite frequently. 4155 These details are part of the science guidelines of the PS-1 surveys, 4156 and are beyond the scope of this document. Rather, the IPP Static 4157 Sky Image Analysis must provide the capability of defining the static 4158 sky analysis in a flexible and dynamic fashion. 4159 4160 \item {\bf Basic Detrend Creation Analysis} : The analysis of most of 4161 the detrend data is grouped together in a common analysis stage. 4162 The differences between the analysis of the bias, dark, flat, and 4163 fringe images is primarily one of how the input images are 4164 pre-processed, what statistic is used to characterize a given input 4165 image, how the input images are scaled before being combined, and 4166 what normalization is applied to the resulting image. All of these 4167 types of detrend images can thus be processed with a single analysis 4168 pipeline which is made aware of these minor differences. This stage 4169 is never the less fairly complex, and as a result is subdivided into 4170 several compenents, as discussed below. 4171 4172 \item {\bf Other analyses} There are a number of other tasks which the 4173 IPP must perform that are not well-defined by the different analysis 4174 types discussed above. Some analysis tasks are not automatically 4175 triggered, and are thus outside the scope of this document; these 4176 are the tasks which are more properly considered as research 4177 projects than analysis systems. The other important automatic tasks 4178 are: 4179 \begin{itemize} 4180 \item {\bf Summit Copy} : In this stage, the data source or data 4181 sources are queried for new exposures and image files, which are 4182 then copied to the IPP data area. This stage also includes the 4183 copying of other metadata which are not included in the image 4184 files. 4185 4186 \item {\bf Image Classification} : new images which are introduced 4187 to the IPP are examined by this analysis stage and placed in the 4188 appropriate table for processing. This step includes a small 4189 amount of accumulating statistics about the images. 4190 4191 \item {\bf Data File management} : a few tasks are necessary to 4192 monitor and maintain the clustered storage system. These tasks 4193 include the automatic duplication and deletion of different types 4194 of files from Nebulous, the file storage archive. This also 4195 includes automatic redistribution of machine assignments as 4196 hardware is added or removed from the system. This collection of 4197 tasks also includes monitoring of system parameters to alert 4198 people in case of dangerous hardware situations. 4199 4200 \item {\bf Irregular Calibration Data} certain types of 4201 calibration information is extracted on different intervals from 4202 the more regular detrend images. These types of calibration data 4203 include improved telescope pointing models, astrometric 4204 calibrations, photometric calibrations, flat-field correction 4205 frames. 4206 \end{itemize} 4207 \end{itemize} 4208 4209 \subsection{Tables, Tasks and Tools} 4210 4211 The following sections discuss the database tables, the tasks within 4212 PanTasks, and the collection of programs used by PanTasks to examine 4213 and manipulate the state tables. These later programs do not, in 4214 general, perform any in depth analysis; instead they perform actions 4215 such as selecting from one table images ready for analysis in a 4216 following processing step. This collection of tools is grouped under 4217 the name of the {\tt ippTools}, and consists of a separate tool for 4218 each of the different major analysis steps. 4219 4220 The {\tt ippTools} make use of {\em glueforge} to simplify the 4221 management of the database table schema. Glueforge provides a single 4222 mechanism to generate a collection of C data structures, database 4223 tables, database access APIs, and I/O routines from a simple table 4224 description configuration file. All APIs generated by {\em glueforge} 4225 for the same type of interaction have common naming schemes. This 4226 technique has several important advantages. It makes the writing of C 4227 database interactions very quick and easy. It also makes it easy to 4228 modify the database schema without disrupting the software 4229 development. Finally, it provides a simple, self-documenting source 4230 for data structure of multiple types which can be shared between 4231 programs or platforms. 4232 4233 Within the following diagrams, we illustrate the database tables used 4234 to track the state of the IPP. We also show the commands provided by 4235 {\tt ippTools} to connect the tables. Finally, we show the IPP tasks 4236 which initiate the different analysis steps. The following set of 4237 diagrams uses several consistent features. The blue-and-grey 4238 rectangles define the metadata database tables. The blue section 4239 contains the table name, while the grey section lists a minimal subset 4240 of the table columns. The ellipses represent programs (or program 4241 portions in some cases) executed by PanTasks. The blue filled 4242 ellipses represent the {\tt ippTools} commands which are executed 4243 locally on the computer hosting PanTasks. The grey-blue ellipses 4244 represent the commands executed on the parallel cluster, monitored by 4245 {\tt pcontrol}. The green ellipses represent commands executed by 4246 hand for testing and manual intervention. 4247 4248 In most of the analysis tasks, we use a two-table approach to the data 4249 in order to avoid excessive latencies. One table is used to track 4250 quantities which are still pending for a particular stage. When the 4251 analysis is completed, these items are moved from the 'pending' tables 4252 to corresponding 'done' tables. Although this introduces a somewhat 4253 higher number of tables and complexity, it will avoid the system from 4254 slowing down as the number of data items grows with time. The pending 4255 tables are searched repeatedly by the {\tt ippTools} programs as they 4256 attempt to select new data of interest. In contrast, the done tables 4257 are searched much less frequently. 4258 4259 \subsection{Summit Copy Tasks} 4260 4261 \begin{figure} 4262 \begin{center} 4263 \includegraphics[scale=0.85]{pics/ipptools.02.ps} 4264 \caption{\label{pcopy} Summit Copy Tasks} 4265 \end{center} 4266 \end{figure} 4267 4268 Figure~\ref{pcopy} illustrates the MDDB tables used to copy data 4269 (images and metadata tables) from the summit. The left-hand portion 4270 of the diagram shows the tables involved in copying images from the 4271 summit system. The table of pending image files lists the URLs of the 4272 individual image files available for transfer, along with their 4273 associated exposure ID and the camera which generated the image. Two 4274 other entries assist in interpreting the file: the class and the class 4275 ID. The final entry in this table is the current copy state of the 4276 file, can have the value of `ready' or `copied'. 4277 4278 The class defines the data grouping represented by this image file and 4279 may have values of: FPA, Chip, Cell. This value indicates that the 4280 provided image file represents the specified portion of the camera 4281 FPA. If the value is FPA, the file represents data from a complete 4282 FPA, though the file may contain pixel data in multiple extensions or 4283 other groupings to be identified later. If the value is chip, the 4284 file contains only data for a single chip, presumably of multiple 4285 chips available, and equivalently for Cell. Further discussion of the 4286 FPA image hierarchy is given in the IPP documents (eg, Modules SDRS). 4287 The class ID gives the identifier used to name the class level 4288 corresponding to this file. This value is necessary to make decisions 4289 on how to copy the data based on the chip / cell before the data is 4290 available to IPP components. Table~\ref{classes} lists likely values 4291 for the class and class ID for some common cameras. The system 4292 described is sufficiently flexible to allow us to transfer the GPC 4293 images by cell if we eventually decide that is more efficient. 4294 4295 The copy process copies the file from the given URL to the appropriate 4296 IPP node and adds an entry to the table of new image files, consisting 4297 of the same information as the pending image file table, though with a 4298 new value for the URL. This URL may be an explicit filename, a 4299 reference to an entry in the image server, or a web address, or 4300 located on the image server (marked with file:, neb:, and http:, 4301 respectively). (TBD: other possible file storage types? perhaps the 4302 path could be abstracted without going to the level of the image 4303 server? eg: ref:DIR0001/file0001.fits might be in a directory which 4304 is defined in a table of directories.) After an image file is 4305 successfully copied, the corresponding state in the `pending chip' 4306 table is updated from `ready' to `copied'. 4307 4308 \begin{table} 4309 \begin{center} 4310 \caption{Camera and Data Classes\label{classes}} 4311 \begin{tabular}{llll} 4312 \hline 4313 \hline 4314 camera & class & classID \\ 4315 \hline 4316 GPC & chip & chip02 \\ 4317 skyprobe & fpa & sp01 \\ 4318 Megacam & fpa & MegacamSpliced \\ 4319 Suprime & chip & chip0 \\ 4320 \hline 4321 \end{tabular} 4322 \end{center} 4323 \end{table} 4324 4325 The right hand portion of this diagram illustrates the process of 4326 copying a metadata table. The table of pending tables lists the URLs 4327 for the tables which are ready, a unique table ID for each table, and 4328 the table type. The copy function copies the listed table and uploads 4329 the data to the IPP version of the same metadata database. Two 4330 examples of metadata tables needed by the IPP for the basic image 4331 processing system are illustrated: the table of new exposures and the 4332 table of pending matches. The first lists the exposures which are 4333 avilable from the summit system, and all represent entries which are 4334 available from the Image server. the second represents the matches 4335 between exposure IDs and chips 4336 4337 \subsection{Phase 0} 4338 4339 \begin{figure} 4340 \begin{center} 4341 \includegraphics[scale=0.85]{pics/ipptools.03.ps} 4342 \caption{\label{phase0} Phase 0 Tasks} 4343 \end{center} 4344 \end{figure} 4345 4346 Figure~\ref{phase0} illustrates phase 0, in which the image files are 4347 categorised, examined for summary information and basic statistics, 4348 and moved to the later phase 'pending' tables to trigger further 4349 analysis. The command {\tt p0search -pending} examines the `new 4350 imfiles' and 'new exposure' tables. It selects images from this table 4351 which have not yet been examined (state is `new'). These are returned 4352 to PanTasks, which sends each image file to a separate analysis node 4353 running the {\tt p0search -update} command. With this command, the 4354 file header is examined and relevant metadata is extracted (eg, RA, 4355 DEC, times, and so forth to be defined later). The process may also 4356 select a portion of the image pixel data to determine a rough bias and 4357 background level. These statistics, whether derived from the header 4358 or the pixel values, are placed along with image summary information 4359 in the `raw image files' table, and the state field of the `new image 4360 files' table is set to `ready'. 4361 4362 The {\tt p0search -update} command is also responsible for moving the 4363 exposures to the tables used for triggering the analysis process. If 4364 the image class is FPA, the image can be advanced without waiting for 4365 any other image files. If the class is Chip or Cell, the process must 4366 also examine the `new exposure' table for this exposure ID. The 4367 number of class files available for this exposure is listed in this 4368 table. The process must the select all image files matching the 4369 exposure ID with state of `ready' and compare the number avalable to 4370 the number expected. If the two match, then a new exposure is ready. 4371 Based on the image type (from the most recently examined image file 4372 header or new exp table?), the exposure is added to the `raw exposure' 4373 table for images of that type. The allowed types are `detrend', (all 4374 bias, dark, flat images), `object', `focus'(??), etc. (** The 4375 different tables represent different analysis modes. This process 4376 also adds an entry to the exp ID / image file match **). This process 4377 also adds all science (OBJECT) exposures to the P1 exposure table (for 4378 mosaic data) or the P2 chip table (for single detector data). These 4379 tables are used to trigger the Phase 1 and Phase 2 analysis stages. 4380 4381 \subsection{Phase 1} 4382 4383 \begin{figure} 4384 \begin{center} 4385 \includegraphics[scale=0.85]{pics/ipptools.04.ps} 4386 \caption{\label{phase1} Phase 1 Tasks} 4387 \end{center} 4388 \end{figure} 4389 4390 Figure~\ref{phase1} shows the tables involved in running the Phase 1 4391 analysis stage. There are paths for exposures to enter the analysis 4392 automatically from the Phase 0 analysis (arrow on left) or to be added 4393 manually based on a selection from the raw exposure table. Exposures 4394 to be analysed by Phase 1 are added to the P1 exposure table with the 4395 state `new'. Exposures may be added multiple times for processing and 4396 reprocessing. The P1 done exposure table keeps a record of the old 4397 attempts for debugging and analysis. Each time an exposure is added 4398 to the P1 exp table, it is given a new, unique version number, 4399 allowing the system as a whole to track different analysis attempts. 4400 This method is used in all of the image analysis stages (and 4401 extrapolated to iterations in the detrend analysis steps below). The 4402 top portion of the diagram shows the use of the command {\tt p1search 4403 -define} to select and submit an exposure or a group of exposures, 4404 potentially selected on the basis of a query from the raw science 4405 exposure table. 4406 4407 The P1 pending exposure table is examined by {\tt p1search -pending} 4408 to select the new exposures, which are sent to PanTasks. PanTasks 4409 initiates a separate analysis job (p1astro) for each exposure, which 4410 are sent to the parallel processing nodes. Within the analysis job, 4411 the chips (image files) associated with the exposure are select from 4412 the raw image file table. The analysis examines the contents of these 4413 files, either extract the guide star information from the image files 4414 (GS table extension) or searches for and centroids the pixels on 4415 appropriate bright stars. The analysis results in astrometric 4416 calibration terms which are written to the astrometric calibration 4417 file for this exposure. The location of the astrometric calibration 4418 file and the statistics of the measurement are written back to the P1 4419 exposure table. The images associated with exposures which are 4420 successfully processed by P1 are then added to the P2 image table, 4421 which is used to trigger the Phase 2 analysis. This last step is 4422 performed by the command {\tt p1search -done}, which is executed 4423 regularly to search for completed Phase 1 jobs. 4424 4425 \subsection{Phase 2} 4426 4427 \begin{figure} 4428 \begin{center} 4429 \includegraphics[scale=0.85]{pics/ipptools.05.ps} 4430 \caption{\label{phase2} Phase 2 Tasks} 4431 \end{center} 4432 \end{figure} 4433 4434 Figure~\ref{phase2} shows the tables involved in running the P2 4435 analysis stage. There are paths for images to enter the analysis 4436 automatically from the P1 analysis (arrow on left) or to be added 4437 manually based on a selection from the raw exposure and raw image file 4438 tables. Image files to be analysed by Phase 2 are added to the P2 4439 pending imfiles table with the state `new'. When images are added to 4440 this table, a single entry is also added to the P2 exposure table 4441 listing the P1 and P2 versions for this exposure. These version 4442 numbers must be integers starting with 1. If this image did not have 4443 a P1 analysis, the P1 version is set to 0. Exposures may be added 4444 multiple times for processing and reprocessing. The P2 image table 4445 keeps a record of the old attempts for debugging and analysis. As 4446 with P1, each time a collection of associated images from an exposure 4447 is added to the P2 image table, it is given a new, unique version 4448 number, allowing the system as a whole to track different analysis 4449 attempts. Note that these version numbers are unique for each {\em 4450 exposure} processed by Phase 2, not just for any image file. The top 4451 portion of the diagram illustrates the behavior of the commands {\tt 4452 p2search -define} and {\tt p2search -quick}. The first may be used to 4453 re-submit the images for an exposure or a group of exposures, 4454 potentially selected on the basis of a query from the raw science 4455 exposure and raw image file tables. The second version sends images 4456 files directly to PanTasks for processing; these entries will not be 4457 included in the processing tables, and is used only for testing 4458 purposes. 4459 4460 The P2 pending image table is examined with the command {\tt p2search 4461 -pending} to select the `new' images. These images are used by 4462 PanTasks to generate P2 analysis jobs, running the analysis command 4463 {\tt ppImage}. The P2 analysis uses the input url to find and load 4464 the image file. The url may be a file on disk, an entry in the image 4465 server, Nebulous, etc. The master detrend images matching the 4466 specific science image and the conditions are selected by examining 4467 the table of master detrend frames. The specific detrend image files 4468 are selected by using the master detrend ID to select the matching the 4469 entries in the table of master detrend files. After the analysis, the 4470 output image, mask, and FITS table of objects, including the 4471 astrometry calibration, are written back to the P2 image table, along 4472 with summary statistics from the P2 analysis. The state is also 4473 updated (to `done'). 4474 4475 The completed images are examined by the command {\tt p2search -done}, 4476 and when all image files for a single exposure are completed, this 4477 command migrates them to the P2 done table. This process is also 4478 responsible for populating the P3 pending tables so exposures may be 4479 processing by Phase 3. 4480 4481 \subsection{Phase 3} 4482 4483 \begin{figure} 4484 \begin{center} 4485 \includegraphics[scale=0.85]{pics/ipptools.06.ps} 4486 \caption{\label{phase3} Phase 3 Tasks} 4487 \end{center} 4488 \end{figure} 4489 4490 Figure~\ref{phase3} illustrates the tables and commands involved in 4491 the Phase 3 analysis. The P3 pending exposure table lists the 4492 exposure ID, the P3 analysis version, the P2 analysis version to be 4493 used as input to this P3 analysis, and the recipe to be used. The 4494 command {\tt p3search -pending} extracts exposures from this table and 4495 provides them to PanTasks for processing. PanTasks launches a Phase 3 4496 analysis (the command {\tt psastro}?) for each exposure. In this 4497 analysis, the P2 exposure and image tables are used, in conjunction 4498 with the P2 version information, to select the P2 output measured 4499 objects and the astrometric calibrations from P2 and P1. These 4500 measured objects are matched with the reference catalog objects, and 4501 calibrated astrometry {\em and eventually photometry} is produced for 4502 the full exposure. The location of the resulting astometry 4503 calibration table is stored back in the P3 exposure table. If the 4504 recipe file specifies, the 2-D photometric and background / fringe 4505 corrections may also be performed at this stage. Since these analyses 4506 require reference data, the recipe may also be used to skip these 4507 analysis if such reference data is unavailable or unreliable. At the 4508 end of Phase 3, the objects from the exposure are inserted into the 4509 photometry database (this is not shown). 4510 4511 The astrometric calibration portion of Phase 3 is principally needed 4512 for a mosaic camera. For single-chip cameras, Phase 3 may be used to 4513 perform the photometric calibration and simply pass the astrometric 4514 results along to the output file to be listed in the P3 exposure 4515 table. In this way, later stages of the analysis (ie, Phase 4) can 4516 use the P3 exposure table as input for all cameras, even if all the 4517 funcionality of Phase 3 is not necessary for that camera. This would 4518 be the case for the skyprobe camera, for example. 4519 4520 \subsection{Phase 4} 4521 4522 \begin{figure} 4523 \begin{center} 4524 \includegraphics[scale=0.85]{pics/ipptools.07.ps} 4525 \caption{\label{phase4} Phase 4 Tasks} 4526 \end{center} 4527 \end{figure} 4528 4529 At the end of Phase 3, the images are ready for Phase 4. The Phase 3 4530 diagram shows the output line adding the exposures to be processed by 4531 Phase 4 to a Phase 4 table. However, this line is just for 4532 illustration purposes. The rules for initiating a Phase 4 run are 4533 somewhat more complicated than those for running Phases 1-3. Groups 4534 of exposures which have an appropriate overlap should be chosen for 4535 the Phase 4 analysis. In the steady-state period of PS-4, it may be 4536 straightforward to choose the exposure groups: they would simply be 4537 the exposures obtained nearly simultaneously by the four separate 4538 cameras. The circumstance for PS-1 will be much more complicated (and 4539 even PS-4 will probably be more complex than it seems at first 4540 glance). For example, in PS-1, we will not have a static sky for most 4541 of the AP Survery. In this circumstance, we cannot run P4, at least 4542 until after the complete AP Reference catalog is built, and 4543 potentially all exposures re-run through Phase 3. It may be useful 4544 for the AP Survey data to split the Phase 4 analysis into two stages: 4545 image combination and image differencing. It may even be the case 4546 that only the combination portion of Phase 4 is performed on the AP 4547 Survey data. 4548 4549 More generally, the image groups selected for Phase 4 analysis may be 4550 chosen on the basis of a query of the AP Database (DVO) with some 4551 rules. 4552 4553 \note{Phase 4 run can be defined by selecting an observation group, a 4554 set of exposures, or a set of rules related to a spatial region (eg, 4555 region, time range, and filter}. 4556 4557 \note{Phase 4 discussion (and diagram) needs more work} 4558 4559 \subsection{Analysis Version and Recipes} 4560 4561 Note that each of the stages P1-P4 refer to the processing version 4562 from the previous stage. This allows the processing stage to request 4563 the correct version of the results from the previous stage, and makes 4564 it possible to run and re-run the analysis at any stage without 4565 deleting the earlier results. As different analysis attempts are 4566 performed for a given image, the versions branch out. 4567 4568 Also note that at every stage, the entries include a recipe 4569 identifier. This is used to select the analysis recipe which should 4570 be used for this version. By default, the recipe should be set to the 4571 current best recipe (use a default name for this?). This feature 4572 allows the user to run test analyses with variations on the recipe 4573 without altering the analysis system. For example, it is possible to 4574 use a different flat-field set by specifying alternate rules for the 4575 flat-field selection in a recipe file. If it is necessary to run the 4576 P1-P3 analysis with the raw master flats, for example, the user simply 4577 defines that selection in the recipe file and submits the images of 4578 interest to P1 (or P2, etc), with the corresponding entry for the 4579 recipe. 4580 4581 The recipe file may also be used to specify alternative analysis paths 4582 and desitinations. For example, it is not necessary that all analysis 4583 stops with P4: the recipe file may be used to halt the analysis at P2 4584 or P3. In addition, the recipe file may be used to specify an 4585 alternative destination for the output results. For example, to 4586 generate the photometric flat-field correction frame from a collection 4587 of dithered images, the user may not want the photometry results in 4588 the main DVO database. By using the recipe to set an alternative DVO 4589 database target, and by specifying the use of the raw master flat 4590 rather than the corrected one, the analysis of the dithered images is 4591 kept isolated from the other photometry database results. The 4592 resulting photometry may be used to construct the new, corrected 4593 flat-field images, and the processing of the same images using the new 4594 flat-field images may be sent to the master DVO database. 4595 4596 \subsection{Basic Detrend Creation} 4597 4598 \begin{figure} 4599 \begin{center} 4600 \includegraphics[scale=0.85]{pics/ipptools.08.ps} 4601 \caption{\label{detrend} Detrend Creation Tasks} 4602 \end{center} 4603 \end{figure} 4604 4605 Figure~\ref{detrend} illustrates the tables needed for the generic 4606 detrend construction process, using the flat-field construction as an 4607 example. This diagram is somewhat more complex than the preceeding 4608 versions. In this diagram, both single jobs and multiple jobs are 4609 represented by the process elements (the blue ellipses). In some 4610 cases, more that one task will be needed to perform the function 4611 illustrated by a single process task. The complexity of this diagram 4612 is enhanced by the need for multiple iterations and both single chip 4613 and full mosaic processing. At the moment, the distinction between 4614 mosaic and single chip cameras is not specifically discussed. 4615 Finally, the triggers which initiate a specific detrend analysis are 4616 glossed over. 4617 4618 The detrend analysis is initiated by choosing a type of detrend image 4619 to be constructed and by specifying the criteria which will be used to 4620 select the input raw detrend frames for the construction. For 4621 example, these criteria could specify that all twilight flat images 4622 over a certain period of days, perhaps with restrictions on the flux 4623 levels or the time-from-sunset of the images. The detrend analysis 4624 run is given an ID (det ID) which will also be used to identify the 4625 resulting master detrend frame. 4626 4627 Given the definition of a master detrend run, the input exposures are 4628 selected from the raw detrend exposure table, and written to the input 4629 detrend exposure table. In the next step, the corresponding image 4630 files are selected from the table of raw image files. Since there 4631 will be a different set of input raw images for each attempt at 4632 creating a master detrend image, and since any given attempt may use 4633 some of the same input images as any other attempt, a separate table 4634 of input raw images is constructed. 4635 4636 Each of the input raw images may be pre-processed before it may be 4637 used to construct the detrend frame. For example, the input 4638 flat-field images should (probably) be dark- and bias-corrected before 4639 they are stacked. The information about these input processed images 4640 is written to the input images table. If no processing is needed, 4641 this step simply copies the appropriate information to the table, and 4642 points back to the raw image, rather than a processed version. 4643 4644 The input processed images are combined (stacked) to create a master 4645 detrend image for the particular data element defined by the image 4646 class (chip/cell/fpa). At this stage, not all input images should 4647 necessarily be included in the stack. If residual statistics have 4648 been measured for the input images (say, using a prior stack), then 4649 some of the input image may be excluded. The table of residual images 4650 is used to guide this process. The information describing the 4651 resulting master image is written to the master images table. 4652 4653 The statistics of the master detrend images must examined so that any 4654 necessary renormalizations may be performed. For example, after 4655 stacking the individual flat images, the resulting stacks must be 4656 renomalized to account for the different ranges of input image fluxes. 4657 This analysis is least-squares solution in which an appropriate scale 4658 is determined for each input exposure and a separate gain is 4659 determined for each of the chips or cells in the camera. This 4660 analysis can only performed after all image stacks (ie, for all chips) 4661 have been constructed. The resulting information is written to the 4662 table of master detrend frames. 4663 4664 Once the master detrend is constructed, the master detrend images may 4665 be used to construct residual images for each of the input images. 4666 These residual statistics, as well as the locations of the residual 4667 images and other related data products (jpeg thumbnails?) are written 4668 to the residual image table. Note the red arrow which by-passes the 4669 stack construction and merge steps and skips directly to the residual 4670 analysis. In some cases, it may be useful to have the input images 4671 confronted with an existing detrend image, and the resulting residual 4672 values used to guide the rest of the process. For example, in the 4673 flat-field analysis, applying an earlier flat can result in a very 4674 good first-pass rejection of poor input images. The logic to make 4675 this leap must be part of PanTasks, since each of the individual 4676 blocks represent complete processing jobs. 4677 4678 Finally, the residual statistics from the complete mosaic (all input 4679 images, all chips) are used to assess the quality of the newly 4680 constructed master detrend image, and to potentially modify the 4681 selection of input images. This latter process is performed by 4682 marking the state of the residual images from this iteration. The 4683 stacking process always examines the state information for the 4684 residual images from the previous iteration, if it exists, when 4685 constructing the master stack. Once a master detrend frame has been 4686 judged of high enough quality, the state of the entry for the frame in 4687 the master detrend frames table is set to an appropriate value to tell 4688 the other routines that this image should be used as a master detrend. 4689 The exact choice of which master detrend frame is used for a given 4690 science image depends on the recipe along with information such as the 4691 time period used or the conditions used. 4692 4693 Note that, although this discussion focuses on the construction of 4694 flat-field images, the same structure should be capable of 4695 constructing the biases, dark, fringes, etc. In some cases, as noted 4696 above, the `process' stage is a null operation. 4697 4698 \begin{figure} 4699 \begin{center} 4700 \includegraphics[scale=0.85]{pics/ipptools.09.ps} 4701 \caption{\label{detprocess} Detrend Creation : Process Tasks} 4702 \end{center} 4703 \end{figure} 4704 4705 \begin{figure} 4706 \begin{center} 4707 \includegraphics[scale=0.85]{pics/ipptools.10.ps} 4708 \caption{\label{detresid} Detrend Creation : Residual Tasks} 4709 \end{center} 4710 \end{figure} 4711 4712 \begin{figure} 4713 \begin{center} 4714 \includegraphics[scale=0.85]{pics/ipptools.11.ps} 4715 \caption{\label{detstack} Detrend Creation : Stack and Norm} 4716 \end{center} 4717 \end{figure} 3256 4718 3257 4719 \section{Interfaces}
Note:
See TracChangeset
for help on using the changeset viewer.
