Index: trunk/doc/dvo/dvo.tex =================================================================== --- trunk/doc/dvo/dvo.tex (revision 6035) +++ trunk/doc/dvo/dvo.tex (revision 6055) @@ -19,5 +19,94 @@ \pagenumbering{arabic} -\subsection{Photometric systems and the DVO Photcodes} +\section{Overview} + +DVO, the Desktop Virtual Observatory, is a software system which +stores data related to astronomical objects derived from various +sources, and provides mechanisms to related multiple detections +together as astronomical objects. DVO deals with two related +concepts: {\em objects} and {\em detections}. The {\em objects} are +descriptions of astronomical objects while the {\em detections} are +the specific measurements of those objects, typically measured from +astronomical images. A collection of {\em detections} may be used to +derive average quantities which describe a particular {\em object}. A +third class of measurement to be considered are those supplied by +external references. Such measurements may be treated as {\em +detections}, with the caveat that access to the raw measurements and +metadata are usually unavailable: the reported measurements and errors +must be accepted as they are reported. + +DVO stores the collections of detections which were derived from +specific images. It provides a mechanism to determine the image from +which a specific detection was derived, and in conjunction with the +Image Server locate the corresponding data file. DVO also makes it +possible to extract all detections derived from a specific image and +to determine quantities such as the pixel coordinates of the detection +on the image. + +DVO also has the capability to associate multiple detections of a +specific object. Several major classes of objects will be present, +each of which must be handled correctly. DVO distinguished the +following types of objects. + +{\bf Stars, compact galaxies, and QSOs} will have nearly fixed +locations relative to other distant stars, with only small deviations +for individual measurements. The association between multiple +detections of such objects is made on the basis of their coincident +positions. DVO determines the average position of the object and the +deviations of the individual detections from that average on the basis +of the ensemble of individual detection. + +{\bf Solar System Objects} do not have a fixed location. Detections +of such objects are linked by their orbits, and depend on both the +position and the time of the image. DVO does not attempt to make this +link; this is the role of the MOPS system. However, it has the +ability to accept identifications made externally with specified +detections and to return the identifier of the moving object +associated with the specific detections. These associations also +include descriptive information such as the offset of the detection +from the predicted location of the detection based on the orbit. This +functionality is required to allow DVO to ignore known moving object +detections from other types of queries. + +{\bf High-proper-motion objects} in the general vicinity of the solar +system fall in between these first two classes of objects. Their +proper motion and parallax response is significant enough ($>0.2$ +arcsec in 1 year) that they are not well-described by an average +location and a collection of offsets. These objects are better +described by a distance and a proper motion vector. DVO provides the +association between the specific detections and an average object +which includes finite parallax and proper motion. + +{\bf Orphaned detections} are not associated with a specific +astronomical object of any of the above classes. Most of these will +be spurious (not representing real objects), some will be from solar +system objects for which orbits are not yet determined, some will be +from faint stars near the detection limits, and some will be from +short-term transients which have only been detected once. DVO +maintains these detections until they have been associated with one of +the objects above. DVO provides mechanisms by which individual +detections may be migrated back and forth between the orphan state and +association with an astronomical object. + +DVO stores the information about the detection, the related objects, +and the images which provided the measurements. For every detection, +DVO provides the mechanisms to link the detection back to the image +which supplied it. DVO also provides the capability to determine the +images containing a specific location but for which no detection was +made. The minimum set of information which must be carried for these +non-detections is the image and the associated object or orphan. + +DVO also stores the relationships between various +photometric systems and the evolution of that relationship. It +provides mechanisms to convert between the measured instrumental +magnitude of a detection with a specific filter, detector, and +telescope, and at a particular time and the implied magnitude in the +average Pan-STARRS photometry system, given a determined set of +calibrations. It also provides the capability to convert magnitudes +in one system to the magnitudes in another system; an example of such +a conversion is between the average Pan-STARRS filter systems and the +various reference systems appropriate for those filters. + +\section{Photometric systems and the DVO Photcodes} One of the major roles of DVO is to relate different photometric @@ -26,8 +115,8 @@ number of different detectors. We may have observations from different telescopes in similar filters. We may have reference data -related to some filter, but obtained and published by other -observers. We would like to related these measurements together in -optimal ways, making use of whatever information we have available. -DVO provides several mechanisms to enable these relationships. +related to some filter, but obtained and published by other observers. +We would like to related these measurements together in optimal ways, +making use of whatever information we have available. DVO provides +several mechanisms to enable these relationships. We identify three distinct types of photometry measurements within @@ -208,93 +297,4 @@ parameters. -\section{Overview} - -DVO, the Desktop Virtual Observatory, is a software system which -stores data related to astronomical objects derived from various -sources, and provides mechanisms to related multiple detections -together as astronomical objects. DVO deals with two related -concepts: {\em objects} and {\em detections}. The {\em objects} are -descriptions of astronomical objects while the {\em detections} are -the specific measurements of those objects, typically measured from -astronomical images. A collection of {\em detections} may be used to -derive average quantities which describe a particular {\em object}. A -third class of measurement to be considered are those supplied by -external references. Such measurements may be treated as {\em -detections}, with the caveat that access to the raw measurements and -metadata are usually unavailable: the reported measurements and errors -must be accepted as they are reported. - -DVO stores the collections of detections which were derived from -specific images. It provides a mechanism to determine the image from -which a specific detection was derived, and in conjunction with the -Image Server locate the corresponding data file. DVO also makes it -possible to extract all detections derived from a specific image and -to determine quantities such as the pixel coordinates of the detection -on the image. - -DVO also has the capability to associate multiple detections of a -specific object. Several major classes of objects will be present, -each of which must be handled correctly. DVO distinguished the -following types of objects. - -{\bf Stars, compact galaxies, and QSOs} will have nearly fixed -locations relative to other distant stars, with only small deviations -for individual measurements. The association between multiple -detections of such objects is made on the basis of their coincident -positions. DVO determines the average position of the object and the -deviations of the individual detections from that average on the basis -of the ensemble of individual detection. - -{\bf Solar System Objects} do not have a fixed location. Detections -of such objects are linked by their orbits, and depend on both the -position and the time of the image. DVO does not attempt to make this -link; this is the role of the MOPS system. However, it has the -ability to accept identifications made externally with specified -detections and to return the identifier of the moving object -associated with the specific detections. These associations also -include descriptive information such as the offset of the detection -from the predicted location of the detection based on the orbit. This -functionality is required to allow DVO to ignore known moving object -detections from other types of queries. - -{\bf High-proper-motion objects} in the general vicinity of the solar -system fall in between these first two classes of objects. Their -proper motion and parallax response is significant enough ($>0.2$ -arcsec in 1 year) that they are not well-described by an average -location and a collection of offsets. These objects are better -described by a distance and a proper motion vector. DVO provides the -association between the specific detections and an average object -which includes finite parallax and proper motion. - -{\bf Orphaned detections} are not associated with a specific -astronomical object of any of the above classes. Most of these will -be spurious (not representing real objects), some will be from solar -system objects for which orbits are not yet determined, some will be -from faint stars near the detection limits, and some will be from -short-term transients which have only been detected once. DVO -maintains these detections until they have been associated with one of -the objects above. DVO provides mechanisms by which individual -detections may be migrated back and forth between the orphan state and -association with an astronomical object. - -DVO stores the information about the detection, the related objects, -and the images which provided the measurements. For every detection, -DVO provides the mechanisms to link the detection back to the image -which supplied it. DVO also provides the capability to determine the -images containing a specific location but for which no detection was -made. The minimum set of information which must be carried for these -non-detections is the image and the associated object or orphan. - -DVO also stores the relationships between various -photometric systems and the evolution of that relationship. It -provides mechanisms to convert between the measured instrumental -magnitude of a detection with a specific filter, detector, and -telescope, and at a particular time and the implied magnitude in the -average Pan-STARRS photometry system, given a determined set of -calibrations. It also provides the capability to convert magnitudes -in one system to the magnitudes in another system; an example of such -a conversion is between the average Pan-STARRS filter systems and the -various reference systems appropriate for those filters. - \section{DVO Database Tables} @@ -499,4 +499,6 @@ data types and input/output methods without significant re-coding. +\tbd{DVO mysql table storage is not yet implemented} + \section{addstar : Insert Image \& Detection Set} @@ -505,6 +507,4 @@ \caption{\label{catalog} \small a figure } \end{figure} - -\tbd{fill out discussion of the addstar client/server implementation} One of the most basic operations needed by DVO is to insert a @@ -527,123 +527,167 @@ faint orphans. -\subsection{addstar -refs : Insert Reference Objects} - -This operation is very similar to the previous one. A collection of -reference objects are added to the database as a collection of -detections. The reference photometry should in general be given its -own photometry code. The reference data is different from the image -detection set because the associated image information is not -included. Thus, no corresponding images are added to the database. - -\section{relphot : Relative Photometry Analysis} +A wide range of options are available to addstar. These can be used +to modify the object matching rules, to reduce the number of tables +which are updated, to specify the output data format, and so forth. A +few options modify the behavoir in substantial ways, as discussed in +the two sections below. + +\tbd{flesh out discussion of the options} + +\subsection{Insert Reference Objects} + +\code{addstar -ref (filename)} + +This mode of addstar reads a text file and adds the listed objects to +the database as a reference photcode type. A collection of reference +objects are added to the database as a collection of detections. The +reference photometry should in general be given its own photometry +code. The reference data is different from the image detection set +because the associated image information is not included. Thus, no +corresponding images are added to the database. + +\subsection{Insert Catalog Objects} + +\code{addstar -cat (name) -region ra ra dec dec} + +In this mode, any of several all-sky or large-scale reference catalogs +are used for the input sources. The catalog objects are added to the +database as reference objects. The valid catalogs consist of 2MASS, +USNO, GSC. Tycho and USNO-B will be added shortly. Specific +photcode names are defined for each of these catalogs, and must be +appropriately requested and defined in the photcode table. The +optional region restriction limits the insert to a subset of the sky. +The user does not always want to add 50GB of 2MASS detections to any +DVO database... + +\subsection{Addstar Client/Server Interactions} + +DVO currently uses stand-alone programs which are run from the command +line (like addstar, or the programs listed below), or it works with +the interactive DVO shell, which allows the user to query portions of +the database. These programs all interact with the database tables +directly, making use of file locking to prevent conflicts. + +Unlike the other DVO programs (currently), it is possible to run +addstar as a client/server system. In this configuration, the program +\code{addstard} is launched to run in the background as a server. It +monitors a socket waiting for clients to contact it. The client +program, \code{addstarc} appears to the user identical to the +stand-alone addstar. However, rather than directly insert data into +the database, \code{addstarc} contacts the addstar server and sends it +the detections and associated image data (along with the information +about the user options). The daemons accepts the incoming data and +then loads this data into the database, just as the stand-alone +addstar does. + +The purpose of the addstar client/server design is three-fold. First, +the client can be used by processes to send data to the DVO database +and then immediately exit. The addstar loading process is one of the +more time-critical functions within the IPP. However, unlike the +other portions of the IPP, the addstar processes must operate in +serial, at least when they are updating the same portion of the sky +(or the image table). If the IPP analysis routines all needed to run +the stand-alone addstar program, they would eventually block waiting +for each addstar to complete, preventing other processing from +continuing. The addstar client / server model allows the processing +node to invoke the addstar client, sending the data to the addstar +server. The addstar server will then be the entity that manages the +serialization of the incoming data stream. The addstar server has two +threads which run in parallel. One thread monitors the socket and +accepts new data sets from addstar clients, adding the data to an +internal queue. The other thread pulls data off of the queue and +updates the database with the data. + +A second advantage of the client/server interaction is that only the +new detections need to be sent across the network. To update the +database, addstar must load the average objects for the region from +the database tables. In the stand-alone mode, the addstar program +loads this data via NFS across the network from whatever device stores +the addstar tables. In the client/server model, the addstar server +always runs locally on the machine which holds the database tables. +Thus, for the server, all database access is local disk access. + +The final advantage of the client/server model is that it enables the +parallel database model, which is not yet implemented as of Jan +2006. In this model, there are multiple addstar servers. Each one has +a fraction of the sky in the local tables. The identification of +which table is managed by this host/addstar server is stored in the +SkyRegion table. The addstar server simply accepts incoming +detections from the addstar clients. Any detections which it receives +which fall within the boundaries of tables that it manages are updated +as normal. The server then identifies the other addstar servers which +are responsible for the other detections. It then sends these +detections to those servers using the same socket communication used +by the addstar clients. The addstar server must also be ready accept +detections from other addstar servers. This relationship is +completely parallel, and any addstar client may send its data to any +addstar server, letting the servers hash out who owns what. The only +difficulty with this model is in handling sources near the boundaries +of the tables. Note that this issues exists whether those tables are +distributed across multiple machines or not. + +Addstar uses the following strategy to handle detections on the table +boundaries. Detections are first added to each table completely +ignoring the neighboring tables. A detection which is close to the +boundary may either be associated with an average object contained +within the table, or not. If it is, the detection is associated with +that average object. If not, a new average object is created at the +location of the detection. So far, this process is identical to the +behavoir in the middle of the table. One a longer time-scale, a +process is run which mediates the table boundaries. In this analysis, +the two neighboring tables are simultaneously examined. The border +region, in a strip wider than the correlation radius, is examined in +detail. If two objects within the border region fall within 2x the +correlation radius of each other, their individual detections are +re-examined. These detections are re-added to a temporary table which +encompases the overlap. the resulting objects will in general have +detections from either side of the boundary. The average objects are +kept within the table as normal, but the detections are allowed to +migrate between the tables to stay with their object. \tbd{this +boundary cleanup process is not implemented to date}. + +\section{Relphot : Relative Photometry Analysis} This operation uses the overlaps of images and multiple observations of the same objects to determine the relative photometry zero-points -for a collection of images. This is a task that wil be run much more +for a collection of images. This is a task that is run much more infrequently than the object insertion tasks. -\section{relphot} - -\begin{verbatim} - * load data - o images: match photcode and time range, reset flags - o measure: select subset matching restritions on: photcode, time, dM, Minst, Mag, dophot == 1 - * iterate to find Mcal, image.flags: - * write out modified Mcal values to image table - * write out modified Mcal values to measure table - * write out modified Mrel values to average table -\end{verbatim} - -relphot has two primary purposes: - -\begin{verbatim} - * calculate Mcal for images / measures - * calculate Mrel for stars -\end{verbatim} - -relphot can also be used to determine the mosaic grid used to generate photometrically corrected flats (-grid option). - -\subsection{data exclusion} - -relphot uses only a subset of the photometry data to calculate Mcal -and Mrel. In the first stage, calculation of the Mcal values, relphot -loads the photometry data from each relevant catalog and creates an -internal subset catalog with the function bcatalog, excluding some of -the irrelevant data. In addition, it uses flags to mark some of the -data as invalid for the processing. bcatalog exclusions - -\begin{verbatim} - * measure.photcode not equivalent to requested photcode - * measure.dophot != 1 - * measure.Mcat > MAG_LIM - * measure.dM > SIGMA_LIM - * measure.Minst out of range (ImagMin - ImagMax) [optional] - * measure.t out of range (TSTART, TSTOP) -\end{verbatim} - -flagged data -flagged image data - -images can be flagged by setting bits of image.code stars can be -flagged by setting bits of average.code measures can be flagged by -setting bits of measure.flag - -\begin{verbatim} -image.code -ID_IMAGE_NOCAL : ignore, irrelevant -ID_IMAGE_POOR : image measured bad -ID_IMAGE_SKIP : externally known bad dMcal > VALUE -FLAG_IMAGE_SCATTER clean_images fabs(Mcal) > VALUE -FLAG_IMAGE_ZEROPT clean_images dMcal > VALUE -FLAG_IMAGE_SCATTER clean_mosaics fabs(Mcal) > VALUE -FLAG_IMAGE_ZEROPT clean_mosaics mark_images does not seem to do -anything useful? -average.code Ngood < MEAS_TOOFEW setMrel Ngood < -MEAS_TOOFEW clean_measures ChiSq > STAR_CHISQ clean_stars dM > -STAR_SCATTER clean_stars average.code (STAR_BAD) is not saved by -relphot: it is set by clean_stars, clean_measures, and setMrel, but -not setMrelOutput. STAR_BAD should only be internal since it depends -on the photcode, but is not associated with a specific photcode in the -data. Just in case, it is reset to 0 in setMrelFinal. measure.flag X,Y -out of range setExclusions 3 sigma clipping clean_measures -\end{verbatim} - -setting Mrel final value - -setMrelFinal is used to set the final average.Mrel values. We do this -in 4 stages. In each stage, we set the Mrel values for stars which -have not already been set, based on the current exclusion settings. At -successive stages, we relax the exclusions, allowing the more spurious -objects to have a valid Mrel value to be set. In this loop, we -actually run setMrelOutput twice: once to get the approximate Mrel -value, then we flag the outlier measurements with -\code{clean_measure}, then we redetermine the Mrel values on this -basis, and mark the stars for exclusion from the next iteration. - -\begin{verbatim} - exclude on - photcode 0 1 2 3 - time range 0 1 2 3 - MEAS_POOR 0 1 2 3 - MEAS_TOOFEW 0 1 2 3 - dophot == 10 0 1 2 - inst mag 0 1 2 - dophot != 1,2 0 1 - ID_IMAGE_POOR 0 1 - ID_IMAGE_SKIP 0 1 - dophot != 1 0 - measure.dM 0 -\end{verbatim} - -for all relphot runs, Mrel is re-calculated, and measures are marked at least if they are outliers in mag or ccd area. setMrel.output needs to do a few things differently from setMrel: - -\begin{verbatim} - * set measure.Mcal (skipped in setMrel.basic) - * set average.Mrel if N < TOO_FEW (not STAR_BAD) (optional!) - * use MAX (stats.error, stats.sigma) (optionally) - * allow STAR_BAD? -\end{verbatim} - -\section{uniphot : Zero Point Analysis} +The relphot analysis is currently performed with a single Sky region +as the starting point. All images (or all chips from all mosaic +iamges) which overlap the sky region are identified in the image +table. This set of images are considered set A. Next, all skyregions +which are overlapped by all of these images are selected. Finally, +all additional images which overlapped the new regions only are +selected. These are considered as image set B. The image selections +are also restricted to images of a single, user-selected photcode. + +All of the objects and detections which are contributed by the images +in sample A are extracted from the average and measure tables. Only a +subset of the detections for which the S/N is greater than a +user-selected limit are kept. Other restrictions, such as time range +or instrumental magnitude ranges may also be specified. The +collection of average objects, their detections, and the images from +which they were derived now define a system of photometry equations. +In this system, every image has a calibration offset magnitude +($M_{cal}$), every object has an average magnitude in a relative +system ($M_{rel}$), and every detection of that object has a magnitude +defined by the equation $M = M_{rel} + M_{cal}$. The goal is to solve +for the values of $M_{ref}$ and $M_{cal}$. + +There are two points to note about this operation. First, the system +of equations is generally much too large to solve directly; we must +use an iterative technique to converge on a solution. Second, it is +important in the analysis to use robust averaging and identify +detections, stars, or images which are deviant in some way. These +should be marked and given set weight in the solution. These cases +may represent poorly measured objects (perhaps detections on or near a +bad column), variable stars, and images obtained in poor weather +conditions. + +Relphot can also be used to determine the mosaic grid used to generate +photometrically corrected flats (-grid option). + +\section{Uniphot : Zero Point Analysis} This operation uses the time history of relative photometry zero @@ -1482,2 +1526,85 @@ \end{itemize} + + +\subsection{Relphot data exclusion} + +relphot uses only a subset of the photometry data to calculate Mcal +and Mrel. In the first stage, calculation of the Mcal values, relphot +loads the photometry data from each relevant catalog and creates an +internal subset catalog with the function bcatalog, excluding some of +the irrelevant data. In addition, it uses flags to mark some of the +data as invalid for the processing. bcatalog exclusions + +\begin{verbatim} + * measure.photcode not equivalent to requested photcode + * measure.dophot != 1 + * measure.Mcat > MAG_LIM + * measure.dM > SIGMA_LIM + * measure.Minst out of range (ImagMin - ImagMax) [optional] + * measure.t out of range (TSTART, TSTOP) +\end{verbatim} + +flagged data +flagged image data + +images can be flagged by setting bits of image.code stars can be +flagged by setting bits of average.code measures can be flagged by +setting bits of measure.flag + +\begin{verbatim} +image.code +ID_IMAGE_NOCAL : ignore, irrelevant +ID_IMAGE_POOR : image measured bad +ID_IMAGE_SKIP : externally known bad dMcal > VALUE +FLAG_IMAGE_SCATTER clean_images fabs(Mcal) > VALUE +FLAG_IMAGE_ZEROPT clean_images dMcal > VALUE +FLAG_IMAGE_SCATTER clean_mosaics fabs(Mcal) > VALUE +FLAG_IMAGE_ZEROPT clean_mosaics mark_images does not seem to do +anything useful? +average.code Ngood < MEAS_TOOFEW setMrel Ngood < +MEAS_TOOFEW clean_measures ChiSq > STAR_CHISQ clean_stars dM > +STAR_SCATTER clean_stars average.code (STAR_BAD) is not saved by +relphot: it is set by clean_stars, clean_measures, and setMrel, but +not setMrelOutput. STAR_BAD should only be internal since it depends +on the photcode, but is not associated with a specific photcode in the +data. Just in case, it is reset to 0 in setMrelFinal. measure.flag X,Y +out of range setExclusions 3 sigma clipping clean_measures +\end{verbatim} + +setting Mrel final value + +setMrelFinal is used to set the final average.Mrel values. We do this +in 4 stages. In each stage, we set the Mrel values for stars which +have not already been set, based on the current exclusion settings. At +successive stages, we relax the exclusions, allowing the more spurious +objects to have a valid Mrel value to be set. In this loop, we +actually run setMrelOutput twice: once to get the approximate Mrel +value, then we flag the outlier measurements with +\code{clean_measure}, then we redetermine the Mrel values on this +basis, and mark the stars for exclusion from the next iteration. + +\begin{verbatim} + exclude on + photcode 0 1 2 3 + time range 0 1 2 3 + MEAS_POOR 0 1 2 3 + MEAS_TOOFEW 0 1 2 3 + dophot == 10 0 1 2 + inst mag 0 1 2 + dophot != 1,2 0 1 + ID_IMAGE_POOR 0 1 + ID_IMAGE_SKIP 0 1 + dophot != 1 0 + measure.dM 0 +\end{verbatim} + +for all relphot runs, Mrel is re-calculated, and measures are marked at least if they are outliers in mag or ccd area. setMrel.output needs to do a few things differently from setMrel: + +\begin{verbatim} + * set measure.Mcal (skipped in setMrel.basic) + * set average.Mrel if N < TOO_FEW (not STAR_BAD) (optional!) + * use MAX (stats.error, stats.sigma) (optionally) + * allow STAR_BAD? +\end{verbatim} +