Changeset 40028 for trunk/doc/release.2015/ps1.datasystem/datasystem.tex

Timestamp:

May 9, 2017, 11:03:29 AM (9 years ago)

Author:

eugene

Message:

updates to DVO and calibration

File:

• trunk/doc/release.2015/ps1.datasystem/datasystem.tex (modified) (9 diffs)

trunk/doc/release.2015/ps1.datasystem/datasystem.tex

@@ -1155 +1155 @@
 
 % overview
-DVO tracks three main classes of information: 1) properties of
+DVO tracks three main classes of information: 1) average properties of
 astronomical objects; 2) measurements of those objects (from which the
-properties are derived); 3) properties of image which provided some or
-all of the measuements.  Figure~\ref{fig:DVO_schema} illustrates the
-schematic relationship between these types of measurements.  
-
-In the most basic implementation, a collection of measurements from a
-set of images are loaded into DVO along with the metadata describing
-the images.  The latter includes properties such as the exposure time,
-airmass, filter, time \& date of the exposure, etc.  Critically, the
-image metadata includes an astrometric transformation relating the
-detection coordinate on the image to the coordinate on the sky.  As
-the collection of measurements are loaded into DVO, the software
-constructs astronomical objects based on those detections.  If
-images overlapped, multiple observations of the same astronomical
-object are grouped together.  Thus, a single DVO database will contain
-a one-to-many relationship between the images and the measurements and
-a many-to-one relationship between the measurements and the derived
-astronomical objects.
+average properties are derived); 3) properties of image which provided
+some or all of the measuements.  Figure~\ref{fig:DVO_schema}
+illustrates the schematic relationship between these types of
+measurements.
+
+In the most basic implementation, a collection of measurements for
+detections from a set of images is loaded into DVO along with the
+metadata describing the images.  The latter includes properties such
+as the exposure time, airmass, filter, time \& date of the exposure,
+etc.  Critically, the image metadata includes an astrometric
+transformation relating the detection coordinate on the image to the
+coordinate on the sky.  As the collection of measurements are loaded
+into DVO, the software constructs astronomical objects based on those
+detections.  If images overlap, multiple observations of the same
+astronomical object are grouped together.  Thus, a single DVO database
+will contain a one-to-many relationship between the images and the
+measurements and a many-to-one relationship between the measurements
+and the derived astronomical objects.
 
 \subsubsection{DVO Schema}
@@ -1182 +1183 @@
 astronomical objects; those which store information about individual
 measurements; those which store information about the images; those
-which store supporting information.
+which store supporting information (metadata).
+
+\subsubsubsection{Photcodes}
+
+% photcodes
+DVO has a special metadata table called \ippdbcolumn{photcode} which
+identifies the photometry filter systems.  Entries in this table are
+used to identify the source of measurements and images.  Each row in
+the \ippdbcolumn{photcode} table includes a \ippdbcolumn{photcode}
+name, a unique numerical ID, and information about that photometry
+system.  
+
+There are 3 classes of photcodes defined within the DVO system.  One
+class of photcodes define the filter systems for the average
+photometry measurements; these are called \ippmisc{SEC} photcodes.  A
+second class of photcode is associated with measurements from a
+specific camera for which image metadata is available are called
+\ippmisc{DEP} photcodes.  There are also those measurements which come
+from external data sources for which DVO does not have any information
+to determine a calibration (e.g., instrumental magnitudes and detector
+coordinates).  These are measurements are reference values and are
+assigned \ippmisc{REF} photcodes.
+
+The names for \ippmisc{SEC} photcodes are the names of filter systems,
+such as $g,r,i$ or $J,H,K$.  For \ippmisc{DEP} and \ippmisc{REF}
+photcodes, the names are constructed from the name of a camera or
+telescope (e.g., GPC1 or 2MASS), the name (or short-hand name) of a
+filter (e.g., \gps{}), and an identifier for the detector, if not
+unique (e.g., XY01 for a GPC1 OTA).  
+
+Additional information is associated with each photcode to define the
+nominal zero point and airmass slope, as well as color trends to
+transform a measurement in the specific photcode to a common system.
+For example, a \ippmisc{DEP} photcode GPC1.g.X01 would have the
+nominal zero point (25.XX) and airmass term (0.14).  The structures
+allow for individual chips to have different color terms to bring them
+to a common filter system.  
+
+Beyond the basic use, DVO has the ability to accept data from other
+kinds of data sources in which measurements are not clearly associated
+with specific images.  DVO ingest methods are defined for several
+large-scale surveys for which the published data represent average
+properties derived from multiple measurements, and for which the
+measurement-to-image relationship is not provided.  Ingests methods
+have been defined for example for 2MASS, WISE, Gaia, USNO-B.  In each
+of these cases, the astrometric and photometric measurements are
+stored in the \ippdbtable{Measure} table, with the data source
+identified by the photcode of the measurement.
 
 \subsubsubsection{Measurement Tables}
 
-The individual measurements of the astronomical objects are carried in
-the table \ippdbtable{Measure}.  For measurements from PS1 in the PV3
-/ DR1 database, this would be values determined by \ippprog{psphot}
-for each \ippstage{chip}, \ippstage{warp}, or \ippstage{stack} stage
-image.  Measurements for other cameras processed by the IPP may also
-be included similarly in a DVO database.  Measurements from other
-sources, such as SDSS, 2MASS, or WISE, can also be included in this
-table (see \S\ref{sec:other.photometry}.
+In most cases, the individual measurements of the astronomical objects
+are carried in the table \ippdbtable{Measure}.  For measurements from
+PS1 in the PV3 / DR1 database, this would consist of values determined
+by \ippprog{psphot} for each \ippstage{chip}, \ippstage{warp}, or
+\ippstage{stack} stage image.  Measurements for other cameras
+processed by the IPP may also be included similarly in a DVO database.
+Measurements from other sources, such as SDSS, 2MASS, or WISE, can
+also be included in this table (see \S\ref{sec:other.photometry}.
 
 The \ippdbtable{Measure} table includes the instrumental magnitudes
@@ -1208 +1256 @@
 discussed below) and the astrometrically calibrated position.
 Astrometric offsets for several systematic corrections discussed below
-are also defined for each measurement.  Since stacks and forced warp
-photometry may have non-significant values, the table is somewhat
-de-normalized in that it also carries instrumental flux values for the
+are also defined for each measurement.  Photometry from chip, warp,
+and stack are all placed in the same table with photcodes
+distinguishing the source \note{show example of stack and warp
+  photcodes}.  Since stacks and forced warp fluxes may have
+non-significant values, the table is somewhat de-normalized: it also
+carries both magnitudes as well as instrumental flux values for the
 PSF, aperture, and Kron photometry.  In this case, we have chosen to
 trade storage space for computing time.
-
-In the \ippdbtable{Measure} table, there are three fields which
-provide two independent links from the specific measurement to the
-associated object: \ippdbtable{Measure}.\ippdbcolumn{catID} specifies
-the spatial partition to which the measurement belongs;
-\ippdbtable{Measure}.\ippdbcolumn{objID} specifies to which entry in
-the \ippdbtable{Average} table the measurement belongs.  These two 32
-bit fields can thus be combined into a single 64 bit ID unique for all
-objects in the database.  \note{PSPS IDs} In addition, the field
-\ippdbtable{Measure}.\ippdbcolumn{averef} specifies the row number in
-the \ippdbtable{Average} table of the associated object.  The
-\ippdbtable{Measure} table may be unsorted, in which case it is slow
-to find the measurements associated with a specific object (a full
-table scan is required).  After the table is sorted and indexed, the
-\ippdbcolumn{Measure} rows for a given object are grouped together.
-In this case, the fields
-\ippdbtable{Average}.\ippdbcolumn{measureOffset} and
-\ippdbcolumn{Average}.\ippdbcolumn{Nmeasure} define an index for the
-code to jump to the list of measurements for a single object.  The
-field \ippdbtable{Measure}.\ippdbcolumn{imageID} defines the link from
-the measurement to the image which supplied the measurement.
 
 For the warp images, we also measure the weak lensing KSB parameters
@@ -1250 +1280 @@
   the Lensing to Measure indexing}
 
-\subsubsubsection{Astronomical Objects}
+\subsubsubsection{Object Tables}
 
 % object -> detection
@@ -1328 +1358 @@
 in our analysis of the astrometry \citep[][see]{magnier2017a}.
 
-\subsubsubsection{Other Tables} 
+\note{move the next paragraph after Average is defined?}
+
+In the \ippdbtable{Measure} table, there are three fields which
+provide two independent links from the specific measurement to the
+associated object: \ippdbtable{Measure}.\ippdbcolumn{catID} specifies
+the spatial partition to which the measurement belongs;
+\ippdbtable{Measure}.\ippdbcolumn{objID} specifies to which entry in
+the \ippdbtable{Average} table the measurement belongs.  These two 32
+bit fields can thus be combined into a single 64 bit ID unique for all
+objects in the database.  \note{PSPS IDs} In addition, the field
+\ippdbtable{Measure}.\ippdbcolumn{averef} specifies the row number in
+the \ippdbtable{Average} table of the associated object.  The
+\ippdbtable{Measure} table may be unsorted, in which case it is slow
+to find the measurements associated with a specific object (a full
+table scan is required).  After the table is sorted and indexed, the
+\ippdbcolumn{Measure} rows for a given object are grouped together.
+In this case, the fields
+\ippdbtable{Average}.\ippdbcolumn{measureOffset} and
+\ippdbcolumn{Average}.\ippdbcolumn{Nmeasure} define an index for the
+code to jump to the list of measurements for a single object.  The
+field \ippdbtable{Measure}.\ippdbcolumn{imageID} defines the link from
+the measurement to the image which supplied the measurement.
+
+\subsubsubsection{Image Tables} 
 
 Measurements which are loaded into DVO may be associated with a
@@ -1356 +1409 @@
 %% \ippdbtable{Measure} and similar tables, 
 
+\subsubsubsection{Other Tables} 
+
+Are there other tables to discuss?
+
 Other tables are used to track information used by the calibration
 system.  This includes the complete set of flat-field corrections
@@ -1361 +1418 @@
 flat-field corrections determined by the astrometry calibration
 analysis \citep[][see]{magnier2017a}.
-
-\subsubsection{MISC INFO TO BE RE-ORGed}
-
-Beyond that basic use, DVO has the ability to accept data from other
-kinds of data sources in which measurements are not clearly associated
-with specific images.  DVO ingest methods are defined for several
-large-scale surveys for which the published data represent average
-properties derived from multiple measurements, and for which the
-measurement-to-image relationship is not provided.  Ingets methods
-have been defined for example for 2MASS, WISE, Gaia, USNO-B.  In each
-of these cases, the astrometric and photometric measurements are
-stored in the \table{Measure} table, with the data source identified
-by the photcode of the measurement.
-
-% photcodes
-Detections in DVO have a special piece of metadata called the
-\ippdbcolumn{photcode} which identifies the source of the measurement.
-A \ippdbcolumn{photcode} has a name which in general consists of the
-name of the camera or telescope (e.g., GPC1 or 2MASS), the name (or
-short-hand name) of the filter used for the measurement (e.g.,
-\gps{}), and an identifier for the detector, if not unique (e.g., XY01
-for a GPC1 OTA).  Along with each name, there is a numerical value for
-the photcode.  A table within the DVO system, \ippdbtable{Photcode},
-lists the photcodes and defines a number of additional pieces of
-information for each photcode.  These include the nominal zero point
-and airmass slope, as well as color trends to transform a measurement
-in the specific photcode to a common system.  There are 3 classes of
-photcodes defined within the DVO system.  Those photcodes associated
-with detections from an image loaded into the database system are
-called \ippmisc{DEP} photcodes.  There are also photcodes associated with
-the average photometry values, called \ippmisc{SEC} photcodes.  There are
-also those measurements which come from external data sources for
-which DVO does not have any information to determine a calibration
-(e.g., instrumental magnitudes and detector coordinates).  These are
-measurements are reference values and are assigned \ippmisc{REF}
-photcodes.
-
-\subsubsection{DVO Data Storage}
-
-% FITS table + compression
-In the implementation of DVO used for the PV3 calibration analysis,
-the database tables are stored on disk using binary FITS tables.  Each
-type of database table is stored as a separate file, or a collection
-of files for table which are spatially partitioned.  The binary FITS
-tables are compressed using the (to date) experimental FITS binary
-table compression strategy outlined by \note{REF}.  Table compression
-is in general an option in DVO; for the PV3 database, the large data
-volume (70TB compressed) drove the decision to compress the tables.
-
-% FITS table compression details
-The FITS binary table compression scheme uses a strategy similar to
-that used for FITS image compression (\note{REF}).  The binary tabular
-data is compressed and stored in the 'HEAP' section of the FITS table
-extension, with pointers to the compressed data stored in the regular
-data section.  Each column in the FITS table is compressed as one (or
-more) blocks.  The standard header keywords which describe the data
-column format (e.g., TFORM1) are replaced with keywords which describe
-the location and size of the compressed data in the HEAP section; the
-information about the uncompressed data is moved to a keyword with 'Z'
-prepended (e.g., ZFORM1) and an additional field is added to define
-the compression algorithm (e.g., ZCTYP1).  The column names (e.g.,
-TTYPE1) and units (e.g., TUNIT1) are retained in their original form.
-
-% FITS table compression details
-The compression algorithm can treat the entire column as a single
-block of data, or it may be broken into a number of chunks, each
-compressed in turn (this must be the same for all columns).
-Additional header information is added to describe the block sizes and
-infomation needed to describe the HEAP data section.  The compression
-algorithms currently defined consist of the GZIP, RICE, PLIO, and
-HCOMPRESS (REFS).  For GZIP, the compression algorithm may transpose
-the byte order before compression: for floating point data of a
-similiar dynamic range, this choice may allow for better compression
-as each byte in the 4 or 8 byte floating point value is more likely to
-be similar to the same byte in other rows than to the other bytes of
-the same row value.  This option is called \code{GZIP_2} in the FITS
-standard, as opposed to the standard order, \code{GZIP_1}.  The DVO
-system can be set to specify the compression options for each column
-in the tables.  In practice, we have chosen a default in which
-floating point numbers use \code{GZIP_2}, character strings use
-\code{GZIP_1}, integers use \code{RICE}.
 
 \subsubsection{Sky Partition}
@@ -1515 +1491 @@
 database on a reasonable timescale.
 
-\subsection{Addstar : DVO Ingest}
+\subsubsection{DVO Data Storage}
+
+% FITS table + compression
+In the implementation of DVO used for the PV3 calibration analysis,
+the database tables are stored on disk using binary FITS tables.  Each
+type of database table is stored as a separate file, or a collection
+of files for table which are spatially partitioned.  The binary FITS
+tables are compressed using the (to date) experimental FITS binary
+table compression strategy outlined by \note{REF}.  Table compression
+is in general an option in DVO; for the PV3 database, the large data
+volume (70TB compressed) drove the decision to compress the tables.
+
+% FITS table compression details
+The FITS binary table compression scheme uses a strategy similar to
+that used for FITS image compression (\note{REF}).  The binary tabular
+data is compressed and stored in the 'HEAP' section of the FITS table
+extension, with pointers to the compressed data stored in the regular
+data section.  Each column in the FITS table is compressed as one (or
+more) blocks.  The standard header keywords which describe the data
+column format (e.g., TFORM1) are replaced with keywords which describe
+the location and size of the compressed data in the HEAP section; the
+information about the uncompressed data is moved to a keyword with 'Z'
+prepended (e.g., ZFORM1) and an additional field is added to define
+the compression algorithm (e.g., ZCTYP1).  The column names (e.g.,
+TTYPE1) and units (e.g., TUNIT1) are retained in their original form.
+
+% FITS table compression details
+The compression algorithm can treat the entire column as a single
+block of data, or it may be broken into a number of chunks, each
+compressed in turn (this must be the same for all columns).
+Additional header information is added to describe the block sizes and
+infomation needed to describe the HEAP data section.  The compression
+algorithms currently defined consist of the GZIP, RICE, PLIO, and
+HCOMPRESS (REFS).  For GZIP, the compression algorithm may transpose
+the byte order before compression: for floating point data of a
+similiar dynamic range, this choice may allow for better compression
+as each byte in the 4 or 8 byte floating point value is more likely to
+be similar to the same byte in other rows than to the other bytes of
+the same row value.  This option is called \code{GZIP_2} in the FITS
+standard, as opposed to the standard order, \code{GZIP_1}.  The DVO
+system can be set to specify the compression options for each column
+in the tables.  In practice, we have chosen a default in which
+floating point numbers use \code{GZIP_2}, character strings use
+\code{GZIP_1}, integers use \code{RICE}.
+
+\subsubsection{Addstar : DVO Ingest}
 \label{sec:addstar}
-\note{CZW: This should be reviewed.}
 
 Upon completion of the processing of each stage, the results of the
-photometry analysis are isolated in a large number of individual
-catalogs, with little connection between the separate measurements of
-astronomical sources.  Unifying these measurements in a DVO database
-is the purpose of the \ippstage{addstar} processing.  The catalogs for
-the \ippstage{camera}, \ippstage{staticsky}, \ippstage{skycal},
-\ippstage{fullforce}, and \ippstage{diff} are processed in this
-fashion, although not every measurement in each catalog are included
-in the final DVO that is constructed.
-
-The construction of this final DVO is performed in a hierarchical
-process.  The individual catalogs are added to a \ippmisc{minidvo},
+photometry analysis are stored in a large number of individual catalog
+files as described in~\ref{XXX}.  The data from these files are loaded
+into a DVO database to define the astronomical objects and to allow
+for calibration analysis.  The program which loads the data into the
+DVO database is called \ippprog{addstar}, and is associated with the
+the \ippstage{addstar} processing stage.  The measurement catalogs
+generated by the \ippstage{camera}, \ippstage{staticsky},
+\ippstage{skycal}, \ippstage{fullforce}, and \ippstage{diff} stages
+are processed loaded into DVOs in this fashion, although not every
+measurement in each catalog are included in the master DVO that is
+constructed.  For a particular re-processing version, a single master
+DVO is constructed for the positive image stages (\ippstage{camera},
+\ippstage{staticsky}, \ippstage{skycal}, \ippstage{fullforce}) and a
+separate one is constructed for the difference image analysis stage
+results.
+
+The construction of the master DVO is performed in a hierarchical
+fashion.  The individual catalogs are added to a \ippmisc{minidvo},
 which is simply a DVO database defined over some subset of possible
-inputs.  These \ippmisc{minidvos} are then merged into larger
-databases to construct the final completely catalog.  \note{describe
-  database tables}
-
-Each catalog that is to be added to DVO has an entry created in the
-\ippdbtable{addRun} database table.  This entry notes which
+inputs.  These \ippmisc{minidvos} are then merged by stage into larger
+databases to construct a single master DVO database.  In the process,
+an intermediate master DVO for each stage is generated.  The
+\ippprog{dvomerge} program is responsible for merging two DVO
+databases together.  In the merge, astronomical objects are joined
+together using essentially the same rules as those used to associated
+detections into objects.  One exception: the match radius may be
+chosen to be a different size depending on the data source.  For
+example, when WISE data is merged with PS1 data, as discussed below, a
+match radius of 3 arcseconds is used due to the large beam-size of the
+WISE telescope.
+
+As of PV3, the process of merging \ippmisc{minidvos} is not highly
+automated, requiring manual attention.  The generation of the
+\ippmisc{minidvos} is automated and managed by the \ippstage{addstar}
+stage.  Each catalog that is to be added to DVO has an entry created
+in the \ippdbtable{addRun} database table.  This entry notes which
 \ippdbcolumn{stage} is the source of the catalog, and links to the
 appropriate database table with the \ippdbcolumn{stage_id} field.  As
 some stages, such as the \ippstage{diff} stage, create more than a
-single catalog, multiple entries with the \ippdbcolumn{stage_id} are
-created, with the \ippdbcolumn{stage_extra1} field containing an
-index to the individual components.  The catalog specified by the
-entry is added to the target \ippmisc{minidvo} by the
-\ippprog{addstar} program, \note{describe what's done?}.  When this
+single catalog for a single exposure, multiple entries with the
+\ippdbcolumn{stage_id} are created, with the
+\ippdbcolumn{stage_extra1} field containing an index to the individual
+components.  The catalog specified by the entry is added to the target
+\ippmisc{minidvo} by the \ippprog{addstar} program, with object
+constructed as described above (\S~\ref{sec:object}).  When this
 completes, an entry containing the statistics of the job is added to
 the \ippdbtable{addProcessedExp} table.
 
+After the master DVO is contructed containing the PS1 data, data from
+other sources are also added to the database.  For the PV3 DVO
+database, data was added from 2MASS, WISE, Gaia, and Tycho.  These
+external data sources are added by first generating a DVO database
+containing just the particular data source, then using the same DVO
+merging method to merge the external data DVO into the PS1 master.  
+
 \subsection{Calibration Operations}
 \label{sec:calibration}
+
+Once the master DVO database has been constructed, high-quality
+astrometric and photometric calibrations can be calculated.  The
+details of the calibration analysis are discussed in
+\cite{Magnier2017c}.  We present a brief summary here.
+
+Astrometric calibration consists of measuring and correcting
+systematic structures along with improved calibration of the
+transformations from chip to focal plane coordinates based on relative
+astrometry.  These steps are performed iteratively.  First, the
+relative astrometry analysis generates an improved solution without
+correction for systematic effects.  Next, systematic effects are
+measured by querying the DVO database to determine the residual
+astrometric error as a function of some parameters.  In the case of
+the PV3 astrometry analysis, systematic errors have been determined as
+a function of position in the camera (essentially an astrometric
+flat-field correction), as a function of the brightness of the star
+(the so-called Koppenh\"offer effect, see~\ref{Magnier2017c}), and as
+a function of airmass and color (Differential chromatic refraction).
+Once the systematic errors have been measured, they are applied back
+to the measurements in the database.  Within the DVO
+\ippdbtable{Measure} table, the different types of systematic effects
+are included as separate offsets (in chip pixel coordinates) for each
+measurement.  A single ``corrected'' version of the chip pixel
+coordinates is stored in which the systematic offsets are combined
+with the raw pixel coordinates for each measurement.  After the
+systematic effects have been applied to the database, relative
+astrometry is again performed this time using the corrected positions.
+
+Photometric calibration involved the efforts of external collaborative
+analysis.
+
+\begin{verbatim}
+* data goes to harvard
+* eddie determines the zero points for photometric data
+* zero points are returned to ifa
+* zero points are applied to the DVO
+* systematic errors are measured (high-resolution flat-field)
+* applied back to DVO
+* relative photometry measured for non-photometric data
+\end{verbatim}
 
 \subsection{IPP to PSPS}
 \label{sec:ipp2psps}
 \note{Default to pointing to Flewelling et al 2017?}
+
+\begin{verbatim}
+\end{verbatim}
 
 \subsection{PSPS Load and Merge}
@@ -1779 +1869 @@
 
 \begin{figure}
- \begin{center}
+\begin{center}
 \begin{verbatim}
 task       example.static.task