Index: /trunk/doc/release.2015/ps1.datasystem/datasystem.tex =================================================================== --- /trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 40027) +++ /trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 40028) @@ -1155,23 +1155,24 @@ % overview -DVO tracks three main classes of information: 1) properties of +DVO tracks three main classes of information: 1) average properties of astronomical objects; 2) measurements of those objects (from which the -properties are derived); 3) properties of image which provided some or -all of the measuements. Figure~\ref{fig:DVO_schema} illustrates the -schematic relationship between these types of measurements. - -In the most basic implementation, a collection of measurements from a -set of images are loaded into DVO along with the metadata describing -the images. The latter includes properties such as the exposure time, -airmass, filter, time \& date of the exposure, etc. Critically, the -image metadata includes an astrometric transformation relating the -detection coordinate on the image to the coordinate on the sky. As -the collection of measurements are loaded into DVO, the software -constructs astronomical objects based on those detections. If -images overlapped, multiple observations of the same astronomical -object are grouped together. Thus, a single DVO database will contain -a one-to-many relationship between the images and the measurements and -a many-to-one relationship between the measurements and the derived -astronomical objects. +average properties are derived); 3) properties of image which provided +some or all of the measuements. Figure~\ref{fig:DVO_schema} +illustrates the schematic relationship between these types of +measurements. + +In the most basic implementation, a collection of measurements for +detections from a set of images is loaded into DVO along with the +metadata describing the images. The latter includes properties such +as the exposure time, airmass, filter, time \& date of the exposure, +etc. Critically, the image metadata includes an astrometric +transformation relating the detection coordinate on the image to the +coordinate on the sky. As the collection of measurements are loaded +into DVO, the software constructs astronomical objects based on those +detections. If images overlap, multiple observations of the same +astronomical object are grouped together. Thus, a single DVO database +will contain a one-to-many relationship between the images and the +measurements and a many-to-one relationship between the measurements +and the derived astronomical objects. \subsubsection{DVO Schema} @@ -1182,16 +1183,63 @@ astronomical objects; those which store information about individual measurements; those which store information about the images; those -which store supporting information. +which store supporting information (metadata). + +\subsubsubsection{Photcodes} + +% photcodes +DVO has a special metadata table called \ippdbcolumn{photcode} which +identifies the photometry filter systems. Entries in this table are +used to identify the source of measurements and images. Each row in +the \ippdbcolumn{photcode} table includes a \ippdbcolumn{photcode} +name, a unique numerical ID, and information about that photometry +system. + +There are 3 classes of photcodes defined within the DVO system. One +class of photcodes define the filter systems for the average +photometry measurements; these are called \ippmisc{SEC} photcodes. A +second class of photcode is associated with measurements from a +specific camera for which image metadata is available are called +\ippmisc{DEP} photcodes. There are also those measurements which come +from external data sources for which DVO does not have any information +to determine a calibration (e.g., instrumental magnitudes and detector +coordinates). These are measurements are reference values and are +assigned \ippmisc{REF} photcodes. + +The names for \ippmisc{SEC} photcodes are the names of filter systems, +such as $g,r,i$ or $J,H,K$. For \ippmisc{DEP} and \ippmisc{REF} +photcodes, the names are constructed from the name of a camera or +telescope (e.g., GPC1 or 2MASS), the name (or short-hand name) of a +filter (e.g., \gps{}), and an identifier for the detector, if not +unique (e.g., XY01 for a GPC1 OTA). + +Additional information is associated with each photcode to define the +nominal zero point and airmass slope, as well as color trends to +transform a measurement in the specific photcode to a common system. +For example, a \ippmisc{DEP} photcode GPC1.g.X01 would have the +nominal zero point (25.XX) and airmass term (0.14). The structures +allow for individual chips to have different color terms to bring them +to a common filter system. + +Beyond the basic use, DVO has the ability to accept data from other +kinds of data sources in which measurements are not clearly associated +with specific images. DVO ingest methods are defined for several +large-scale surveys for which the published data represent average +properties derived from multiple measurements, and for which the +measurement-to-image relationship is not provided. Ingests methods +have been defined for example for 2MASS, WISE, Gaia, USNO-B. In each +of these cases, the astrometric and photometric measurements are +stored in the \ippdbtable{Measure} table, with the data source +identified by the photcode of the measurement. \subsubsubsection{Measurement Tables} -The individual measurements of the astronomical objects are carried in -the table \ippdbtable{Measure}. For measurements from PS1 in the PV3 -/ DR1 database, this would be values determined by \ippprog{psphot} -for each \ippstage{chip}, \ippstage{warp}, or \ippstage{stack} stage -image. Measurements for other cameras processed by the IPP may also -be included similarly in a DVO database. Measurements from other -sources, such as SDSS, 2MASS, or WISE, can also be included in this -table (see \S\ref{sec:other.photometry}. +In most cases, the individual measurements of the astronomical objects +are carried in the table \ippdbtable{Measure}. For measurements from +PS1 in the PV3 / DR1 database, this would consist of values determined +by \ippprog{psphot} for each \ippstage{chip}, \ippstage{warp}, or +\ippstage{stack} stage image. Measurements for other cameras +processed by the IPP may also be included similarly in a DVO database. +Measurements from other sources, such as SDSS, 2MASS, or WISE, can +also be included in this table (see \S\ref{sec:other.photometry}. The \ippdbtable{Measure} table includes the instrumental magnitudes @@ -1208,30 +1256,12 @@ discussed below) and the astrometrically calibrated position. Astrometric offsets for several systematic corrections discussed below -are also defined for each measurement. Since stacks and forced warp -photometry may have non-significant values, the table is somewhat -de-normalized in that it also carries instrumental flux values for the +are also defined for each measurement. Photometry from chip, warp, +and stack are all placed in the same table with photcodes +distinguishing the source \note{show example of stack and warp + photcodes}. Since stacks and forced warp fluxes may have +non-significant values, the table is somewhat de-normalized: it also +carries both magnitudes as well as instrumental flux values for the PSF, aperture, and Kron photometry. In this case, we have chosen to trade storage space for computing time. - -In the \ippdbtable{Measure} table, there are three fields which -provide two independent links from the specific measurement to the -associated object: \ippdbtable{Measure}.\ippdbcolumn{catID} specifies -the spatial partition to which the measurement belongs; -\ippdbtable{Measure}.\ippdbcolumn{objID} specifies to which entry in -the \ippdbtable{Average} table the measurement belongs. These two 32 -bit fields can thus be combined into a single 64 bit ID unique for all -objects in the database. \note{PSPS IDs} In addition, the field -\ippdbtable{Measure}.\ippdbcolumn{averef} specifies the row number in -the \ippdbtable{Average} table of the associated object. The -\ippdbtable{Measure} table may be unsorted, in which case it is slow -to find the measurements associated with a specific object (a full -table scan is required). After the table is sorted and indexed, the -\ippdbcolumn{Measure} rows for a given object are grouped together. -In this case, the fields -\ippdbtable{Average}.\ippdbcolumn{measureOffset} and -\ippdbcolumn{Average}.\ippdbcolumn{Nmeasure} define an index for the -code to jump to the list of measurements for a single object. The -field \ippdbtable{Measure}.\ippdbcolumn{imageID} defines the link from -the measurement to the image which supplied the measurement. For the warp images, we also measure the weak lensing KSB parameters @@ -1250,5 +1280,5 @@ the Lensing to Measure indexing} -\subsubsubsection{Astronomical Objects} +\subsubsubsection{Object Tables} % object -> detection @@ -1328,5 +1358,28 @@ in our analysis of the astrometry \citep[][see]{magnier2017a}. -\subsubsubsection{Other Tables} +\note{move the next paragraph after Average is defined?} + +In the \ippdbtable{Measure} table, there are three fields which +provide two independent links from the specific measurement to the +associated object: \ippdbtable{Measure}.\ippdbcolumn{catID} specifies +the spatial partition to which the measurement belongs; +\ippdbtable{Measure}.\ippdbcolumn{objID} specifies to which entry in +the \ippdbtable{Average} table the measurement belongs. These two 32 +bit fields can thus be combined into a single 64 bit ID unique for all +objects in the database. \note{PSPS IDs} In addition, the field +\ippdbtable{Measure}.\ippdbcolumn{averef} specifies the row number in +the \ippdbtable{Average} table of the associated object. The +\ippdbtable{Measure} table may be unsorted, in which case it is slow +to find the measurements associated with a specific object (a full +table scan is required). After the table is sorted and indexed, the +\ippdbcolumn{Measure} rows for a given object are grouped together. +In this case, the fields +\ippdbtable{Average}.\ippdbcolumn{measureOffset} and +\ippdbcolumn{Average}.\ippdbcolumn{Nmeasure} define an index for the +code to jump to the list of measurements for a single object. The +field \ippdbtable{Measure}.\ippdbcolumn{imageID} defines the link from +the measurement to the image which supplied the measurement. + +\subsubsubsection{Image Tables} Measurements which are loaded into DVO may be associated with a @@ -1356,4 +1409,8 @@ %% \ippdbtable{Measure} and similar tables, +\subsubsubsection{Other Tables} + +Are there other tables to discuss? + Other tables are used to track information used by the calibration system. This includes the complete set of flat-field corrections @@ -1361,85 +1418,4 @@ flat-field corrections determined by the astrometry calibration analysis \citep[][see]{magnier2017a}. - -\subsubsection{MISC INFO TO BE RE-ORGed} - -Beyond that basic use, DVO has the ability to accept data from other -kinds of data sources in which measurements are not clearly associated -with specific images. DVO ingest methods are defined for several -large-scale surveys for which the published data represent average -properties derived from multiple measurements, and for which the -measurement-to-image relationship is not provided. Ingets methods -have been defined for example for 2MASS, WISE, Gaia, USNO-B. In each -of these cases, the astrometric and photometric measurements are -stored in the \table{Measure} table, with the data source identified -by the photcode of the measurement. - -% photcodes -Detections in DVO have a special piece of metadata called the -\ippdbcolumn{photcode} which identifies the source of the measurement. -A \ippdbcolumn{photcode} has a name which in general consists of the -name of the camera or telescope (e.g., GPC1 or 2MASS), the name (or -short-hand name) of the filter used for the measurement (e.g., -\gps{}), and an identifier for the detector, if not unique (e.g., XY01 -for a GPC1 OTA). Along with each name, there is a numerical value for -the photcode. A table within the DVO system, \ippdbtable{Photcode}, -lists the photcodes and defines a number of additional pieces of -information for each photcode. These include the nominal zero point -and airmass slope, as well as color trends to transform a measurement -in the specific photcode to a common system. There are 3 classes of -photcodes defined within the DVO system. Those photcodes associated -with detections from an image loaded into the database system are -called \ippmisc{DEP} photcodes. There are also photcodes associated with -the average photometry values, called \ippmisc{SEC} photcodes. There are -also those measurements which come from external data sources for -which DVO does not have any information to determine a calibration -(e.g., instrumental magnitudes and detector coordinates). These are -measurements are reference values and are assigned \ippmisc{REF} -photcodes. - -\subsubsection{DVO Data Storage} - -% FITS table + compression -In the implementation of DVO used for the PV3 calibration analysis, -the database tables are stored on disk using binary FITS tables. Each -type of database table is stored as a separate file, or a collection -of files for table which are spatially partitioned. The binary FITS -tables are compressed using the (to date) experimental FITS binary -table compression strategy outlined by \note{REF}. Table compression -is in general an option in DVO; for the PV3 database, the large data -volume (70TB compressed) drove the decision to compress the tables. - -% FITS table compression details -The FITS binary table compression scheme uses a strategy similar to -that used for FITS image compression (\note{REF}). The binary tabular -data is compressed and stored in the 'HEAP' section of the FITS table -extension, with pointers to the compressed data stored in the regular -data section. Each column in the FITS table is compressed as one (or -more) blocks. The standard header keywords which describe the data -column format (e.g., TFORM1) are replaced with keywords which describe -the location and size of the compressed data in the HEAP section; the -information about the uncompressed data is moved to a keyword with 'Z' -prepended (e.g., ZFORM1) and an additional field is added to define -the compression algorithm (e.g., ZCTYP1). The column names (e.g., -TTYPE1) and units (e.g., TUNIT1) are retained in their original form. - -% FITS table compression details -The compression algorithm can treat the entire column as a single -block of data, or it may be broken into a number of chunks, each -compressed in turn (this must be the same for all columns). -Additional header information is added to describe the block sizes and -infomation needed to describe the HEAP data section. The compression -algorithms currently defined consist of the GZIP, RICE, PLIO, and -HCOMPRESS (REFS). For GZIP, the compression algorithm may transpose -the byte order before compression: for floating point data of a -similiar dynamic range, this choice may allow for better compression -as each byte in the 4 or 8 byte floating point value is more likely to -be similar to the same byte in other rows than to the other bytes of -the same row value. This option is called \code{GZIP_2} in the FITS -standard, as opposed to the standard order, \code{GZIP_1}. The DVO -system can be set to specify the compression options for each column -in the tables. In practice, we have chosen a default in which -floating point numbers use \code{GZIP_2}, character strings use -\code{GZIP_1}, integers use \code{RICE}. \subsubsection{Sky Partition} @@ -1515,44 +1491,158 @@ database on a reasonable timescale. -\subsection{Addstar : DVO Ingest} +\subsubsection{DVO Data Storage} + +% FITS table + compression +In the implementation of DVO used for the PV3 calibration analysis, +the database tables are stored on disk using binary FITS tables. Each +type of database table is stored as a separate file, or a collection +of files for table which are spatially partitioned. The binary FITS +tables are compressed using the (to date) experimental FITS binary +table compression strategy outlined by \note{REF}. Table compression +is in general an option in DVO; for the PV3 database, the large data +volume (70TB compressed) drove the decision to compress the tables. + +% FITS table compression details +The FITS binary table compression scheme uses a strategy similar to +that used for FITS image compression (\note{REF}). The binary tabular +data is compressed and stored in the 'HEAP' section of the FITS table +extension, with pointers to the compressed data stored in the regular +data section. Each column in the FITS table is compressed as one (or +more) blocks. The standard header keywords which describe the data +column format (e.g., TFORM1) are replaced with keywords which describe +the location and size of the compressed data in the HEAP section; the +information about the uncompressed data is moved to a keyword with 'Z' +prepended (e.g., ZFORM1) and an additional field is added to define +the compression algorithm (e.g., ZCTYP1). The column names (e.g., +TTYPE1) and units (e.g., TUNIT1) are retained in their original form. + +% FITS table compression details +The compression algorithm can treat the entire column as a single +block of data, or it may be broken into a number of chunks, each +compressed in turn (this must be the same for all columns). +Additional header information is added to describe the block sizes and +infomation needed to describe the HEAP data section. The compression +algorithms currently defined consist of the GZIP, RICE, PLIO, and +HCOMPRESS (REFS). For GZIP, the compression algorithm may transpose +the byte order before compression: for floating point data of a +similiar dynamic range, this choice may allow for better compression +as each byte in the 4 or 8 byte floating point value is more likely to +be similar to the same byte in other rows than to the other bytes of +the same row value. This option is called \code{GZIP_2} in the FITS +standard, as opposed to the standard order, \code{GZIP_1}. The DVO +system can be set to specify the compression options for each column +in the tables. In practice, we have chosen a default in which +floating point numbers use \code{GZIP_2}, character strings use +\code{GZIP_1}, integers use \code{RICE}. + +\subsubsection{Addstar : DVO Ingest} \label{sec:addstar} -\note{CZW: This should be reviewed.} Upon completion of the processing of each stage, the results of the -photometry analysis are isolated in a large number of individual -catalogs, with little connection between the separate measurements of -astronomical sources. Unifying these measurements in a DVO database -is the purpose of the \ippstage{addstar} processing. The catalogs for -the \ippstage{camera}, \ippstage{staticsky}, \ippstage{skycal}, -\ippstage{fullforce}, and \ippstage{diff} are processed in this -fashion, although not every measurement in each catalog are included -in the final DVO that is constructed. - -The construction of this final DVO is performed in a hierarchical -process. The individual catalogs are added to a \ippmisc{minidvo}, +photometry analysis are stored in a large number of individual catalog +files as described in~\ref{XXX}. The data from these files are loaded +into a DVO database to define the astronomical objects and to allow +for calibration analysis. The program which loads the data into the +DVO database is called \ippprog{addstar}, and is associated with the +the \ippstage{addstar} processing stage. The measurement catalogs +generated by the \ippstage{camera}, \ippstage{staticsky}, +\ippstage{skycal}, \ippstage{fullforce}, and \ippstage{diff} stages +are processed loaded into DVOs in this fashion, although not every +measurement in each catalog are included in the master DVO that is +constructed. For a particular re-processing version, a single master +DVO is constructed for the positive image stages (\ippstage{camera}, +\ippstage{staticsky}, \ippstage{skycal}, \ippstage{fullforce}) and a +separate one is constructed for the difference image analysis stage +results. + +The construction of the master DVO is performed in a hierarchical +fashion. The individual catalogs are added to a \ippmisc{minidvo}, which is simply a DVO database defined over some subset of possible -inputs. These \ippmisc{minidvos} are then merged into larger -databases to construct the final completely catalog. \note{describe - database tables} - -Each catalog that is to be added to DVO has an entry created in the -\ippdbtable{addRun} database table. This entry notes which +inputs. These \ippmisc{minidvos} are then merged by stage into larger +databases to construct a single master DVO database. In the process, +an intermediate master DVO for each stage is generated. The +\ippprog{dvomerge} program is responsible for merging two DVO +databases together. In the merge, astronomical objects are joined +together using essentially the same rules as those used to associated +detections into objects. One exception: the match radius may be +chosen to be a different size depending on the data source. For +example, when WISE data is merged with PS1 data, as discussed below, a +match radius of 3 arcseconds is used due to the large beam-size of the +WISE telescope. + +As of PV3, the process of merging \ippmisc{minidvos} is not highly +automated, requiring manual attention. The generation of the +\ippmisc{minidvos} is automated and managed by the \ippstage{addstar} +stage. Each catalog that is to be added to DVO has an entry created +in the \ippdbtable{addRun} database table. This entry notes which \ippdbcolumn{stage} is the source of the catalog, and links to the appropriate database table with the \ippdbcolumn{stage_id} field. As some stages, such as the \ippstage{diff} stage, create more than a -single catalog, multiple entries with the \ippdbcolumn{stage_id} are -created, with the \ippdbcolumn{stage_extra1} field containing an -index to the individual components. The catalog specified by the -entry is added to the target \ippmisc{minidvo} by the -\ippprog{addstar} program, \note{describe what's done?}. When this +single catalog for a single exposure, multiple entries with the +\ippdbcolumn{stage_id} are created, with the +\ippdbcolumn{stage_extra1} field containing an index to the individual +components. The catalog specified by the entry is added to the target +\ippmisc{minidvo} by the \ippprog{addstar} program, with object +constructed as described above (\S~\ref{sec:object}). When this completes, an entry containing the statistics of the job is added to the \ippdbtable{addProcessedExp} table. +After the master DVO is contructed containing the PS1 data, data from +other sources are also added to the database. For the PV3 DVO +database, data was added from 2MASS, WISE, Gaia, and Tycho. These +external data sources are added by first generating a DVO database +containing just the particular data source, then using the same DVO +merging method to merge the external data DVO into the PS1 master. + \subsection{Calibration Operations} \label{sec:calibration} + +Once the master DVO database has been constructed, high-quality +astrometric and photometric calibrations can be calculated. The +details of the calibration analysis are discussed in +\cite{Magnier2017c}. We present a brief summary here. + +Astrometric calibration consists of measuring and correcting +systematic structures along with improved calibration of the +transformations from chip to focal plane coordinates based on relative +astrometry. These steps are performed iteratively. First, the +relative astrometry analysis generates an improved solution without +correction for systematic effects. Next, systematic effects are +measured by querying the DVO database to determine the residual +astrometric error as a function of some parameters. In the case of +the PV3 astrometry analysis, systematic errors have been determined as +a function of position in the camera (essentially an astrometric +flat-field correction), as a function of the brightness of the star +(the so-called Koppenh\"offer effect, see~\ref{Magnier2017c}), and as +a function of airmass and color (Differential chromatic refraction). +Once the systematic errors have been measured, they are applied back +to the measurements in the database. Within the DVO +\ippdbtable{Measure} table, the different types of systematic effects +are included as separate offsets (in chip pixel coordinates) for each +measurement. A single ``corrected'' version of the chip pixel +coordinates is stored in which the systematic offsets are combined +with the raw pixel coordinates for each measurement. After the +systematic effects have been applied to the database, relative +astrometry is again performed this time using the corrected positions. + +Photometric calibration involved the efforts of external collaborative +analysis. + +\begin{verbatim} +* data goes to harvard +* eddie determines the zero points for photometric data +* zero points are returned to ifa +* zero points are applied to the DVO +* systematic errors are measured (high-resolution flat-field) +* applied back to DVO +* relative photometry measured for non-photometric data +\end{verbatim} \subsection{IPP to PSPS} \label{sec:ipp2psps} \note{Default to pointing to Flewelling et al 2017?} + +\begin{verbatim} +\end{verbatim} \subsection{PSPS Load and Merge} @@ -1779,5 +1869,5 @@ \begin{figure} - \begin{center} +\begin{center} \begin{verbatim} task example.static.task