Index: trunk/doc/release.2015/ps1.datasystem/datasystem.tex =================================================================== --- trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 40026) +++ trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 40027) @@ -1138,22 +1138,4 @@ \end{table} -\begin{verbatim} -DVO section outline or list of topics: - -* schema overview [ignoring sky partitioning] - * measurements -> objects - * images -* object definition -* tables in detail -* adding other data types (2mass, etc) -* storage details - * FITS - * compressed FITS -* sky partitioning -* parallelized DVO -* addstar / ingest process [stage -> this goes elsewhere] -* dvo shell description? -\end{verbatim} - \subsection{DVO} \label{sec:DVO} @@ -1193,4 +1175,6 @@ astronomical objects. +\subsubsection{DVO Schema} + Table~\ref{tab:DVO_schema} lists the full collection of database tables used by DVO. These tables fall into one of several classes: @@ -1200,14 +1184,71 @@ which store supporting information. -Beyond that basic use, DVO has the ability to accept data from other -kinds of data sources in which measurements are not clearly associated -with specific images. DVO ingest methods are defined for several -large-scale surveys for which the published data represent average -properties derived from multiple measurements, and for which the -measurement-to-image relationship is not provided. Ingets methods -have been defined for example for 2MASS, WISE, Gaia, USNO-B. In each -of these cases, the astrometric and photometric measurements are -stored in the \table{Measure} table, with the data source identified -by the photcode of the measurement. +\subsubsubsection{Measurement Tables} + +The individual measurements of the astronomical objects are carried in +the table \ippdbtable{Measure}. For measurements from PS1 in the PV3 +/ DR1 database, this would be values determined by \ippprog{psphot} +for each \ippstage{chip}, \ippstage{warp}, or \ippstage{stack} stage +image. Measurements for other cameras processed by the IPP may also +be included similarly in a DVO database. Measurements from other +sources, such as SDSS, 2MASS, or WISE, can also be included in this +table (see \S\ref{sec:other.photometry}. + +The \ippdbtable{Measure} table includes the instrumental magnitudes +for the PSF, aperture, and Kron photometry; raw position +(\ippdbcolumn{Xccd}, \ippdbcolumn{Yccd}) and second moments +(\ippdbcolumn{Mxx}, \ippdbcolumn{Myy}, \ippdbcolumn{Mxy}), along with +shape parameters of the PSF model at the position of the object +(\ippdbcolumn{FWx}, \ippdbcolumn{FWy}, \ippdbcolumn{theta}). Metadata +about the exposure that the measurement was derived from is also +included, such as the exposure time, the date \& time of the +observation, airmass, azimuth, and \ippdbcolumn{photcode} information +specifying the filter. The \ippdbtable{Measure} table also carries +the calibration magnitude offsets ($M_{\rm cal}$ and $M_{\rm flat}$, +discussed below) and the astrometrically calibrated position. +Astrometric offsets for several systematic corrections discussed below +are also defined for each measurement. Since stacks and forced warp +photometry may have non-significant values, the table is somewhat +de-normalized in that it also carries instrumental flux values for the +PSF, aperture, and Kron photometry. In this case, we have chosen to +trade storage space for computing time. + +In the \ippdbtable{Measure} table, there are three fields which +provide two independent links from the specific measurement to the +associated object: \ippdbtable{Measure}.\ippdbcolumn{catID} specifies +the spatial partition to which the measurement belongs; +\ippdbtable{Measure}.\ippdbcolumn{objID} specifies to which entry in +the \ippdbtable{Average} table the measurement belongs. These two 32 +bit fields can thus be combined into a single 64 bit ID unique for all +objects in the database. \note{PSPS IDs} In addition, the field +\ippdbtable{Measure}.\ippdbcolumn{averef} specifies the row number in +the \ippdbtable{Average} table of the associated object. The +\ippdbtable{Measure} table may be unsorted, in which case it is slow +to find the measurements associated with a specific object (a full +table scan is required). After the table is sorted and indexed, the +\ippdbcolumn{Measure} rows for a given object are grouped together. +In this case, the fields +\ippdbtable{Average}.\ippdbcolumn{measureOffset} and +\ippdbcolumn{Average}.\ippdbcolumn{Nmeasure} define an index for the +code to jump to the list of measurements for a single object. The +field \ippdbtable{Measure}.\ippdbcolumn{imageID} defines the link from +the measurement to the image which supplied the measurement. + +For the warp images, we also measure the weak lensing KSB parameters +related to the shear and smear tensors \citep{1995ApJ...449..460K}. +These measurements are stored in the \ippdbcolumn{Lensing} table, +along with the radial aperture fluxes for radii numbers 5, 6, \& 7 +(respectively 3.0, 4.63, and 7.43 arcsec). This table contains one +row for every warp row. \note{warp row hasn't been defined anywhere.} +Similarly to the \ippdbtable{Measure} table, the fields +\ippdbcolumn{objID}, \ippdbcolumn{catID}, and \ippdbcolumn{averef} +define links from the \ippdbtable{Lensing} table to the +\ippdbtable{Average} table. In a similar fashion, the fields +\ippdbtable{Average}.\ippdbcolumn{lensingOffset} and +\ippdbtable{Average}.\ippdbcolumn{Nlensing} are the index into the +sorted \ippdbtable{Lensing} table entries. \note{discuss failure of + the Lensing to Measure indexing} + +\subsubsubsection{Astronomical Objects} % object -> detection @@ -1221,144 +1262,4 @@ assigned to that object. If more than one object exists within the database, the detection is associated with the closest object. - -% photcodes -Detections in DVO have a special piece of metadata called the -\ippdbcolumn{photcode} which identifies the source of the measurement. -A \ippdbcolumn{photcode} has a name which in general consists of the -name of the camera or telescope (e.g., GPC1 or 2MASS), the name (or -short-hand name) of the filter used for the measurement (e.g., -\gps{}), and an identifier for the detector, if not unique (e.g., XY01 -for a GPC1 OTA). Along with each name, there is a numerical value for -the photcode. A table within the DVO system, \ippdbtable{Photcode}, -lists the photcodes and defines a number of additional pieces of -information for each photcode. These include the nominal zero point -and airmass slope, as well as color trends to transform a measurement -in the specific photcode to a common system. There are 3 classes of -photcodes defined within the DVO system. Those photcodes associated -with detections from an image loaded into the database system are -called \ippmisc{DEP} photcodes. There are also photcodes associated with -the average photometry values, called \ippmisc{SEC} photcodes. There are -also those measurements which come from external data sources for -which DVO does not have any information to determine a calibration -(e.g., instrumental magnitudes and detector coordinates). These are -measurements are reference values and are assigned \ippmisc{REF} -photcodes. - -% FITS table + compression -In the implementation of DVO used for the PV3 calibration analysis, -the database tables are stored on disk using binary FITS tables. Each -type of database table is stored as a separate file, or a collection -of files for table which are spatially partitioned. The binary FITS -tables are compressed using the (to date) experimental FITS binary -table compression strategy outlined by \note{REF}. Table compression -is in general an option in DVO; for the PV3 database, the large data -volume (70TB compressed) drove the decision to compress the tables. - -% FITS table compression details -The FITS binary table compression scheme uses a strategy similar to -that used for FITS image compression (\note{REF}). The binary tabular -data is compressed and stored in the 'HEAP' section of the FITS table -extension, with pointers to the compressed data stored in the regular -data section. Each column in the FITS table is compressed as one (or -more) blocks. The standard header keywords which describe the data -column format (e.g., TFORM1) are replaced with keywords which describe -the location and size of the compressed data in the HEAP section; the -information about the uncompressed data is moved to a keyword with 'Z' -prepended (e.g., ZFORM1) and an additional field is added to define -the compression algorithm (e.g., ZCTYP1). The column names (e.g., -TTYPE1) and units (e.g., TUNIT1) are retained in their original form. - -% FITS table compression details -The compression algorithm can treat the entire column as a single -block of data, or it may be broken into a number of chunks, each -compressed in turn (this must be the same for all columns). -Additional header information is added to describe the block sizes and -infomation needed to describe the HEAP data section. The compression -algorithms currently defined consist of the GZIP, RICE, PLIO, and -HCOMPRESS (REFS). For GZIP, the compression algorithm may transpose -the byte order before compression: for floating point data of a -similiar dynamic range, this choice may allow for better compression -as each byte in the 4 or 8 byte floating point value is more likely to -be similar to the same byte in other rows than to the other bytes of -the same row value. This option is called \code{GZIP_2} in the FITS -standard, as opposed to the standard order, \code{GZIP_1}. The DVO -system can be set to specify the compression options for each column -in the tables. In practice, we have chosen a default in which -floating point numbers use \code{GZIP_2}, character strings use -\code{GZIP_1}, integers use \code{RICE}. - -\subsubsection{Sky Partition} - -DVO includes two major classes of database tables: those containing -information about astronomical objects in the sky and those containing -other supporting information. The object-related tables are -partitioned on the basis of position in the sky: objects within a -region bounded by lines of constant RA,DEC are contained in a specific -file. The boundaries and the associated partition names are stored in -one of the supporting tables, \ippdbtable{SkyTable}. This table -contains the definitions of the boundaries for each sky region -(\ippdbcolumn{R_MIN}, \ippdbcolumn{R_MAX}, \ippdbcolumn{D_MIN}, -\ippdbcolumn{D_MAX}), the name of the sky region, an ID -(\ippdbcolumn{INDEX}, equal to the sequence number of the region in -the table), and index entries to enable navigation within the table. -The regions are defined in a hierarchical sense, with a series of -levels each containing a finer mesh of regions covering the sky. - -In the default used by the PV3 DVO, the partitioning scheme is based -on the one used by the Hubble Space Telescope Guide Star Catalog -files. \note{add figure} Level 0 is a single region covering the full -sky. Level 1 divides the sky in Declination into bands -7.5\degree\ high. Level 2 subdivides these Declination bands in the -RA direction, with spacing related to the stellar density. Level 3 -divides these RA chunks into 4 - 8 smaller partitions. This level -exactly matches the HST GSC layout, and uses the same naming -convention to identify the partitions: \code{n0000/0000}, -etc. \note{more on the names?}. Level 4 further divides these regions -by a factor of 16. In the \ippdbtable{SkyTable}, a region at one -level has a pointer to its parent region (the one which contains it) -and a sequence pointing to its children (regions it contains). The -\ippdbtable{SkyTable} enables fast lookups of the on-disk partitions -which map to a specific coordinate on the sky. In general, a single -DVO will have the full sky represented with tables at a single -level. Although it is possible for mixed levels to be used, this mode -is not well tested and is avoided in the PV3 DVO database. For the -PV3 master database, the partitioning at the \note{should this be - 4th?} 5th level results in \approx 150,000 regions to cover the full -sky, of which \approx 110,000 are used for the PV3 $3\pi$ data. The -densest portions of the bulge contain at most \approx 300,000 -astronomical objects in the database files, with an associated maximum -of \approx 30 million measurements in these files. With the compression -scheme described above, the largest database files are \approx -3GB, which can be loaded into memory in 30 seconds on the processing -machines that contain partition data. - -\note{is the use of the term 'partition host' consistent in this paper - and the calibration paper?} - -% parallel partitions -The DVO software system allows the tables which are partitioned across -the sky to also be distributed across multiple computers, which we -call partition hosts. A single file defines the names of these -partition hosts and the location of the database partition on the -disks of that machine. The \ippdbtable{SkyTable} contains elements to -define by ID the parition host to which a partitioned set of tables -has been assigned. Operations which query the database, or perform -other operations on the database, are aware of the partitioning scheme -and will launch their operations as remote processes on the machines -which contain the data they need. For example, a query for data from -a small region will launch sub-query operations on the machines which -contain the data overlapping the region of interest. These remote -query operations will select the database information which matches -the query request (i.e., applying restrictions as defined) and return -the results to the master process. The results from the various -partition hosts are then merged into a single result by the master -process. When the parallel partitioning for a DVO instance is -defined, the tables are randomly assigned to the partition hosts. As -a result, queries which span more than a single parition are likely to -spread the I/O load across a large number of machines. This -parallelization is critical to querying and manipulating the enormous -database on a reasonable timescale. - -\subsubsection{Astronomical Objects} Two tables carry the most important information about the astronomical @@ -1395,70 +1296,4 @@ rows $9i \rightarrow 9i + 8$ ($i$ is zero counting). -The individual measurements of the astronomical objects are carried in -the table \ippdbtable{Measure}. For measurements from PS1 in the PV3 -/ DR1 database, this would be values determined by \ippprog{psphot} -for each \ippstage{chip}, \ippstage{warp}, or \ippstage{stack} stage -image. Measurements for other cameras processed by the IPP may also -be included similarly in a DVO database. Measurements from other -sources, such as SDSS, 2MASS, or WISE, can also be included in this -table (see \S\ref{sec:other.photometry}. - -The \ippdbtable{Measure} table includes the instrumental magnitudes -for the PSF, aperture, and Kron photometry; raw position -(\ippdbcolumn{Xccd}, \ippdbcolumn{Yccd}) and second moments -(\ippdbcolumn{Mxx}, \ippdbcolumn{Myy}, \ippdbcolumn{Mxy}), along with -shape parameters of the PSF model at the position of the object -(\ippdbcolumn{FWx}, \ippdbcolumn{FWy}, \ippdbcolumn{theta}). Metadata -about the exposure that the measurement was derived from is also -included, such as the exposure time, the date \& time of the -observation, airmass, azimuth, and \ippdbcolumn{photcode} information -specifying the filter. The \ippdbtable{Measure} table also carries -the calibration magnitude offsets ($M_{\rm cal}$ and $M_{\rm flat}$, -discussed below) and the astrometrically calibrated position. -Astrometric offsets for several systematic corrections discussed below -are also defined for each measurement. Since stacks and forced warp -photometry may have non-significant values, the table is somewhat -de-normalized in that it also carries instrumental flux values for the -PSF, aperture, and Kron photometry. In this case, we have chosen to -trade storage space for computing time. - -In the \ippdbtable{Measure} table, there are three fields which -provide two independent links from the specific measurement to the -associated object: \ippdbtable{Measure}.\ippdbcolumn{catID} specifies -the spatial partition to which the measurement belongs; -\ippdbtable{Measure}.\ippdbcolumn{objID} specifies to which entry in -the \ippdbtable{Average} table the measurement belongs. These two 32 -bit fields can thus be combined into a single 64 bit ID unique for all -objects in the database. \note{PSPS IDs} In addition, the field -\ippdbtable{Measure}.\ippdbcolumn{averef} specifies the row number in -the \ippdbtable{Average} table of the associated object. The -\ippdbtable{Measure} table may be unsorted, in which case it is slow -to find the measurements associated with a specific object (a full -table scan is required). After the table is sorted and indexed, the -\ippdbcolumn{Measure} rows for a given object are grouped together. -In this case, the fields -\ippdbtable{Average}.\ippdbcolumn{measureOffset} and -\ippdbcolumn{Average}.\ippdbcolumn{Nmeasure} define an index for the -code to jump to the list of measurements for a single object. The -field \ippdbtable{Measure}.\ippdbcolumn{imageID} defines the link from -the measurement to the image which supplied the measurement. - -\note{some discussion of the db construction, addstar, dvomerge, etc?} - -For the warp images, we also measure the weak lensing KSB parameters -related to the shear and smear tensors \citep{1995ApJ...449..460K}. -These measurements are stored in the \ippdbcolumn{Lensing} table, -along with the radial aperture fluxes for radii numbers 5, 6, \& 7 -(respectively 3.0, 4.63, and 7.43 arcsec). This table contains one -row for every warp row. \note{warp row hasn't been defined anywhere.} -Similarly to the \ippdbtable{Measure} table, the fields -\ippdbcolumn{objID}, \ippdbcolumn{catID}, and \ippdbcolumn{averef} -define links from the \ippdbtable{Lensing} table to the -\ippdbtable{Average} table. In a similar fashion, the fields -\ippdbtable{Average}.\ippdbcolumn{lensingOffset} and -\ippdbtable{Average}.\ippdbcolumn{Nlensing} are the index into the -sorted \ippdbtable{Lensing} table entries. \note{discuss failure of - the Lensing to Measure indexing} - The values stored in the \ippdbtable{Lensing} table are used to calculate average values for each of these types of measurements in @@ -1493,5 +1328,5 @@ in our analysis of the astrometry \citep[][see]{magnier2017a}. -\subsubsection{Other Tables} +\subsubsubsection{Other Tables} Measurements which are loaded into DVO may be associated with a @@ -1526,4 +1361,157 @@ flat-field corrections determined by the astrometry calibration analysis \citep[][see]{magnier2017a}. + +\subsubsection{MISC INFO TO BE RE-ORGed} + +Beyond that basic use, DVO has the ability to accept data from other +kinds of data sources in which measurements are not clearly associated +with specific images. DVO ingest methods are defined for several +large-scale surveys for which the published data represent average +properties derived from multiple measurements, and for which the +measurement-to-image relationship is not provided. Ingets methods +have been defined for example for 2MASS, WISE, Gaia, USNO-B. In each +of these cases, the astrometric and photometric measurements are +stored in the \table{Measure} table, with the data source identified +by the photcode of the measurement. + +% photcodes +Detections in DVO have a special piece of metadata called the +\ippdbcolumn{photcode} which identifies the source of the measurement. +A \ippdbcolumn{photcode} has a name which in general consists of the +name of the camera or telescope (e.g., GPC1 or 2MASS), the name (or +short-hand name) of the filter used for the measurement (e.g., +\gps{}), and an identifier for the detector, if not unique (e.g., XY01 +for a GPC1 OTA). Along with each name, there is a numerical value for +the photcode. A table within the DVO system, \ippdbtable{Photcode}, +lists the photcodes and defines a number of additional pieces of +information for each photcode. These include the nominal zero point +and airmass slope, as well as color trends to transform a measurement +in the specific photcode to a common system. There are 3 classes of +photcodes defined within the DVO system. Those photcodes associated +with detections from an image loaded into the database system are +called \ippmisc{DEP} photcodes. There are also photcodes associated with +the average photometry values, called \ippmisc{SEC} photcodes. There are +also those measurements which come from external data sources for +which DVO does not have any information to determine a calibration +(e.g., instrumental magnitudes and detector coordinates). These are +measurements are reference values and are assigned \ippmisc{REF} +photcodes. + +\subsubsection{DVO Data Storage} + +% FITS table + compression +In the implementation of DVO used for the PV3 calibration analysis, +the database tables are stored on disk using binary FITS tables. Each +type of database table is stored as a separate file, or a collection +of files for table which are spatially partitioned. The binary FITS +tables are compressed using the (to date) experimental FITS binary +table compression strategy outlined by \note{REF}. Table compression +is in general an option in DVO; for the PV3 database, the large data +volume (70TB compressed) drove the decision to compress the tables. + +% FITS table compression details +The FITS binary table compression scheme uses a strategy similar to +that used for FITS image compression (\note{REF}). The binary tabular +data is compressed and stored in the 'HEAP' section of the FITS table +extension, with pointers to the compressed data stored in the regular +data section. Each column in the FITS table is compressed as one (or +more) blocks. The standard header keywords which describe the data +column format (e.g., TFORM1) are replaced with keywords which describe +the location and size of the compressed data in the HEAP section; the +information about the uncompressed data is moved to a keyword with 'Z' +prepended (e.g., ZFORM1) and an additional field is added to define +the compression algorithm (e.g., ZCTYP1). The column names (e.g., +TTYPE1) and units (e.g., TUNIT1) are retained in their original form. + +% FITS table compression details +The compression algorithm can treat the entire column as a single +block of data, or it may be broken into a number of chunks, each +compressed in turn (this must be the same for all columns). +Additional header information is added to describe the block sizes and +infomation needed to describe the HEAP data section. The compression +algorithms currently defined consist of the GZIP, RICE, PLIO, and +HCOMPRESS (REFS). For GZIP, the compression algorithm may transpose +the byte order before compression: for floating point data of a +similiar dynamic range, this choice may allow for better compression +as each byte in the 4 or 8 byte floating point value is more likely to +be similar to the same byte in other rows than to the other bytes of +the same row value. This option is called \code{GZIP_2} in the FITS +standard, as opposed to the standard order, \code{GZIP_1}. The DVO +system can be set to specify the compression options for each column +in the tables. In practice, we have chosen a default in which +floating point numbers use \code{GZIP_2}, character strings use +\code{GZIP_1}, integers use \code{RICE}. + +\subsubsection{Sky Partition} + +DVO includes two major classes of database tables: those containing +information about astronomical objects in the sky and those containing +other supporting information. The object-related tables are +partitioned on the basis of position in the sky: objects within a +region bounded by lines of constant RA,DEC are contained in a specific +file. The boundaries and the associated partition names are stored in +one of the supporting tables, \ippdbtable{SkyTable}. This table +contains the definitions of the boundaries for each sky region +(\ippdbcolumn{R_MIN}, \ippdbcolumn{R_MAX}, \ippdbcolumn{D_MIN}, +\ippdbcolumn{D_MAX}), the name of the sky region, an ID +(\ippdbcolumn{INDEX}, equal to the sequence number of the region in +the table), and index entries to enable navigation within the table. +The regions are defined in a hierarchical sense, with a series of +levels each containing a finer mesh of regions covering the sky. + +In the default used by the PV3 DVO, the partitioning scheme is based +on the one used by the Hubble Space Telescope Guide Star Catalog +files. \note{add figure} Level 0 is a single region covering the full +sky. Level 1 divides the sky in Declination into bands +7.5\degree\ high. Level 2 subdivides these Declination bands in the +RA direction, with spacing related to the stellar density. Level 3 +divides these RA chunks into 4 - 8 smaller partitions. This level +exactly matches the HST GSC layout, and uses the same naming +convention to identify the partitions: \code{n0000/0000}, +etc. \note{more on the names?}. Level 4 further divides these regions +by a factor of 16. In the \ippdbtable{SkyTable}, a region at one +level has a pointer to its parent region (the one which contains it) +and a sequence pointing to its children (regions it contains). The +\ippdbtable{SkyTable} enables fast lookups of the on-disk partitions +which map to a specific coordinate on the sky. In general, a single +DVO will have the full sky represented with tables at a single +level. Although it is possible for mixed levels to be used, this mode +is not well tested and is avoided in the PV3 DVO database. For the +PV3 master database, the partitioning at the \note{should this be + 4th?} 5th level results in \approx 150,000 regions to cover the full +sky, of which \approx 110,000 are used for the PV3 $3\pi$ data. The +densest portions of the bulge contain at most \approx 300,000 +astronomical objects in the database files, with an associated maximum +of \approx 30 million measurements in these files. With the compression +scheme described above, the largest database files are \approx +3GB, which can be loaded into memory in 30 seconds on the processing +machines that contain partition data. + +\note{is the use of the term 'partition host' consistent in this paper + and the calibration paper?} + +% parallel partitions +The DVO software system allows the tables which are partitioned across +the sky to also be distributed across multiple computers, which we +call partition hosts. A single file defines the names of these +partition hosts and the location of the database partition on the +disks of that machine. The \ippdbtable{SkyTable} contains elements to +define by ID the parition host to which a partitioned set of tables +has been assigned. Operations which query the database, or perform +other operations on the database, are aware of the partitioning scheme +and will launch their operations as remote processes on the machines +which contain the data they need. For example, a query for data from +a small region will launch sub-query operations on the machines which +contain the data overlapping the region of interest. These remote +query operations will select the database information which matches +the query request (i.e., applying restrictions as defined) and return +the results to the master process. The results from the various +partition hosts are then merged into a single result by the master +process. When the parallel partitioning for a DVO instance is +defined, the tables are randomly assigned to the partition hosts. As +a result, queries which span more than a single parition are likely to +spread the I/O load across a large number of machines. This +parallelization is critical to querying and manipulating the enormous +database on a reasonable timescale. \subsection{Addstar : DVO Ingest}