Index: trunk/doc/release.2015/ps1.calibration/calibration.tex =================================================================== --- trunk/doc/release.2015/ps1.calibration/calibration.tex (revision 39837) +++ trunk/doc/release.2015/ps1.calibration/calibration.tex (revision 39838) @@ -458,9 +458,161 @@ detection is associated with the closest object. -* Object-related tables - -* Other tables - -* Table storag +In the implementation of DVO used for the PV3 calibration analysis, +the database tables are stored on disk using binary FITS tables. Each +type of database table is stored as a separate file, or a collection +of files if the table is spatially partitioned. The binary FITS +tables may be optionally compressed using the (to date) experimental +FITS binary table compression strategy outlined by REF. In this +compression scheme, using a strategy similar to that used for FITS +image compression (REF), the data stored in the binary table is +compressed and stored in the 'HEAP' section of the FITS table. In +brief, each column in the FITS table is compressed as one (or more) +blocks. The standard fields which describe the data column format +(e.g., TFORM1) are replaced with columns which describe the location +and size of the compressed data in the HEAP section; the information +about the uncompressed data is moved to a field with 'Z' prepended +(e.g., ZFORM1) and an additional field is added to define the +compression algorithm (e.g., ZCTYP1). The column names (e.g., TTYPE1) +and units (e.g., TUNIT1) are retained in their original form. The +compression algorithm can treat the entire column as a single block of +data, or it may be broken into a number of chunks, each compressed in +turn (this must be the same for all columns). Additional header +information is added to describe the block sizes and infomation needed +to describe the HEAP data section. The compression algorithms +currently defined consist of the GZIP, RICE, PLIO, and HCOMPRESS +(REFS). For GZIP, the compression algorithm may transpose the byte +order before compression: for floating point data of a similiar +dynamic range, this choice may allow for better compression as each +byte in the 4 or 8 byte floating point value is more likely to be +similar to the same byte in other rows than to the other bytes of the +same row value. This option is called \code{GZIP_2} in the FITS +standard, as opposed to the standard order, \code{GZIP_1}. The DVO +system can be set to specify the compression options for each column +in the tables. In practice, we have chosen a default in which +floating point numbers used \code{GZIP_2}, character strings use +\code{GZIP_1}, integers use \code{RICE}. + +\subsection{Tables which describe objects} + +Two tables carry the most important information about the astronomical +objects in the database: Average and SecFilt. These two tables +specify the main average properties of the astronomical object. The +Average table includes the astrometric information ($\alpha, \delta, +\mu \alpha, \mu \delta, \pi$) and associated errors, data quality +flags for each object, links to the other tables, and a number of IDs, +with one row for each astronomical object. \note{go into complete + detail here on the IDs?}. The SecFilt table\footnote{The name + SecFilt is a bit of a historical misnomer: originally, DVO was + designed for a monochromatic survey and data for a single + photometric band was maintained in the Average table. Later, DVO + was adapted to a multifilter system and additional filters were + added to the SecFilt (Secondary Filter) table. Eventually, the + schema was normalized and all photometric data placed in SecFilt, + with the Primary filter concept being dropped, but the name has + since stuck.} contains average photometric information for a +collection of filters. A given DVO instance has a specified set of +filters for which average photometry is stored in the SecFilt table. +The number and choice of filters for the SecFilt may be modified by +the database administrator fairly easily, but the process of updating +the database is somewhat expensive (\approx 24 hours for the current +PV3 instance). Thus the choice is semi-static for a given DVO +instance. In the case of the PV3 DVO instance, 9 average bandpasses +are defined: {\it g, r, i, z, y, J, H, K, w}. The SecFilt table +contains one row for each filter for each object, thus the PV3 DVO +contains 9 times as many rows as the Average table. The order of the +table is fixed in relation to the Average table: row $i$ of Average +defines the object with photometry contained in rows $9i \rtarrow 9i + +8$ ($i$ is zero counting). + +The individual measurements of the astronomical objects are carried in +the table \code{Measure}. This table lists the values measured by +\code{psphot} for each chip, warp, or stack image. This includes the +instrumental magnitudes for the PSF, aperture, and Kron photometry; +raw position (Xccd, Yccd) and second moments (Mxx, Myy, Mxy), along +with shape parameters of the PSF model at the position of the object +(FWx, FWy, theta). This table also includes metadata such as the +exposure time, the date \& time of the observation, airmass, azimuth, +and information specifying the filter \note{describe the photcodes}. +The \code{Measure} table also carried the calibration magnitude offsts +($M_{\rm cal}$ and $M_{\rm flat}$ discussed below) and the +astrometrically calibrated position. Astrometric offsets for several +systematic corrections discussed below are also defined for each +measurement. Since stacks and forced warp photometry may have +non-significant values, the table is somewhat de-normalized in that it +also carried instrumental flux values for the PSF, aperture, and Kron +photometry. + +In the \code{Measure} table, there are three fields which provide two +independent links from the specific measurement to the associated +object: \code{Measure.catID} specifies the spatial partition to which +the measurement belongs; \code{Measure.objID} specifies to which entry +in the \code{Average} table the measurement belongs. These two 32 bit +fields can thus be combined into a single 64 bit unique ID for all +objects in the database. In addition, the field \code{Measure.averef} +specifies the row number in the \code{Average} table of the associated +object. The \code{Measure} table may be unsorted, in which case it is +slow to find the measurements associated with a specific object (a +full table scan is required). After the table is sorted, the +\code{Measure} rows for a given object are grouped together. In the +case, the fields \code{Average.measureOffset} and +\code{Average.Nmeasure} define an index for the code to jump to the +list of measurements for a single object. The field +\code{Measure.imageID} defines the link from the measurement to the +image which supplied the measurement. + +\note{some discussion of the db construction, addstar, dvomerge, etc?} + +For the warp images, we also measure the weak lensing KSB parameters +related to the shear and smear tensors. These measurements are stored +in the \code{Lensing} table, along with the radial aperture fluxes for +radii numbers 5, 6, \& 7 (XX, XX, XX arcsec). This table contains one +row for every warp row. Similarly to the \code{Measure} table, the fields +\code{objID}, \code{catID}, and \code{averef} define links from the +\code{Lensing} table to the \code{Average} table. In a similar +fashion, the fields \code{Average.lensingOffset} and +\code{Average.Nlensing} are the index into the sorted \code{Lensing} +table entries. \note{discuss failure of the Lensing to Measure + indexing} + +The values stored in the \code{Lensing} table are used to calculate +average values for each of these types of measurements in each +filter. The \code{Lensobj} table stores the averaged KSB and radial +aperture photometry for each of the 5 filters \grizy. This table +contains one entry per object per filter. The table is not generally +stored unsorted as it is calculated after the full database is +populated. The link from \code{Average} to \code{Lensobj} is defined +by the fields \code{Average.offsetLensobj} and +\code{Average.Nlensobj}. Each \code{Lensobj} row also includes the +photcode (filter) for which the average lensing (and radial aperture) +properties have been calculated. + +The \code{Galphot} table stores the results of the forced galaxy +fitting analysis for each object that has been measured. This table +contains one row per filter and model type (Sersic, Exponential, +DeVaucouleur) if forced galaxy models have been calculate for the +object. \note{need to expand on this somewhat} + +The \code{Starpar} table carries measurements provide by Greg Green \& +Eddie Schlafly from their analysis of the SED of objects in the PS1 +$3\pi$ data, using the \note{PV1?} version of the analysis (Green et +al REF). In this work, the goal was a 3D model of the dust in the +Galaxy based on Pan-STARRS (\note{and WISE \& 2MASS?}) photometry. As +part of this analysis, the authors fit the SEDs of all \note{stellar?} +sources with stellar models including free parameters of extinction, +distance modulus, metallicity, and absolute r-band magnitude. While +these photometric distance modulus measurements are not extremely +precise (see below), they provide a constraint on the distance is used +in our analysis of the astrometry (see Section~\ref{sec:astrometry}). + +\subsubsection{Sky Partition} + +\note{SkyTable} + +\subsection{Other Tables} + +\note{Image Table} +\note{Photcode Table} +\note{FlatCorrection} +\note{AstromOffsets} \section{Photometry Calibration}