Changeset 39838

Timestamp:

Dec 6, 2016, 5:29:12 PM (10 years ago)

Author:

eugene

Message:

updates to cal

File:

• trunk/doc/release.2015/ps1.calibration/calibration.tex (modified) (1 diff)

trunk/doc/release.2015/ps1.calibration/calibration.tex

@@ -458 +458 @@
 detection is associated with the closest object.  
 
-* Object-related tables
-
-* Other tables 
-
-* Table storag
+In the implementation of DVO used for the PV3 calibration analysis,
+the database tables are stored on disk using binary FITS tables.  Each
+type of database table is stored as a separate file, or a collection
+of files if the table is spatially partitioned.  The binary FITS
+tables may be optionally compressed using the (to date) experimental
+FITS binary table compression strategy outlined by REF.  In this
+compression scheme, using a strategy similar to that used for FITS
+image compression (REF), the data stored in the binary table is
+compressed and stored in the 'HEAP' section of the FITS table.  In
+brief, each column in the FITS table is compressed as one (or more)
+blocks.  The standard fields which describe the data column format
+(e.g., TFORM1) are replaced with columns which describe the location
+and size of the compressed data in the HEAP section; the information
+about the uncompressed data is moved to a field with 'Z' prepended
+(e.g., ZFORM1) and an additional field is added to define the
+compression algorithm (e.g., ZCTYP1).  The column names (e.g., TTYPE1)
+and units (e.g., TUNIT1) are retained in their original form.  The
+compression algorithm can treat the entire column as a single block of
+data, or it may be broken into a number of chunks, each compressed in
+turn (this must be the same for all columns).  Additional header
+information is added to describe the block sizes and infomation needed
+to describe the HEAP data section.  The compression algorithms
+currently defined consist of the GZIP, RICE, PLIO, and HCOMPRESS
+(REFS).  For GZIP, the compression algorithm may transpose the byte
+order before compression: for floating point data of a similiar
+dynamic range, this choice may allow for better compression as each
+byte in the 4 or 8 byte floating point value is more likely to be
+similar to the same byte in other rows than to the other bytes of the
+same row value.  This option is called \code{GZIP_2} in the FITS
+standard, as opposed to the standard order, \code{GZIP_1}.  The DVO
+system can be set to specify the compression options for each column
+in the tables.  In practice, we have chosen a default in which
+floating point numbers used \code{GZIP_2}, character strings use
+\code{GZIP_1}, integers use \code{RICE}.  
+
+\subsection{Tables which describe objects} 
+
+Two tables carry the most important information about the astronomical
+objects in the database: Average and SecFilt.  These two tables
+specify the main average properties of the astronomical object.  The
+Average table includes the astrometric information ($\alpha, \delta,
+\mu \alpha, \mu \delta, \pi$) and associated errors, data quality
+flags for each object, links to the other tables, and a number of IDs,
+with one row for each astronomical object.  \note{go into complete
+  detail here on the IDs?}.  The SecFilt table\footnote{The name
+  SecFilt is a bit of a historical misnomer: originally, DVO was
+  designed for a monochromatic survey and data for a single
+  photometric band was maintained in the Average table.  Later, DVO
+  was adapted to a multifilter system and additional filters were
+  added to the SecFilt (Secondary Filter) table.  Eventually, the
+  schema was normalized and all photometric data placed in SecFilt,
+  with the Primary filter concept being dropped, but the name has
+  since stuck.} contains average photometric information for a
+collection of filters.  A given DVO instance has a specified set of
+filters for which average photometry is stored in the SecFilt table.
+The number and choice of filters for the SecFilt may be modified by
+the database administrator fairly easily, but the process of updating
+the database is somewhat expensive (\approx 24 hours for the current
+PV3 instance).  Thus the choice is semi-static for a given DVO
+instance.  In the case of the PV3 DVO instance, 9 average bandpasses
+are defined: {\it g, r, i, z, y, J, H, K, w}.  The SecFilt table
+contains one row for each filter for each object, thus the PV3 DVO
+contains 9 times as many rows as the Average table.  The order of the
+table is fixed in relation to the Average table: row $i$ of Average
+defines the object with photometry contained in rows $9i \rtarrow 9i +
+8$ ($i$ is zero counting).  
+
+The individual measurements of the astronomical objects are carried in
+the table \code{Measure}.  This table lists the values measured by
+\code{psphot} for each chip, warp, or stack image.  This includes the
+instrumental magnitudes for the PSF, aperture, and Kron photometry;
+raw position (Xccd, Yccd) and second moments (Mxx, Myy, Mxy), along
+with shape parameters of the PSF model at the position of the object
+(FWx, FWy, theta).  This table also includes metadata such as the
+exposure time, the date \& time of the observation, airmass, azimuth,
+and information specifying the filter \note{describe the photcodes}.
+The \code{Measure} table also carried the calibration magnitude offsts
+($M_{\rm cal}$ and $M_{\rm flat}$ discussed below) and the
+astrometrically calibrated position.  Astrometric offsets for several
+systematic corrections discussed below are also defined for each
+measurement.  Since stacks and forced warp photometry may have
+non-significant values, the table is somewhat de-normalized in that it
+also carried instrumental flux values for the PSF, aperture, and Kron
+photometry.  
+
+In the \code{Measure} table, there are three fields which provide two
+independent links from the specific measurement to the associated
+object: \code{Measure.catID} specifies the spatial partition to which
+the measurement belongs; \code{Measure.objID} specifies to which entry
+in the \code{Average} table the measurement belongs.  These two 32 bit
+fields can thus be combined into a single 64 bit unique ID for all
+objects in the database.  In addition, the field \code{Measure.averef}
+specifies the row number in the \code{Average} table of the associated
+object.  The \code{Measure} table may be unsorted, in which case it is
+slow to find the measurements associated with a specific object (a
+full table scan is required).  After the table is sorted, the
+\code{Measure} rows for a given object are grouped together.  In the
+case, the fields \code{Average.measureOffset} and
+\code{Average.Nmeasure} define an index for the code to jump to the
+list of measurements for a single object.  The field
+\code{Measure.imageID} defines the link from the measurement to the
+image which supplied the measurement.
+
+\note{some discussion of the db construction, addstar, dvomerge, etc?}
+
+For the warp images, we also measure the weak lensing KSB parameters
+related to the shear and smear tensors.  These measurements are stored
+in the \code{Lensing} table, along with the radial aperture fluxes for
+radii numbers 5, 6, \& 7 (XX, XX, XX arcsec).  This table contains one
+row for every warp row.  Similarly to the \code{Measure} table, the fields
+\code{objID}, \code{catID}, and \code{averef} define links from the
+\code{Lensing} table to the \code{Average} table.  In a similar
+fashion, the fields \code{Average.lensingOffset} and
+\code{Average.Nlensing} are the index into the sorted \code{Lensing}
+table entries.  \note{discuss failure of the Lensing to Measure
+  indexing}
+
+The values stored in the \code{Lensing} table are used to calculate
+average values for each of these types of measurements in each
+filter.  The \code{Lensobj} table stores the averaged KSB and radial
+aperture photometry for each of the 5 filters \grizy.  This table
+contains one entry per object per filter.  The table is not generally
+stored unsorted as it is calculated after the full database is
+populated.  The link from \code{Average} to \code{Lensobj} is defined
+by the fields \code{Average.offsetLensobj} and
+\code{Average.Nlensobj}.  Each \code{Lensobj} row also includes the
+photcode (filter) for which the average lensing (and radial aperture)
+properties have been calculated. 
+
+The \code{Galphot} table stores the results of the forced galaxy
+fitting analysis for each object that has been measured.  This table
+contains one row per filter and model type (Sersic, Exponential,
+DeVaucouleur) if forced galaxy models have been calculate for the
+object.  \note{need to expand on this somewhat}
+
+The \code{Starpar} table carries measurements provide by Greg Green \&
+Eddie Schlafly from their analysis of the SED of objects in the PS1
+$3\pi$ data, using the \note{PV1?} version of the analysis (Green et
+al REF).  In this work, the goal was a 3D model of the dust in the
+Galaxy based on Pan-STARRS (\note{and WISE \& 2MASS?}) photometry.  As
+part of this analysis, the authors fit the SEDs of all \note{stellar?}
+sources with stellar models including free parameters of extinction,
+distance modulus, metallicity, and absolute r-band magnitude.  While
+these photometric distance modulus measurements are not extremely
+precise (see below), they provide a constraint on the distance is used
+in our analysis of the astrometry (see Section~\ref{sec:astrometry}).
+
+\subsubsection{Sky Partition}
+
+\note{SkyTable}
+
+\subsection{Other Tables} 
+
+\note{Image Table}
+\note{Photcode Table}
+\note{FlatCorrection}
+\note{AstromOffsets}
 
 \section{Photometry Calibration}