Changeset 39846 for trunk/doc/release.2015/ps1.calibration/calibration.tex

Timestamp:

Dec 11, 2016, 9:09:09 PM (10 years ago)

Author:

eugene

Message:

updates

File:

• trunk/doc/release.2015/ps1.calibration/calibration.tex (modified) (1 diff)

trunk/doc/release.2015/ps1.calibration/calibration.tex

@@ -503 +503 @@
 the data from the exposure are loaded into the DVO database.
 
-\section{DVO Description}
-
-The Pan-STARRS IPP uses an internal database system, distinct from the
-publically visible database system, to determine the association
-between multiple detections of the same astronomical object and as
-part of the astrometric and photometric calibration process.  This
-database system, called the ``Desktop Virtual Observatory'' (DVO) was
-developed originally for the LONEOS project, and used as part of the
-CFHT Elixir system (Magnier \& Cuillandre REF).  The capabilities of
-this databasing system have been somewhat expanded for the Pan-STARRS
-context.  
-
-One of the main purposes of the DVO system is to define the
-relationship between individual detections of an astronomical object
-and the definition of that object.  Before describing the database
-schema, we will discuss the process by which detections are associated
-with objects.  New detections are generally added to the database in a
-group associated with, for example, an image from the GPC1 camera.  As
-new detections are loaded, they are compared to the objects already
-stored in the database.  If an object is already found in the database
-within the match radius, the new detection is associated to that
-object. If more than one object exists within the database, the
-detection is associated with the closest object.  
-
-Detections in DVO have a special piece of metadata called the
-\code{photcode} which identifies the source of the measurement.  A
-\code{photcode} has a name which in general consists of the name of
-the camera or telescope (e.g., GPC1 or 2MASS), the name (or short-hand
-name) of the filter used for the measurement (e.g., $g$), and an
-identifier for the detector, if not unique (e.g., XY01 for GPC1).
-Along with each name, there is a numerical value for the photcode.  A
-table within the DVO system, \code{Photcode}, lists the photcoes and
-defines a number of additional pieces of information for each
-photcode.  These include the nominal zero point and airmass slope, as
-well as color trends to transform a measurement in the specific
-photcode to a common system.  There are 3 classes of photcodes defined
-within the DVO system.  Those photcodes associated with detections
-from an image loaded into the database system are called \code{DEP}
-photcodes.  There are also photcodes associated with the average
-photometry values, called SEC photcodes.  There are also those
-measurements which come from external data sources for which DVO does
-not have any information to determine a calibration (e.g.,
-instrumental magnitudes and detector coordinates).  These are
-measurements are reference values and are assigned REF photcodes.
-
-In the implementation of DVO used for the PV3 calibration analysis,
-the database tables are stored on disk using binary FITS tables.  Each
-type of database table is stored as a separate file, or a collection
-of files for table which are spatially partitioned.  The binary FITS
-tables may be optionally compressed using the (to date) experimental
-FITS binary table compression strategy outlined by REF.  In this
-compression scheme, using a strategy similar to that used for FITS
-image compression (REF), the data stored in the binary table is
-compressed and stored in the 'HEAP' section of the FITS table.  In
-brief, each column in the FITS table is compressed as one (or more)
-blocks.  The standard fields which describe the data column format
-(e.g., TFORM1) are replaced with columns which describe the location
-and size of the compressed data in the HEAP section; the information
-about the uncompressed data is moved to a field with 'Z' prepended
-(e.g., ZFORM1) and an additional field is added to define the
-compression algorithm (e.g., ZCTYP1).  The column names (e.g., TTYPE1)
-and units (e.g., TUNIT1) are retained in their original form.  The
-compression algorithm can treat the entire column as a single block of
-data, or it may be broken into a number of chunks, each compressed in
-turn (this must be the same for all columns).  Additional header
-information is added to describe the block sizes and infomation needed
-to describe the HEAP data section.  The compression algorithms
-currently defined consist of the GZIP, RICE, PLIO, and HCOMPRESS
-(REFS).  For GZIP, the compression algorithm may transpose the byte
-order before compression: for floating point data of a similiar
-dynamic range, this choice may allow for better compression as each
-byte in the 4 or 8 byte floating point value is more likely to be
-similar to the same byte in other rows than to the other bytes of the
-same row value.  This option is called \code{GZIP_2} in the FITS
-standard, as opposed to the standard order, \code{GZIP_1}.  The DVO
-system can be set to specify the compression options for each column
-in the tables.  In practice, we have chosen a default in which
-floating point numbers used \code{GZIP_2}, character strings use
-\code{GZIP_1}, integers use \code{RICE}.  
-
-\subsubsection{Sky Partition}
-
-DVO includes two major classes of database tables: those containing
-information directly about astronomical objects in the sky and those
-containing other supporting information.  The object-related tables
-are partitioned on the basis of position in the sky: objects within a
-region bounded by lines of constant RA,DEC are contained in a specific
-file.  The boundaries and the associated partition names are stored in
-one of the supporting tables, \code{SkyTable}.  This table contains
-the definitions of the boundaries for each sky region (\code{R_MIN},
-\code{R_MAX}, \code{D_MIN}, \code{D_MAX}), the name of the sky region,
-an ID (\code{INDEX}, equal to the sequence number of the region in the
-table), and index entries to enable navigation within the table.  The
-regions are defined in a hierarchical sense, with a series of levels
-each containing a finer mesh of regions covering the sky.  
-
-In the default used by the PV3 DVO, the partitioning scheme is based
-on the one used by the Hubble Space Telescope Guide Star Catalog
-files.  Level 0 is a single region covering the full sky.  Level 1
-divides the sky in Declination into bands 7.5\degree\ high.  Level 2
-subdivides these Declination bands in the RA direction, with spacing
-related to the stellar density.  Level 3 divides these RA chunks into
-4 - 8 smaller partitions.  This level exactly matches the HST GSC
-layout, and uses the same naming convention to identify the
-partitions: n0000/0000, etc. \note{more on the names?}.  Level 4
-further divides these regions by a factor of 16.  In the
-\code{SkyTable}, a region at one level has a pointer to its parent
-region (the one which contains it) and a sequence pointing to its
-children (regions it contains).  The \code{SkyTable} enables fast
-lookups of the on-disk partitions which map to a specific coordinate
-on the sky.  In general, a single DVO will have the full sky
-represented with tables at a single level, though it is possible for
-mixed levels to be used, this mode is not well tested.  For the PV3
-master database, the partitioning at the 5th level results in \approx
-150,000 regions to cover the full sky, of which \approx 110,000 are
-used for the PV3 $3\pi$ data.  The densest portions of the bulge
-contain at most \approx 300k astronomical objects in the database
-files, with an associated maximum of 30M measurements in these files.
-With the compression scheme described above, this makes the largest
-database files \approx 3GB, which can be loaded into memory in 30
-seconds on our partition machines.
-
-The DVO software system allows the tables which are partitioned across
-the sky to also be distributed across multiple computers, which we
-call partition hosts.  A single file defines the names of these
-partition hosts and the location of the database partition on the
-disks of that machine.  The \code{SkyTable} contains elements to
-define by ID the parition host to which a partitioned set of tables
-has been assigned.  Operations which query the database, or perform
-other operations on the database, are aware of the partitioning scheme
-and will launch their operations as remote processes on the machines
-which contain the data they need.  For example, a query for data from
-a small region will launch sub-query operations on the machines which
-contain the data overlapping the region of interest.  These remote
-query operations will select the database information which matches
-the query request (i.e., applying restrictions as defined) and return
-to the master process the results.  The results from the various
-partition hosts are then merged into a single result by the master
-process.  This parallelization is critical to querying and
-manipulating the enormous database on a reasonable timescale.
-
-\subsection{Tables which describe objects} 
-
-Two tables carry the most important information about the astronomical
-objects in the database: Average and SecFilt.  These two tables
-specify the main average properties of the astronomical object.  The
-Average table includes the astrometric information ($\alpha, \delta,
-\mu \alpha, \mu \delta, \pi$) and associated errors, data quality
-flags for each object, links to the other tables, and a number of IDs,
-with one row for each astronomical object.  \note{go into complete
-  detail here on the IDs?}.  The SecFilt table\footnote{The name
-  SecFilt is a bit of a historical misnomer: originally, DVO was
-  designed for a monochromatic survey and data for a single
-  photometric band was maintained in the Average table.  Later, DVO
-  was adapted to a multifilter system and additional filters were
-  added to the SecFilt (Secondary Filter) table.  Eventually, the
-  schema was normalized and all photometric data placed in SecFilt,
-  with the Primary filter concept being dropped, but the name has
-  since stuck.} contains average photometric information for a
-collection of filters.  A given DVO instance has a specified set of
-filters for which average photometry is stored in the SecFilt table.
-The number and choice of filters for the SecFilt may be modified by
-the database administrator fairly easily, but the process of updating
-the database is somewhat expensive (\approx 24 hours for the current
-PV3 instance).  Thus the choice is semi-static for a given DVO
-instance.  In the case of the PV3 DVO instance, 9 average bandpasses
-are defined: {\it g, r, i, z, y, J, H, K, w}.  The SecFilt table
-contains one row for each filter for each object, thus the PV3 DVO
-contains 9 times as many rows as the Average table.  The order of the
-table is fixed in relation to the Average table: row $i$ of Average
-defines the object with photometry contained in rows $9i \rightarrow 9i +
-8$ ($i$ is zero counting).  
-
-The individual measurements of the astronomical objects are carried in
-the table \code{Measure}.  This table lists the values measured by
-\code{psphot} for each chip, warp, or stack image.  This includes the
-instrumental magnitudes for the PSF, aperture, and Kron photometry;
-raw position (Xccd, Yccd) and second moments (Mxx, Myy, Mxy), along
-with shape parameters of the PSF model at the position of the object
-(FWx, FWy, theta).  This table also includes metadata such as the
-exposure time, the date \& time of the observation, airmass, azimuth,
-and information specifying the filter \note{describe the photcodes}.
-The \code{Measure} table also carried the calibration magnitude offsts
-($M_{\rm cal}$ and $M_{\rm flat}$ discussed below) and the
-astrometrically calibrated position.  Astrometric offsets for several
-systematic corrections discussed below are also defined for each
-measurement.  Since stacks and forced warp photometry may have
-non-significant values, the table is somewhat de-normalized in that it
-also carried instrumental flux values for the PSF, aperture, and Kron
-photometry.  
-
-In the \code{Measure} table, there are three fields which provide two
-independent links from the specific measurement to the associated
-object: \code{Measure.catID} specifies the spatial partition to which
-the measurement belongs; \code{Measure.objID} specifies to which entry
-in the \code{Average} table the measurement belongs.  These two 32 bit
-fields can thus be combined into a single 64 bit unique ID for all
-objects in the database.  In addition, the field \code{Measure.averef}
-specifies the row number in the \code{Average} table of the associated
-object.  The \code{Measure} table may be unsorted, in which case it is
-slow to find the measurements associated with a specific object (a
-full table scan is required).  After the table is sorted, the
-\code{Measure} rows for a given object are grouped together.  In the
-case, the fields \code{Average.measureOffset} and
-\code{Average.Nmeasure} define an index for the code to jump to the
-list of measurements for a single object.  The field
-\code{Measure.imageID} defines the link from the measurement to the
-image which supplied the measurement.
-
-\note{some discussion of the db construction, addstar, dvomerge, etc?}
-
-For the warp images, we also measure the weak lensing KSB parameters
-related to the shear and smear tensors.  These measurements are stored
-in the \code{Lensing} table, along with the radial aperture fluxes for
-radii numbers 5, 6, \& 7 (XX, XX, XX arcsec).  This table contains one
-row for every warp row.  Similarly to the \code{Measure} table, the fields
-\code{objID}, \code{catID}, and \code{averef} define links from the
-\code{Lensing} table to the \code{Average} table.  In a similar
-fashion, the fields \code{Average.lensingOffset} and
-\code{Average.Nlensing} are the index into the sorted \code{Lensing}
-table entries.  \note{discuss failure of the Lensing to Measure
-  indexing}
-
-The values stored in the \code{Lensing} table are used to calculate
-average values for each of these types of measurements in each
-filter.  The \code{Lensobj} table stores the averaged KSB and radial
-aperture photometry for each of the 5 filters \grizy.  This table
-contains one entry per object per filter.  The table is not generally
-stored unsorted as it is calculated after the full database is
-populated.  The link from \code{Average} to \code{Lensobj} is defined
-by the fields \code{Average.offsetLensobj} and
-\code{Average.Nlensobj}.  Each \code{Lensobj} row also includes the
-photcode (filter) for which the average lensing (and radial aperture)
-properties have been calculated. 
-
-The \code{Galphot} table stores the results of the forced galaxy
-fitting analysis for each object that has been measured.  This table
-contains one row per filter and model type (Sersic, Exponential,
-DeVaucouleur) if forced galaxy models have been calculate for the
-object.  \note{need to expand on this somewhat}
-
-The \code{Starpar} table carries measurements provide by Greg Green \&
-Eddie Schlafly from their analysis of the SED of objects in the PS1
-$3\pi$ data, using the \note{PV1?} version of the analysis (Green et
-al REF).  In this work, the goal was a 3D model of the dust in the
-Galaxy based on Pan-STARRS (\note{and WISE \& 2MASS?}) photometry.  As
-part of this analysis, the authors fit the SEDs of all \note{stellar?}
-sources with stellar models including free parameters of extinction,
-distance modulus, metallicity, and absolute r-band magnitude.  While
-these photometric distance modulus measurements are not extremely
-precise (see below), they provide a constraint on the distance is used
-in our analysis of the astrometry (see Section~\ref{sec:astrometry}).
-
-\subsection{Other Tables} 
-
-Data from GPC1 (and other cameras processed by the IPP) are loaded
-into DVO in units \code{smf} files generated by the Camera calibration
-stage.  As described above, these files contain all measurements from
-a complete exposure, with measurements from each chip grouped into
-separate FITS tables.  When these measurements are loaded into the
-\code{Measure} and similar tables, a subset of the information from
-the chip header is used to populated a row in the DVO \code{Image}
-table.  This table contains one row for each chip known to DVO, with
-information such as the filter (\code{photcode}), the exposure time,
-the airmass, the astrometric calibration terms, the photometric
-zero point, etc.  For GPC1 and other mosaic cameras, an additional row
-is defined to carry the projection and camera distortion elements of
-the astrometry model.  As chips are loaded into this table, they are
-assigned an internal ID (a running sequence in the table).  Images may
-also be assigned an external ID: in the case of the GPC1 images, this
-ID is defined by the processing mysql database and is guaranteed to be
-unique within the processing system. 
-
-Other tables are used to track information used by the calibration
-system.  This includes the complete set of flat-field corrections
-determined by the photometry calibration analysis (see
-Section~\ref{sec:relphot}) and the astrometric flat-field corrections
-determined by the astrometry calibration analysis (see Section~\ref{sec:relastro})
-
 \section{Photometry Calibration}