Index: trunk/doc/release.2015/ps1.calibration/calibration.tex =================================================================== --- trunk/doc/release.2015/ps1.calibration/calibration.tex (revision 39845) +++ trunk/doc/release.2015/ps1.calibration/calibration.tex (revision 39846) @@ -503,283 +503,4 @@ the data from the exposure are loaded into the DVO database. -\section{DVO Description} - -The Pan-STARRS IPP uses an internal database system, distinct from the -publically visible database system, to determine the association -between multiple detections of the same astronomical object and as -part of the astrometric and photometric calibration process. This -database system, called the ``Desktop Virtual Observatory'' (DVO) was -developed originally for the LONEOS project, and used as part of the -CFHT Elixir system (Magnier \& Cuillandre REF). The capabilities of -this databasing system have been somewhat expanded for the Pan-STARRS -context. - -One of the main purposes of the DVO system is to define the -relationship between individual detections of an astronomical object -and the definition of that object. Before describing the database -schema, we will discuss the process by which detections are associated -with objects. New detections are generally added to the database in a -group associated with, for example, an image from the GPC1 camera. As -new detections are loaded, they are compared to the objects already -stored in the database. If an object is already found in the database -within the match radius, the new detection is associated to that -object. If more than one object exists within the database, the -detection is associated with the closest object. - -Detections in DVO have a special piece of metadata called the -\code{photcode} which identifies the source of the measurement. A -\code{photcode} has a name which in general consists of the name of -the camera or telescope (e.g., GPC1 or 2MASS), the name (or short-hand -name) of the filter used for the measurement (e.g., $g$), and an -identifier for the detector, if not unique (e.g., XY01 for GPC1). -Along with each name, there is a numerical value for the photcode. A -table within the DVO system, \code{Photcode}, lists the photcoes and -defines a number of additional pieces of information for each -photcode. These include the nominal zero point and airmass slope, as -well as color trends to transform a measurement in the specific -photcode to a common system. There are 3 classes of photcodes defined -within the DVO system. Those photcodes associated with detections -from an image loaded into the database system are called \code{DEP} -photcodes. There are also photcodes associated with the average -photometry values, called SEC photcodes. There are also those -measurements which come from external data sources for which DVO does -not have any information to determine a calibration (e.g., -instrumental magnitudes and detector coordinates). These are -measurements are reference values and are assigned REF photcodes. - -In the implementation of DVO used for the PV3 calibration analysis, -the database tables are stored on disk using binary FITS tables. Each -type of database table is stored as a separate file, or a collection -of files for table which are spatially partitioned. The binary FITS -tables may be optionally compressed using the (to date) experimental -FITS binary table compression strategy outlined by REF. In this -compression scheme, using a strategy similar to that used for FITS -image compression (REF), the data stored in the binary table is -compressed and stored in the 'HEAP' section of the FITS table. In -brief, each column in the FITS table is compressed as one (or more) -blocks. The standard fields which describe the data column format -(e.g., TFORM1) are replaced with columns which describe the location -and size of the compressed data in the HEAP section; the information -about the uncompressed data is moved to a field with 'Z' prepended -(e.g., ZFORM1) and an additional field is added to define the -compression algorithm (e.g., ZCTYP1). The column names (e.g., TTYPE1) -and units (e.g., TUNIT1) are retained in their original form. The -compression algorithm can treat the entire column as a single block of -data, or it may be broken into a number of chunks, each compressed in -turn (this must be the same for all columns). Additional header -information is added to describe the block sizes and infomation needed -to describe the HEAP data section. The compression algorithms -currently defined consist of the GZIP, RICE, PLIO, and HCOMPRESS -(REFS). For GZIP, the compression algorithm may transpose the byte -order before compression: for floating point data of a similiar -dynamic range, this choice may allow for better compression as each -byte in the 4 or 8 byte floating point value is more likely to be -similar to the same byte in other rows than to the other bytes of the -same row value. This option is called \code{GZIP_2} in the FITS -standard, as opposed to the standard order, \code{GZIP_1}. The DVO -system can be set to specify the compression options for each column -in the tables. In practice, we have chosen a default in which -floating point numbers used \code{GZIP_2}, character strings use -\code{GZIP_1}, integers use \code{RICE}. - -\subsubsection{Sky Partition} - -DVO includes two major classes of database tables: those containing -information directly about astronomical objects in the sky and those -containing other supporting information. The object-related tables -are partitioned on the basis of position in the sky: objects within a -region bounded by lines of constant RA,DEC are contained in a specific -file. The boundaries and the associated partition names are stored in -one of the supporting tables, \code{SkyTable}. This table contains -the definitions of the boundaries for each sky region (\code{R_MIN}, -\code{R_MAX}, \code{D_MIN}, \code{D_MAX}), the name of the sky region, -an ID (\code{INDEX}, equal to the sequence number of the region in the -table), and index entries to enable navigation within the table. The -regions are defined in a hierarchical sense, with a series of levels -each containing a finer mesh of regions covering the sky. - -In the default used by the PV3 DVO, the partitioning scheme is based -on the one used by the Hubble Space Telescope Guide Star Catalog -files. Level 0 is a single region covering the full sky. Level 1 -divides the sky in Declination into bands 7.5\degree\ high. Level 2 -subdivides these Declination bands in the RA direction, with spacing -related to the stellar density. Level 3 divides these RA chunks into -4 - 8 smaller partitions. This level exactly matches the HST GSC -layout, and uses the same naming convention to identify the -partitions: n0000/0000, etc. \note{more on the names?}. Level 4 -further divides these regions by a factor of 16. In the -\code{SkyTable}, a region at one level has a pointer to its parent -region (the one which contains it) and a sequence pointing to its -children (regions it contains). The \code{SkyTable} enables fast -lookups of the on-disk partitions which map to a specific coordinate -on the sky. In general, a single DVO will have the full sky -represented with tables at a single level, though it is possible for -mixed levels to be used, this mode is not well tested. For the PV3 -master database, the partitioning at the 5th level results in \approx -150,000 regions to cover the full sky, of which \approx 110,000 are -used for the PV3 $3\pi$ data. The densest portions of the bulge -contain at most \approx 300k astronomical objects in the database -files, with an associated maximum of 30M measurements in these files. -With the compression scheme described above, this makes the largest -database files \approx 3GB, which can be loaded into memory in 30 -seconds on our partition machines. - -The DVO software system allows the tables which are partitioned across -the sky to also be distributed across multiple computers, which we -call partition hosts. A single file defines the names of these -partition hosts and the location of the database partition on the -disks of that machine. The \code{SkyTable} contains elements to -define by ID the parition host to which a partitioned set of tables -has been assigned. Operations which query the database, or perform -other operations on the database, are aware of the partitioning scheme -and will launch their operations as remote processes on the machines -which contain the data they need. For example, a query for data from -a small region will launch sub-query operations on the machines which -contain the data overlapping the region of interest. These remote -query operations will select the database information which matches -the query request (i.e., applying restrictions as defined) and return -to the master process the results. The results from the various -partition hosts are then merged into a single result by the master -process. This parallelization is critical to querying and -manipulating the enormous database on a reasonable timescale. - -\subsection{Tables which describe objects} - -Two tables carry the most important information about the astronomical -objects in the database: Average and SecFilt. These two tables -specify the main average properties of the astronomical object. The -Average table includes the astrometric information ($\alpha, \delta, -\mu \alpha, \mu \delta, \pi$) and associated errors, data quality -flags for each object, links to the other tables, and a number of IDs, -with one row for each astronomical object. \note{go into complete - detail here on the IDs?}. The SecFilt table\footnote{The name - SecFilt is a bit of a historical misnomer: originally, DVO was - designed for a monochromatic survey and data for a single - photometric band was maintained in the Average table. Later, DVO - was adapted to a multifilter system and additional filters were - added to the SecFilt (Secondary Filter) table. Eventually, the - schema was normalized and all photometric data placed in SecFilt, - with the Primary filter concept being dropped, but the name has - since stuck.} contains average photometric information for a -collection of filters. A given DVO instance has a specified set of -filters for which average photometry is stored in the SecFilt table. -The number and choice of filters for the SecFilt may be modified by -the database administrator fairly easily, but the process of updating -the database is somewhat expensive (\approx 24 hours for the current -PV3 instance). Thus the choice is semi-static for a given DVO -instance. In the case of the PV3 DVO instance, 9 average bandpasses -are defined: {\it g, r, i, z, y, J, H, K, w}. The SecFilt table -contains one row for each filter for each object, thus the PV3 DVO -contains 9 times as many rows as the Average table. The order of the -table is fixed in relation to the Average table: row $i$ of Average -defines the object with photometry contained in rows $9i \rightarrow 9i + -8$ ($i$ is zero counting). - -The individual measurements of the astronomical objects are carried in -the table \code{Measure}. This table lists the values measured by -\code{psphot} for each chip, warp, or stack image. This includes the -instrumental magnitudes for the PSF, aperture, and Kron photometry; -raw position (Xccd, Yccd) and second moments (Mxx, Myy, Mxy), along -with shape parameters of the PSF model at the position of the object -(FWx, FWy, theta). This table also includes metadata such as the -exposure time, the date \& time of the observation, airmass, azimuth, -and information specifying the filter \note{describe the photcodes}. -The \code{Measure} table also carried the calibration magnitude offsts -($M_{\rm cal}$ and $M_{\rm flat}$ discussed below) and the -astrometrically calibrated position. Astrometric offsets for several -systematic corrections discussed below are also defined for each -measurement. Since stacks and forced warp photometry may have -non-significant values, the table is somewhat de-normalized in that it -also carried instrumental flux values for the PSF, aperture, and Kron -photometry. - -In the \code{Measure} table, there are three fields which provide two -independent links from the specific measurement to the associated -object: \code{Measure.catID} specifies the spatial partition to which -the measurement belongs; \code{Measure.objID} specifies to which entry -in the \code{Average} table the measurement belongs. These two 32 bit -fields can thus be combined into a single 64 bit unique ID for all -objects in the database. In addition, the field \code{Measure.averef} -specifies the row number in the \code{Average} table of the associated -object. The \code{Measure} table may be unsorted, in which case it is -slow to find the measurements associated with a specific object (a -full table scan is required). After the table is sorted, the -\code{Measure} rows for a given object are grouped together. In the -case, the fields \code{Average.measureOffset} and -\code{Average.Nmeasure} define an index for the code to jump to the -list of measurements for a single object. The field -\code{Measure.imageID} defines the link from the measurement to the -image which supplied the measurement. - -\note{some discussion of the db construction, addstar, dvomerge, etc?} - -For the warp images, we also measure the weak lensing KSB parameters -related to the shear and smear tensors. These measurements are stored -in the \code{Lensing} table, along with the radial aperture fluxes for -radii numbers 5, 6, \& 7 (XX, XX, XX arcsec). This table contains one -row for every warp row. Similarly to the \code{Measure} table, the fields -\code{objID}, \code{catID}, and \code{averef} define links from the -\code{Lensing} table to the \code{Average} table. In a similar -fashion, the fields \code{Average.lensingOffset} and -\code{Average.Nlensing} are the index into the sorted \code{Lensing} -table entries. \note{discuss failure of the Lensing to Measure - indexing} - -The values stored in the \code{Lensing} table are used to calculate -average values for each of these types of measurements in each -filter. The \code{Lensobj} table stores the averaged KSB and radial -aperture photometry for each of the 5 filters \grizy. This table -contains one entry per object per filter. The table is not generally -stored unsorted as it is calculated after the full database is -populated. The link from \code{Average} to \code{Lensobj} is defined -by the fields \code{Average.offsetLensobj} and -\code{Average.Nlensobj}. Each \code{Lensobj} row also includes the -photcode (filter) for which the average lensing (and radial aperture) -properties have been calculated. - -The \code{Galphot} table stores the results of the forced galaxy -fitting analysis for each object that has been measured. This table -contains one row per filter and model type (Sersic, Exponential, -DeVaucouleur) if forced galaxy models have been calculate for the -object. \note{need to expand on this somewhat} - -The \code{Starpar} table carries measurements provide by Greg Green \& -Eddie Schlafly from their analysis of the SED of objects in the PS1 -$3\pi$ data, using the \note{PV1?} version of the analysis (Green et -al REF). In this work, the goal was a 3D model of the dust in the -Galaxy based on Pan-STARRS (\note{and WISE \& 2MASS?}) photometry. As -part of this analysis, the authors fit the SEDs of all \note{stellar?} -sources with stellar models including free parameters of extinction, -distance modulus, metallicity, and absolute r-band magnitude. While -these photometric distance modulus measurements are not extremely -precise (see below), they provide a constraint on the distance is used -in our analysis of the astrometry (see Section~\ref{sec:astrometry}). - -\subsection{Other Tables} - -Data from GPC1 (and other cameras processed by the IPP) are loaded -into DVO in units \code{smf} files generated by the Camera calibration -stage. As described above, these files contain all measurements from -a complete exposure, with measurements from each chip grouped into -separate FITS tables. When these measurements are loaded into the -\code{Measure} and similar tables, a subset of the information from -the chip header is used to populated a row in the DVO \code{Image} -table. This table contains one row for each chip known to DVO, with -information such as the filter (\code{photcode}), the exposure time, -the airmass, the astrometric calibration terms, the photometric -zero point, etc. For GPC1 and other mosaic cameras, an additional row -is defined to carry the projection and camera distortion elements of -the astrometry model. As chips are loaded into this table, they are -assigned an internal ID (a running sequence in the table). Images may -also be assigned an external ID: in the case of the GPC1 images, this -ID is defined by the processing mysql database and is guaranteed to be -unique within the processing system. - -Other tables are used to track information used by the calibration -system. This includes the complete set of flat-field corrections -determined by the photometry calibration analysis (see -Section~\ref{sec:relphot}) and the astrometric flat-field corrections -determined by the astrometry calibration analysis (see Section~\ref{sec:relastro}) - \section{Photometry Calibration}