Index: trunk/doc/release.2015/ps1.analysis/stages.tex =================================================================== --- trunk/doc/release.2015/ps1.analysis/stages.tex (revision 39845) +++ trunk/doc/release.2015/ps1.analysis/stages.tex (revision 39846) @@ -5,4 +5,5 @@ \RequirePackage{color} +\RequirePackage{code} \input{astro.sty} @@ -18,8 +19,8 @@ % Pick a terse version of the title here; -\shorttitle{PS1 Data Processing Stages} +\shorttitle{PS1 Data Processing System} \shortauthors{E.A. Magnier et al} \begin{document} -\title{Pan-STARRS Data Processing Stages} +\title{Pan-STARRS Data Processing System} % this is a crude trick to get the order of affiliations right. These @@ -92,5 +93,7 @@ % \section{INTRODUCTION}\label{sec:intro} -\section{Processing Database} +\section{IPP Software Subsystems} + +\subsection{Processing Database} A critical element in the Pan-STARRS IPP infrastructure is the @@ -161,5 +164,290 @@ crashes. -\section{Download from Summit} +\subsection{Nebulous} + +\subsection{Pantasks \& Parallel Processing} + +\subsection{DVO} + +The Pan-STARRS IPP uses an internal database system, distinct from the +publically visible database system, to determine the association +between multiple detections of the same astronomical object and as +part of the astrometric and photometric calibration process. This +database system, called the ``Desktop Virtual Observatory'' (DVO) was +developed originally for the LONEOS project, and used as part of the +CFHT Elixir system (Magnier \& Cuillandre REF). The capabilities of +this databasing system have been somewhat expanded for the Pan-STARRS +context. + +One of the main purposes of the DVO system is to define the +relationship between individual detections of an astronomical object +and the definition of that object. Before describing the database +schema, we will discuss the process by which detections are associated +with objects. New detections are generally added to the database in a +group associated with, for example, an image from the GPC1 camera. As +new detections are loaded, they are compared to the objects already +stored in the database. If an object is already found in the database +within the match radius, the new detection is associated to that +object. If more than one object exists within the database, the +detection is associated with the closest object. + +Detections in DVO have a special piece of metadata called the +\code{photcode} which identifies the source of the measurement. A +\code{photcode} has a name which in general consists of the name of +the camera or telescope (e.g., GPC1 or 2MASS), the name (or short-hand +name) of the filter used for the measurement (e.g., $g$), and an +identifier for the detector, if not unique (e.g., XY01 for GPC1). +Along with each name, there is a numerical value for the photcode. A +table within the DVO system, \code{Photcode}, lists the photcoes and +defines a number of additional pieces of information for each +photcode. These include the nominal zero point and airmass slope, as +well as color trends to transform a measurement in the specific +photcode to a common system. There are 3 classes of photcodes defined +within the DVO system. Those photcodes associated with detections +from an image loaded into the database system are called \code{DEP} +photcodes. There are also photcodes associated with the average +photometry values, called SEC photcodes. There are also those +measurements which come from external data sources for which DVO does +not have any information to determine a calibration (e.g., +instrumental magnitudes and detector coordinates). These are +measurements are reference values and are assigned REF photcodes. + +In the implementation of DVO used for the PV3 calibration analysis, +the database tables are stored on disk using binary FITS tables. Each +type of database table is stored as a separate file, or a collection +of files for table which are spatially partitioned. The binary FITS +tables may be optionally compressed using the (to date) experimental +FITS binary table compression strategy outlined by REF. In this +compression scheme, using a strategy similar to that used for FITS +image compression (REF), the data stored in the binary table is +compressed and stored in the 'HEAP' section of the FITS table. In +brief, each column in the FITS table is compressed as one (or more) +blocks. The standard fields which describe the data column format +(e.g., TFORM1) are replaced with columns which describe the location +and size of the compressed data in the HEAP section; the information +about the uncompressed data is moved to a field with 'Z' prepended +(e.g., ZFORM1) and an additional field is added to define the +compression algorithm (e.g., ZCTYP1). The column names (e.g., TTYPE1) +and units (e.g., TUNIT1) are retained in their original form. The +compression algorithm can treat the entire column as a single block of +data, or it may be broken into a number of chunks, each compressed in +turn (this must be the same for all columns). Additional header +information is added to describe the block sizes and infomation needed +to describe the HEAP data section. The compression algorithms +currently defined consist of the GZIP, RICE, PLIO, and HCOMPRESS +(REFS). For GZIP, the compression algorithm may transpose the byte +order before compression: for floating point data of a similiar +dynamic range, this choice may allow for better compression as each +byte in the 4 or 8 byte floating point value is more likely to be +similar to the same byte in other rows than to the other bytes of the +same row value. This option is called \code{GZIP_2} in the FITS +standard, as opposed to the standard order, \code{GZIP_1}. The DVO +system can be set to specify the compression options for each column +in the tables. In practice, we have chosen a default in which +floating point numbers used \code{GZIP_2}, character strings use +\code{GZIP_1}, integers use \code{RICE}. + +\subsubsection{Sky Partition} + +DVO includes two major classes of database tables: those containing +information directly about astronomical objects in the sky and those +containing other supporting information. The object-related tables +are partitioned on the basis of position in the sky: objects within a +region bounded by lines of constant RA,DEC are contained in a specific +file. The boundaries and the associated partition names are stored in +one of the supporting tables, \code{SkyTable}. This table contains +the definitions of the boundaries for each sky region (\code{R_MIN}, +\code{R_MAX}, \code{D_MIN}, \code{D_MAX}), the name of the sky region, +an ID (\code{INDEX}, equal to the sequence number of the region in the +table), and index entries to enable navigation within the table. The +regions are defined in a hierarchical sense, with a series of levels +each containing a finer mesh of regions covering the sky. + +In the default used by the PV3 DVO, the partitioning scheme is based +on the one used by the Hubble Space Telescope Guide Star Catalog +files. Level 0 is a single region covering the full sky. Level 1 +divides the sky in Declination into bands 7.5\degree\ high. Level 2 +subdivides these Declination bands in the RA direction, with spacing +related to the stellar density. Level 3 divides these RA chunks into +4 - 8 smaller partitions. This level exactly matches the HST GSC +layout, and uses the same naming convention to identify the +partitions: n0000/0000, etc. \note{more on the names?}. Level 4 +further divides these regions by a factor of 16. In the +\code{SkyTable}, a region at one level has a pointer to its parent +region (the one which contains it) and a sequence pointing to its +children (regions it contains). The \code{SkyTable} enables fast +lookups of the on-disk partitions which map to a specific coordinate +on the sky. In general, a single DVO will have the full sky +represented with tables at a single level, though it is possible for +mixed levels to be used, this mode is not well tested. For the PV3 +master database, the partitioning at the 5th level results in \approx +150,000 regions to cover the full sky, of which \approx 110,000 are +used for the PV3 $3\pi$ data. The densest portions of the bulge +contain at most \approx 300k astronomical objects in the database +files, with an associated maximum of 30M measurements in these files. +With the compression scheme described above, this makes the largest +database files \approx 3GB, which can be loaded into memory in 30 +seconds on our partition machines. + +The DVO software system allows the tables which are partitioned across +the sky to also be distributed across multiple computers, which we +call partition hosts. A single file defines the names of these +partition hosts and the location of the database partition on the +disks of that machine. The \code{SkyTable} contains elements to +define by ID the parition host to which a partitioned set of tables +has been assigned. Operations which query the database, or perform +other operations on the database, are aware of the partitioning scheme +and will launch their operations as remote processes on the machines +which contain the data they need. For example, a query for data from +a small region will launch sub-query operations on the machines which +contain the data overlapping the region of interest. These remote +query operations will select the database information which matches +the query request (i.e., applying restrictions as defined) and return +to the master process the results. The results from the various +partition hosts are then merged into a single result by the master +process. This parallelization is critical to querying and +manipulating the enormous database on a reasonable timescale. + +\subsection{Tables which describe objects} + +Two tables carry the most important information about the astronomical +objects in the database: Average and SecFilt. These two tables +specify the main average properties of the astronomical object. The +Average table includes the astrometric information ($\alpha, \delta, +\mu \alpha, \mu \delta, \pi$) and associated errors, data quality +flags for each object, links to the other tables, and a number of IDs, +with one row for each astronomical object. \note{go into complete + detail here on the IDs?}. The SecFilt table\footnote{The name + SecFilt is a bit of a historical misnomer: originally, DVO was + designed for a monochromatic survey and data for a single + photometric band was maintained in the Average table. Later, DVO + was adapted to a multifilter system and additional filters were + added to the SecFilt (Secondary Filter) table. Eventually, the + schema was normalized and all photometric data placed in SecFilt, + with the Primary filter concept being dropped, but the name has + since stuck.} contains average photometric information for a +collection of filters. A given DVO instance has a specified set of +filters for which average photometry is stored in the SecFilt table. +The number and choice of filters for the SecFilt may be modified by +the database administrator fairly easily, but the process of updating +the database is somewhat expensive (\approx 24 hours for the current +PV3 instance). Thus the choice is semi-static for a given DVO +instance. In the case of the PV3 DVO instance, 9 average bandpasses +are defined: {\it g, r, i, z, y, J, H, K, w}. The SecFilt table +contains one row for each filter for each object, thus the PV3 DVO +contains 9 times as many rows as the Average table. The order of the +table is fixed in relation to the Average table: row $i$ of Average +defines the object with photometry contained in rows $9i \rightarrow 9i + +8$ ($i$ is zero counting). + +The individual measurements of the astronomical objects are carried in +the table \code{Measure}. This table lists the values measured by +\code{psphot} for each chip, warp, or stack image. This includes the +instrumental magnitudes for the PSF, aperture, and Kron photometry; +raw position (Xccd, Yccd) and second moments (Mxx, Myy, Mxy), along +with shape parameters of the PSF model at the position of the object +(FWx, FWy, theta). This table also includes metadata such as the +exposure time, the date \& time of the observation, airmass, azimuth, +and information specifying the filter \note{describe the photcodes}. +The \code{Measure} table also carried the calibration magnitude offsts +($M_{\rm cal}$ and $M_{\rm flat}$ discussed below) and the +astrometrically calibrated position. Astrometric offsets for several +systematic corrections discussed below are also defined for each +measurement. Since stacks and forced warp photometry may have +non-significant values, the table is somewhat de-normalized in that it +also carried instrumental flux values for the PSF, aperture, and Kron +photometry. + +In the \code{Measure} table, there are three fields which provide two +independent links from the specific measurement to the associated +object: \code{Measure.catID} specifies the spatial partition to which +the measurement belongs; \code{Measure.objID} specifies to which entry +in the \code{Average} table the measurement belongs. These two 32 bit +fields can thus be combined into a single 64 bit unique ID for all +objects in the database. In addition, the field \code{Measure.averef} +specifies the row number in the \code{Average} table of the associated +object. The \code{Measure} table may be unsorted, in which case it is +slow to find the measurements associated with a specific object (a +full table scan is required). After the table is sorted, the +\code{Measure} rows for a given object are grouped together. In the +case, the fields \code{Average.measureOffset} and +\code{Average.Nmeasure} define an index for the code to jump to the +list of measurements for a single object. The field +\code{Measure.imageID} defines the link from the measurement to the +image which supplied the measurement. + +\note{some discussion of the db construction, addstar, dvomerge, etc?} + +For the warp images, we also measure the weak lensing KSB parameters +related to the shear and smear tensors. These measurements are stored +in the \code{Lensing} table, along with the radial aperture fluxes for +radii numbers 5, 6, \& 7 (XX, XX, XX arcsec). This table contains one +row for every warp row. Similarly to the \code{Measure} table, the fields +\code{objID}, \code{catID}, and \code{averef} define links from the +\code{Lensing} table to the \code{Average} table. In a similar +fashion, the fields \code{Average.lensingOffset} and +\code{Average.Nlensing} are the index into the sorted \code{Lensing} +table entries. \note{discuss failure of the Lensing to Measure + indexing} + +The values stored in the \code{Lensing} table are used to calculate +average values for each of these types of measurements in each +filter. The \code{Lensobj} table stores the averaged KSB and radial +aperture photometry for each of the 5 filters \grizy. This table +contains one entry per object per filter. The table is not generally +stored unsorted as it is calculated after the full database is +populated. The link from \code{Average} to \code{Lensobj} is defined +by the fields \code{Average.offsetLensobj} and +\code{Average.Nlensobj}. Each \code{Lensobj} row also includes the +photcode (filter) for which the average lensing (and radial aperture) +properties have been calculated. + +The \code{Galphot} table stores the results of the forced galaxy +fitting analysis for each object that has been measured. This table +contains one row per filter and model type (Sersic, Exponential, +DeVaucouleur) if forced galaxy models have been calculate for the +object. \note{need to expand on this somewhat} + +The \code{Starpar} table carries measurements provide by Greg Green \& +Eddie Schlafly from their analysis of the SED of objects in the PS1 +$3\pi$ data, using the \note{PV1?} version of the analysis (Green et +al REF). In this work, the goal was a 3D model of the dust in the +Galaxy based on Pan-STARRS (\note{and WISE \& 2MASS?}) photometry. As +part of this analysis, the authors fit the SEDs of all \note{stellar?} +sources with stellar models including free parameters of extinction, +distance modulus, metallicity, and absolute r-band magnitude. While +these photometric distance modulus measurements are not extremely +precise (see below), they provide a constraint on the distance is used +in our analysis of the astrometry (see Section~\ref{sec:astrometry}). + +\subsection{Other Tables} + +Data from GPC1 (and other cameras processed by the IPP) are loaded +into DVO in units \code{smf} files generated by the Camera calibration +stage. As described above, these files contain all measurements from +a complete exposure, with measurements from each chip grouped into +separate FITS tables. When these measurements are loaded into the +\code{Measure} and similar tables, a subset of the information from +the chip header is used to populated a row in the DVO \code{Image} +table. This table contains one row for each chip known to DVO, with +information such as the filter (\code{photcode}), the exposure time, +the airmass, the astrometric calibration terms, the photometric +zero point, etc. For GPC1 and other mosaic cameras, an additional row +is defined to carry the projection and camera distortion elements of +the astrometry model. As chips are loaded into this table, they are +assigned an internal ID (a running sequence in the table). Images may +also be assigned an external ID: in the case of the GPC1 images, this +ID is defined by the processing mysql database and is guaranteed to be +unique within the processing system. + +Other tables are used to track information used by the calibration +system. This includes the complete set of flat-field corrections +determined by the photometry calibration analysis (see +Section~\ref{sec:relphot}) and the astrometric flat-field corrections +determined by the astrometry calibration analysis (see Section~\ref{sec:relastro}) + +\section{IPP Data Processing Stages} + +\subsection{Download from Summit} As exposures are taken by the PS1 telescope \& camera system, the 60 @@ -197,5 +485,5 @@ chips. -\section{Image Registration} +\subsection{Image Registration} Once chips for an exposure have all been downloaded, the exposure is @@ -223,5 +511,5 @@ database tables (rawExp and rawImfile). -\section{Chip Processing} +\subsection{Chip Processing} The science analysis of an exposure begins with the processing of the @@ -260,5 +548,5 @@ the processing monitor tool. -\section{Camera Calibration} +\subsection{Camera Calibration} After sources have been detected and measured for each of the chip, @@ -300,5 +588,5 @@ monitoring system to visualize the data processing. -\section{Warp} +\subsection{Warp} Once astrometric and photometric calibrations have been performed, @@ -316,5 +604,5 @@ available} from the image extraction tools \note{in DR2}. -\section{Stack} +\subsection{Stack} The skycell images generated by the Warp process are added together to @@ -348,5 +636,5 @@ transients from a given season. -\section{Stack Photometry} +\subsection{Stack Photometry} The stack images are generated in the Stack stage of the IPP, but the @@ -389,5 +677,5 @@ is used for the Camera and Stack calibration stages. -\section{Forced Warp Photometry} +\subsection{Forced Warp Photometry} Traditionally, projects which use multiple exposures to increase the @@ -462,5 +750,5 @@ measurement as the signal-to-noise increases by $\sqrt{N}$. -\section{Forced Galaxy Models} +\subsection{Forced Galaxy Models} The convolved galaxy models are also re-measured on the warp images by @@ -515,5 +803,5 @@ and objects}. -\section{Difference Images} +\subsection{Difference Images} Two of the primary science drivers for the Pan-STARRS system are the @@ -548,10 +836,19 @@ diffs'. -\begin{verbatim} -DVO Ingest -Calibration -IPP to PSPS -PSPS Load & Merge -\end{verbatim} +\subsection{Addstar : DVO Ingest} + +\subsection{Calibration Operations} + +\subsection{IPP to PSPS} + +\subsection{PSPS Load \& Merge} + +\section{IPP Hardware Systems} + +\subsection{Kihei Processing Cluster} + +\subsection{Los Alamos National Labs} + +\subsection{UH Cray Cluster} \end{document} Index: trunk/doc/release.2015/ps1.calibration/calibration.tex =================================================================== --- trunk/doc/release.2015/ps1.calibration/calibration.tex (revision 39845) +++ trunk/doc/release.2015/ps1.calibration/calibration.tex (revision 39846) @@ -503,283 +503,4 @@ the data from the exposure are loaded into the DVO database. -\section{DVO Description} - -The Pan-STARRS IPP uses an internal database system, distinct from the -publically visible database system, to determine the association -between multiple detections of the same astronomical object and as -part of the astrometric and photometric calibration process. This -database system, called the ``Desktop Virtual Observatory'' (DVO) was -developed originally for the LONEOS project, and used as part of the -CFHT Elixir system (Magnier \& Cuillandre REF). The capabilities of -this databasing system have been somewhat expanded for the Pan-STARRS -context. - -One of the main purposes of the DVO system is to define the -relationship between individual detections of an astronomical object -and the definition of that object. Before describing the database -schema, we will discuss the process by which detections are associated -with objects. New detections are generally added to the database in a -group associated with, for example, an image from the GPC1 camera. As -new detections are loaded, they are compared to the objects already -stored in the database. If an object is already found in the database -within the match radius, the new detection is associated to that -object. If more than one object exists within the database, the -detection is associated with the closest object. - -Detections in DVO have a special piece of metadata called the -\code{photcode} which identifies the source of the measurement. A -\code{photcode} has a name which in general consists of the name of -the camera or telescope (e.g., GPC1 or 2MASS), the name (or short-hand -name) of the filter used for the measurement (e.g., $g$), and an -identifier for the detector, if not unique (e.g., XY01 for GPC1). -Along with each name, there is a numerical value for the photcode. A -table within the DVO system, \code{Photcode}, lists the photcoes and -defines a number of additional pieces of information for each -photcode. These include the nominal zero point and airmass slope, as -well as color trends to transform a measurement in the specific -photcode to a common system. There are 3 classes of photcodes defined -within the DVO system. Those photcodes associated with detections -from an image loaded into the database system are called \code{DEP} -photcodes. There are also photcodes associated with the average -photometry values, called SEC photcodes. There are also those -measurements which come from external data sources for which DVO does -not have any information to determine a calibration (e.g., -instrumental magnitudes and detector coordinates). These are -measurements are reference values and are assigned REF photcodes. - -In the implementation of DVO used for the PV3 calibration analysis, -the database tables are stored on disk using binary FITS tables. Each -type of database table is stored as a separate file, or a collection -of files for table which are spatially partitioned. The binary FITS -tables may be optionally compressed using the (to date) experimental -FITS binary table compression strategy outlined by REF. In this -compression scheme, using a strategy similar to that used for FITS -image compression (REF), the data stored in the binary table is -compressed and stored in the 'HEAP' section of the FITS table. In -brief, each column in the FITS table is compressed as one (or more) -blocks. The standard fields which describe the data column format -(e.g., TFORM1) are replaced with columns which describe the location -and size of the compressed data in the HEAP section; the information -about the uncompressed data is moved to a field with 'Z' prepended -(e.g., ZFORM1) and an additional field is added to define the -compression algorithm (e.g., ZCTYP1). The column names (e.g., TTYPE1) -and units (e.g., TUNIT1) are retained in their original form. The -compression algorithm can treat the entire column as a single block of -data, or it may be broken into a number of chunks, each compressed in -turn (this must be the same for all columns). Additional header -information is added to describe the block sizes and infomation needed -to describe the HEAP data section. The compression algorithms -currently defined consist of the GZIP, RICE, PLIO, and HCOMPRESS -(REFS). For GZIP, the compression algorithm may transpose the byte -order before compression: for floating point data of a similiar -dynamic range, this choice may allow for better compression as each -byte in the 4 or 8 byte floating point value is more likely to be -similar to the same byte in other rows than to the other bytes of the -same row value. This option is called \code{GZIP_2} in the FITS -standard, as opposed to the standard order, \code{GZIP_1}. The DVO -system can be set to specify the compression options for each column -in the tables. In practice, we have chosen a default in which -floating point numbers used \code{GZIP_2}, character strings use -\code{GZIP_1}, integers use \code{RICE}. - -\subsubsection{Sky Partition} - -DVO includes two major classes of database tables: those containing -information directly about astronomical objects in the sky and those -containing other supporting information. The object-related tables -are partitioned on the basis of position in the sky: objects within a -region bounded by lines of constant RA,DEC are contained in a specific -file. The boundaries and the associated partition names are stored in -one of the supporting tables, \code{SkyTable}. This table contains -the definitions of the boundaries for each sky region (\code{R_MIN}, -\code{R_MAX}, \code{D_MIN}, \code{D_MAX}), the name of the sky region, -an ID (\code{INDEX}, equal to the sequence number of the region in the -table), and index entries to enable navigation within the table. The -regions are defined in a hierarchical sense, with a series of levels -each containing a finer mesh of regions covering the sky. - -In the default used by the PV3 DVO, the partitioning scheme is based -on the one used by the Hubble Space Telescope Guide Star Catalog -files. Level 0 is a single region covering the full sky. Level 1 -divides the sky in Declination into bands 7.5\degree\ high. Level 2 -subdivides these Declination bands in the RA direction, with spacing -related to the stellar density. Level 3 divides these RA chunks into -4 - 8 smaller partitions. This level exactly matches the HST GSC -layout, and uses the same naming convention to identify the -partitions: n0000/0000, etc. \note{more on the names?}. Level 4 -further divides these regions by a factor of 16. In the -\code{SkyTable}, a region at one level has a pointer to its parent -region (the one which contains it) and a sequence pointing to its -children (regions it contains). The \code{SkyTable} enables fast -lookups of the on-disk partitions which map to a specific coordinate -on the sky. In general, a single DVO will have the full sky -represented with tables at a single level, though it is possible for -mixed levels to be used, this mode is not well tested. For the PV3 -master database, the partitioning at the 5th level results in \approx -150,000 regions to cover the full sky, of which \approx 110,000 are -used for the PV3 $3\pi$ data. The densest portions of the bulge -contain at most \approx 300k astronomical objects in the database -files, with an associated maximum of 30M measurements in these files. -With the compression scheme described above, this makes the largest -database files \approx 3GB, which can be loaded into memory in 30 -seconds on our partition machines. - -The DVO software system allows the tables which are partitioned across -the sky to also be distributed across multiple computers, which we -call partition hosts. A single file defines the names of these -partition hosts and the location of the database partition on the -disks of that machine. The \code{SkyTable} contains elements to -define by ID the parition host to which a partitioned set of tables -has been assigned. Operations which query the database, or perform -other operations on the database, are aware of the partitioning scheme -and will launch their operations as remote processes on the machines -which contain the data they need. For example, a query for data from -a small region will launch sub-query operations on the machines which -contain the data overlapping the region of interest. These remote -query operations will select the database information which matches -the query request (i.e., applying restrictions as defined) and return -to the master process the results. The results from the various -partition hosts are then merged into a single result by the master -process. This parallelization is critical to querying and -manipulating the enormous database on a reasonable timescale. - -\subsection{Tables which describe objects} - -Two tables carry the most important information about the astronomical -objects in the database: Average and SecFilt. These two tables -specify the main average properties of the astronomical object. The -Average table includes the astrometric information ($\alpha, \delta, -\mu \alpha, \mu \delta, \pi$) and associated errors, data quality -flags for each object, links to the other tables, and a number of IDs, -with one row for each astronomical object. \note{go into complete - detail here on the IDs?}. The SecFilt table\footnote{The name - SecFilt is a bit of a historical misnomer: originally, DVO was - designed for a monochromatic survey and data for a single - photometric band was maintained in the Average table. Later, DVO - was adapted to a multifilter system and additional filters were - added to the SecFilt (Secondary Filter) table. Eventually, the - schema was normalized and all photometric data placed in SecFilt, - with the Primary filter concept being dropped, but the name has - since stuck.} contains average photometric information for a -collection of filters. A given DVO instance has a specified set of -filters for which average photometry is stored in the SecFilt table. -The number and choice of filters for the SecFilt may be modified by -the database administrator fairly easily, but the process of updating -the database is somewhat expensive (\approx 24 hours for the current -PV3 instance). Thus the choice is semi-static for a given DVO -instance. In the case of the PV3 DVO instance, 9 average bandpasses -are defined: {\it g, r, i, z, y, J, H, K, w}. The SecFilt table -contains one row for each filter for each object, thus the PV3 DVO -contains 9 times as many rows as the Average table. The order of the -table is fixed in relation to the Average table: row $i$ of Average -defines the object with photometry contained in rows $9i \rightarrow 9i + -8$ ($i$ is zero counting). - -The individual measurements of the astronomical objects are carried in -the table \code{Measure}. This table lists the values measured by -\code{psphot} for each chip, warp, or stack image. This includes the -instrumental magnitudes for the PSF, aperture, and Kron photometry; -raw position (Xccd, Yccd) and second moments (Mxx, Myy, Mxy), along -with shape parameters of the PSF model at the position of the object -(FWx, FWy, theta). This table also includes metadata such as the -exposure time, the date \& time of the observation, airmass, azimuth, -and information specifying the filter \note{describe the photcodes}. -The \code{Measure} table also carried the calibration magnitude offsts -($M_{\rm cal}$ and $M_{\rm flat}$ discussed below) and the -astrometrically calibrated position. Astrometric offsets for several -systematic corrections discussed below are also defined for each -measurement. Since stacks and forced warp photometry may have -non-significant values, the table is somewhat de-normalized in that it -also carried instrumental flux values for the PSF, aperture, and Kron -photometry. - -In the \code{Measure} table, there are three fields which provide two -independent links from the specific measurement to the associated -object: \code{Measure.catID} specifies the spatial partition to which -the measurement belongs; \code{Measure.objID} specifies to which entry -in the \code{Average} table the measurement belongs. These two 32 bit -fields can thus be combined into a single 64 bit unique ID for all -objects in the database. In addition, the field \code{Measure.averef} -specifies the row number in the \code{Average} table of the associated -object. The \code{Measure} table may be unsorted, in which case it is -slow to find the measurements associated with a specific object (a -full table scan is required). After the table is sorted, the -\code{Measure} rows for a given object are grouped together. In the -case, the fields \code{Average.measureOffset} and -\code{Average.Nmeasure} define an index for the code to jump to the -list of measurements for a single object. The field -\code{Measure.imageID} defines the link from the measurement to the -image which supplied the measurement. - -\note{some discussion of the db construction, addstar, dvomerge, etc?} - -For the warp images, we also measure the weak lensing KSB parameters -related to the shear and smear tensors. These measurements are stored -in the \code{Lensing} table, along with the radial aperture fluxes for -radii numbers 5, 6, \& 7 (XX, XX, XX arcsec). This table contains one -row for every warp row. Similarly to the \code{Measure} table, the fields -\code{objID}, \code{catID}, and \code{averef} define links from the -\code{Lensing} table to the \code{Average} table. In a similar -fashion, the fields \code{Average.lensingOffset} and -\code{Average.Nlensing} are the index into the sorted \code{Lensing} -table entries. \note{discuss failure of the Lensing to Measure - indexing} - -The values stored in the \code{Lensing} table are used to calculate -average values for each of these types of measurements in each -filter. The \code{Lensobj} table stores the averaged KSB and radial -aperture photometry for each of the 5 filters \grizy. This table -contains one entry per object per filter. The table is not generally -stored unsorted as it is calculated after the full database is -populated. The link from \code{Average} to \code{Lensobj} is defined -by the fields \code{Average.offsetLensobj} and -\code{Average.Nlensobj}. Each \code{Lensobj} row also includes the -photcode (filter) for which the average lensing (and radial aperture) -properties have been calculated. - -The \code{Galphot} table stores the results of the forced galaxy -fitting analysis for each object that has been measured. This table -contains one row per filter and model type (Sersic, Exponential, -DeVaucouleur) if forced galaxy models have been calculate for the -object. \note{need to expand on this somewhat} - -The \code{Starpar} table carries measurements provide by Greg Green \& -Eddie Schlafly from their analysis of the SED of objects in the PS1 -$3\pi$ data, using the \note{PV1?} version of the analysis (Green et -al REF). In this work, the goal was a 3D model of the dust in the -Galaxy based on Pan-STARRS (\note{and WISE \& 2MASS?}) photometry. As -part of this analysis, the authors fit the SEDs of all \note{stellar?} -sources with stellar models including free parameters of extinction, -distance modulus, metallicity, and absolute r-band magnitude. While -these photometric distance modulus measurements are not extremely -precise (see below), they provide a constraint on the distance is used -in our analysis of the astrometry (see Section~\ref{sec:astrometry}). - -\subsection{Other Tables} - -Data from GPC1 (and other cameras processed by the IPP) are loaded -into DVO in units \code{smf} files generated by the Camera calibration -stage. As described above, these files contain all measurements from -a complete exposure, with measurements from each chip grouped into -separate FITS tables. When these measurements are loaded into the -\code{Measure} and similar tables, a subset of the information from -the chip header is used to populated a row in the DVO \code{Image} -table. This table contains one row for each chip known to DVO, with -information such as the filter (\code{photcode}), the exposure time, -the airmass, the astrometric calibration terms, the photometric -zero point, etc. For GPC1 and other mosaic cameras, an additional row -is defined to carry the projection and camera distortion elements of -the astrometry model. As chips are loaded into this table, they are -assigned an internal ID (a running sequence in the table). Images may -also be assigned an external ID: in the case of the GPC1 images, this -ID is defined by the processing mysql database and is guaranteed to be -unique within the processing system. - -Other tables are used to track information used by the calibration -system. This includes the complete set of flat-field corrections -determined by the photometry calibration analysis (see -Section~\ref{sec:relphot}) and the astrometric flat-field corrections -determined by the astrometry calibration analysis (see Section~\ref{sec:relastro}) - \section{Photometry Calibration}