Changeset 39846

Timestamp:

Dec 11, 2016, 9:09:09 PM (10 years ago)

Author:

eugene

Message:

updates

Location:

trunk/doc/release.2015

Files:

• ps1.analysis/stages.tex (modified) (14 diffs)
• ps1.calibration/calibration.tex (modified) (1 diff)

trunk/doc/release.2015/ps1.analysis/stages.tex

@@ -5 +5 @@
 
 \RequirePackage{color}
+\RequirePackage{code}
 \input{astro.sty}
 
@@ -18 +19 @@
 
 % Pick a terse version of the title here;
-\shorttitle{PS1 Data Processing Stages}
+\shorttitle{PS1 Data Processing System}
 \shortauthors{E.A. Magnier et al}
 \begin{document}
-\title{Pan-STARRS Data Processing Stages}
+\title{Pan-STARRS Data Processing System}
 
 % this is a crude trick to get the order of affiliations right.  These
@@ -92 +93 @@
 % \section{INTRODUCTION}\label{sec:intro}
 
-\section{Processing Database}
+\section{IPP Software Subsystems}
+
+\subsection{Processing Database}
 
 A critical element in the Pan-STARRS IPP infrastructure is the
@@ -161 +164 @@
 crashes.
 
-\section{Download from Summit}
+\subsection{Nebulous}
+
+\subsection{Pantasks \& Parallel Processing}
+
+\subsection{DVO}
+
+The Pan-STARRS IPP uses an internal database system, distinct from the
+publically visible database system, to determine the association
+between multiple detections of the same astronomical object and as
+part of the astrometric and photometric calibration process.  This
+database system, called the ``Desktop Virtual Observatory'' (DVO) was
+developed originally for the LONEOS project, and used as part of the
+CFHT Elixir system (Magnier \& Cuillandre REF).  The capabilities of
+this databasing system have been somewhat expanded for the Pan-STARRS
+context.  
+
+One of the main purposes of the DVO system is to define the
+relationship between individual detections of an astronomical object
+and the definition of that object.  Before describing the database
+schema, we will discuss the process by which detections are associated
+with objects.  New detections are generally added to the database in a
+group associated with, for example, an image from the GPC1 camera.  As
+new detections are loaded, they are compared to the objects already
+stored in the database.  If an object is already found in the database
+within the match radius, the new detection is associated to that
+object. If more than one object exists within the database, the
+detection is associated with the closest object.  
+
+Detections in DVO have a special piece of metadata called the
+\code{photcode} which identifies the source of the measurement.  A
+\code{photcode} has a name which in general consists of the name of
+the camera or telescope (e.g., GPC1 or 2MASS), the name (or short-hand
+name) of the filter used for the measurement (e.g., $g$), and an
+identifier for the detector, if not unique (e.g., XY01 for GPC1).
+Along with each name, there is a numerical value for the photcode.  A
+table within the DVO system, \code{Photcode}, lists the photcoes and
+defines a number of additional pieces of information for each
+photcode.  These include the nominal zero point and airmass slope, as
+well as color trends to transform a measurement in the specific
+photcode to a common system.  There are 3 classes of photcodes defined
+within the DVO system.  Those photcodes associated with detections
+from an image loaded into the database system are called \code{DEP}
+photcodes.  There are also photcodes associated with the average
+photometry values, called SEC photcodes.  There are also those
+measurements which come from external data sources for which DVO does
+not have any information to determine a calibration (e.g.,
+instrumental magnitudes and detector coordinates).  These are
+measurements are reference values and are assigned REF photcodes.
+
+In the implementation of DVO used for the PV3 calibration analysis,
+the database tables are stored on disk using binary FITS tables.  Each
+type of database table is stored as a separate file, or a collection
+of files for table which are spatially partitioned.  The binary FITS
+tables may be optionally compressed using the (to date) experimental
+FITS binary table compression strategy outlined by REF.  In this
+compression scheme, using a strategy similar to that used for FITS
+image compression (REF), the data stored in the binary table is
+compressed and stored in the 'HEAP' section of the FITS table.  In
+brief, each column in the FITS table is compressed as one (or more)
+blocks.  The standard fields which describe the data column format
+(e.g., TFORM1) are replaced with columns which describe the location
+and size of the compressed data in the HEAP section; the information
+about the uncompressed data is moved to a field with 'Z' prepended
+(e.g., ZFORM1) and an additional field is added to define the
+compression algorithm (e.g., ZCTYP1).  The column names (e.g., TTYPE1)
+and units (e.g., TUNIT1) are retained in their original form.  The
+compression algorithm can treat the entire column as a single block of
+data, or it may be broken into a number of chunks, each compressed in
+turn (this must be the same for all columns).  Additional header
+information is added to describe the block sizes and infomation needed
+to describe the HEAP data section.  The compression algorithms
+currently defined consist of the GZIP, RICE, PLIO, and HCOMPRESS
+(REFS).  For GZIP, the compression algorithm may transpose the byte
+order before compression: for floating point data of a similiar
+dynamic range, this choice may allow for better compression as each
+byte in the 4 or 8 byte floating point value is more likely to be
+similar to the same byte in other rows than to the other bytes of the
+same row value.  This option is called \code{GZIP_2} in the FITS
+standard, as opposed to the standard order, \code{GZIP_1}.  The DVO
+system can be set to specify the compression options for each column
+in the tables.  In practice, we have chosen a default in which
+floating point numbers used \code{GZIP_2}, character strings use
+\code{GZIP_1}, integers use \code{RICE}.  
+
+\subsubsection{Sky Partition}
+
+DVO includes two major classes of database tables: those containing
+information directly about astronomical objects in the sky and those
+containing other supporting information.  The object-related tables
+are partitioned on the basis of position in the sky: objects within a
+region bounded by lines of constant RA,DEC are contained in a specific
+file.  The boundaries and the associated partition names are stored in
+one of the supporting tables, \code{SkyTable}.  This table contains
+the definitions of the boundaries for each sky region (\code{R_MIN},
+\code{R_MAX}, \code{D_MIN}, \code{D_MAX}), the name of the sky region,
+an ID (\code{INDEX}, equal to the sequence number of the region in the
+table), and index entries to enable navigation within the table.  The
+regions are defined in a hierarchical sense, with a series of levels
+each containing a finer mesh of regions covering the sky.  
+
+In the default used by the PV3 DVO, the partitioning scheme is based
+on the one used by the Hubble Space Telescope Guide Star Catalog
+files.  Level 0 is a single region covering the full sky.  Level 1
+divides the sky in Declination into bands 7.5\degree\ high.  Level 2
+subdivides these Declination bands in the RA direction, with spacing
+related to the stellar density.  Level 3 divides these RA chunks into
+4 - 8 smaller partitions.  This level exactly matches the HST GSC
+layout, and uses the same naming convention to identify the
+partitions: n0000/0000, etc. \note{more on the names?}.  Level 4
+further divides these regions by a factor of 16.  In the
+\code{SkyTable}, a region at one level has a pointer to its parent
+region (the one which contains it) and a sequence pointing to its
+children (regions it contains).  The \code{SkyTable} enables fast
+lookups of the on-disk partitions which map to a specific coordinate
+on the sky.  In general, a single DVO will have the full sky
+represented with tables at a single level, though it is possible for
+mixed levels to be used, this mode is not well tested.  For the PV3
+master database, the partitioning at the 5th level results in \approx
+150,000 regions to cover the full sky, of which \approx 110,000 are
+used for the PV3 $3\pi$ data.  The densest portions of the bulge
+contain at most \approx 300k astronomical objects in the database
+files, with an associated maximum of 30M measurements in these files.
+With the compression scheme described above, this makes the largest
+database files \approx 3GB, which can be loaded into memory in 30
+seconds on our partition machines.
+
+The DVO software system allows the tables which are partitioned across
+the sky to also be distributed across multiple computers, which we
+call partition hosts.  A single file defines the names of these
+partition hosts and the location of the database partition on the
+disks of that machine.  The \code{SkyTable} contains elements to
+define by ID the parition host to which a partitioned set of tables
+has been assigned.  Operations which query the database, or perform
+other operations on the database, are aware of the partitioning scheme
+and will launch their operations as remote processes on the machines
+which contain the data they need.  For example, a query for data from
+a small region will launch sub-query operations on the machines which
+contain the data overlapping the region of interest.  These remote
+query operations will select the database information which matches
+the query request (i.e., applying restrictions as defined) and return
+to the master process the results.  The results from the various
+partition hosts are then merged into a single result by the master
+process.  This parallelization is critical to querying and
+manipulating the enormous database on a reasonable timescale.
+
+\subsection{Tables which describe objects} 
+
+Two tables carry the most important information about the astronomical
+objects in the database: Average and SecFilt.  These two tables
+specify the main average properties of the astronomical object.  The
+Average table includes the astrometric information ($\alpha, \delta,
+\mu \alpha, \mu \delta, \pi$) and associated errors, data quality
+flags for each object, links to the other tables, and a number of IDs,
+with one row for each astronomical object.  \note{go into complete
+  detail here on the IDs?}.  The SecFilt table\footnote{The name
+  SecFilt is a bit of a historical misnomer: originally, DVO was
+  designed for a monochromatic survey and data for a single
+  photometric band was maintained in the Average table.  Later, DVO
+  was adapted to a multifilter system and additional filters were
+  added to the SecFilt (Secondary Filter) table.  Eventually, the
+  schema was normalized and all photometric data placed in SecFilt,
+  with the Primary filter concept being dropped, but the name has
+  since stuck.} contains average photometric information for a
+collection of filters.  A given DVO instance has a specified set of
+filters for which average photometry is stored in the SecFilt table.
+The number and choice of filters for the SecFilt may be modified by
+the database administrator fairly easily, but the process of updating
+the database is somewhat expensive (\approx 24 hours for the current
+PV3 instance).  Thus the choice is semi-static for a given DVO
+instance.  In the case of the PV3 DVO instance, 9 average bandpasses
+are defined: {\it g, r, i, z, y, J, H, K, w}.  The SecFilt table
+contains one row for each filter for each object, thus the PV3 DVO
+contains 9 times as many rows as the Average table.  The order of the
+table is fixed in relation to the Average table: row $i$ of Average
+defines the object with photometry contained in rows $9i \rightarrow 9i +
+8$ ($i$ is zero counting).  
+
+The individual measurements of the astronomical objects are carried in
+the table \code{Measure}.  This table lists the values measured by
+\code{psphot} for each chip, warp, or stack image.  This includes the
+instrumental magnitudes for the PSF, aperture, and Kron photometry;
+raw position (Xccd, Yccd) and second moments (Mxx, Myy, Mxy), along
+with shape parameters of the PSF model at the position of the object
+(FWx, FWy, theta).  This table also includes metadata such as the
+exposure time, the date \& time of the observation, airmass, azimuth,
+and information specifying the filter \note{describe the photcodes}.
+The \code{Measure} table also carried the calibration magnitude offsts
+($M_{\rm cal}$ and $M_{\rm flat}$ discussed below) and the
+astrometrically calibrated position.  Astrometric offsets for several
+systematic corrections discussed below are also defined for each
+measurement.  Since stacks and forced warp photometry may have
+non-significant values, the table is somewhat de-normalized in that it
+also carried instrumental flux values for the PSF, aperture, and Kron
+photometry.  
+
+In the \code{Measure} table, there are three fields which provide two
+independent links from the specific measurement to the associated
+object: \code{Measure.catID} specifies the spatial partition to which
+the measurement belongs; \code{Measure.objID} specifies to which entry
+in the \code{Average} table the measurement belongs.  These two 32 bit
+fields can thus be combined into a single 64 bit unique ID for all
+objects in the database.  In addition, the field \code{Measure.averef}
+specifies the row number in the \code{Average} table of the associated
+object.  The \code{Measure} table may be unsorted, in which case it is
+slow to find the measurements associated with a specific object (a
+full table scan is required).  After the table is sorted, the
+\code{Measure} rows for a given object are grouped together.  In the
+case, the fields \code{Average.measureOffset} and
+\code{Average.Nmeasure} define an index for the code to jump to the
+list of measurements for a single object.  The field
+\code{Measure.imageID} defines the link from the measurement to the
+image which supplied the measurement.
+
+\note{some discussion of the db construction, addstar, dvomerge, etc?}
+
+For the warp images, we also measure the weak lensing KSB parameters
+related to the shear and smear tensors.  These measurements are stored
+in the \code{Lensing} table, along with the radial aperture fluxes for
+radii numbers 5, 6, \& 7 (XX, XX, XX arcsec).  This table contains one
+row for every warp row.  Similarly to the \code{Measure} table, the fields
+\code{objID}, \code{catID}, and \code{averef} define links from the
+\code{Lensing} table to the \code{Average} table.  In a similar
+fashion, the fields \code{Average.lensingOffset} and
+\code{Average.Nlensing} are the index into the sorted \code{Lensing}
+table entries.  \note{discuss failure of the Lensing to Measure
+  indexing}
+
+The values stored in the \code{Lensing} table are used to calculate
+average values for each of these types of measurements in each
+filter.  The \code{Lensobj} table stores the averaged KSB and radial
+aperture photometry for each of the 5 filters \grizy.  This table
+contains one entry per object per filter.  The table is not generally
+stored unsorted as it is calculated after the full database is
+populated.  The link from \code{Average} to \code{Lensobj} is defined
+by the fields \code{Average.offsetLensobj} and
+\code{Average.Nlensobj}.  Each \code{Lensobj} row also includes the
+photcode (filter) for which the average lensing (and radial aperture)
+properties have been calculated. 
+
+The \code{Galphot} table stores the results of the forced galaxy
+fitting analysis for each object that has been measured.  This table
+contains one row per filter and model type (Sersic, Exponential,
+DeVaucouleur) if forced galaxy models have been calculate for the
+object.  \note{need to expand on this somewhat}
+
+The \code{Starpar} table carries measurements provide by Greg Green \&
+Eddie Schlafly from their analysis of the SED of objects in the PS1
+$3\pi$ data, using the \note{PV1?} version of the analysis (Green et
+al REF).  In this work, the goal was a 3D model of the dust in the
+Galaxy based on Pan-STARRS (\note{and WISE \& 2MASS?}) photometry.  As
+part of this analysis, the authors fit the SEDs of all \note{stellar?}
+sources with stellar models including free parameters of extinction,
+distance modulus, metallicity, and absolute r-band magnitude.  While
+these photometric distance modulus measurements are not extremely
+precise (see below), they provide a constraint on the distance is used
+in our analysis of the astrometry (see Section~\ref{sec:astrometry}).
+
+\subsection{Other Tables} 
+
+Data from GPC1 (and other cameras processed by the IPP) are loaded
+into DVO in units \code{smf} files generated by the Camera calibration
+stage.  As described above, these files contain all measurements from
+a complete exposure, with measurements from each chip grouped into
+separate FITS tables.  When these measurements are loaded into the
+\code{Measure} and similar tables, a subset of the information from
+the chip header is used to populated a row in the DVO \code{Image}
+table.  This table contains one row for each chip known to DVO, with
+information such as the filter (\code{photcode}), the exposure time,
+the airmass, the astrometric calibration terms, the photometric
+zero point, etc.  For GPC1 and other mosaic cameras, an additional row
+is defined to carry the projection and camera distortion elements of
+the astrometry model.  As chips are loaded into this table, they are
+assigned an internal ID (a running sequence in the table).  Images may
+also be assigned an external ID: in the case of the GPC1 images, this
+ID is defined by the processing mysql database and is guaranteed to be
+unique within the processing system. 
+
+Other tables are used to track information used by the calibration
+system.  This includes the complete set of flat-field corrections
+determined by the photometry calibration analysis (see
+Section~\ref{sec:relphot}) and the astrometric flat-field corrections
+determined by the astrometry calibration analysis (see Section~\ref{sec:relastro})
+
+\section{IPP Data Processing Stages}
+
+\subsection{Download from Summit}
 
 As exposures are taken by the PS1 telescope \& camera system, the 60
@@ -197 +485 @@
 chips.  
 
-\section{Image Registration}
+\subsection{Image Registration}
 
 Once chips for an exposure have all been downloaded, the exposure is
@@ -223 +511 @@
 database tables (rawExp and rawImfile).
 
-\section{Chip Processing}
+\subsection{Chip Processing}
 
 The science analysis of an exposure begins with the processing of the
@@ -260 +548 @@
 the processing monitor tool.
 
-\section{Camera Calibration}
+\subsection{Camera Calibration}
 
 After sources have been detected and measured for each of the chip,
@@ -300 +588 @@
 monitoring system to visualize the data processing.
 
-\section{Warp}
+\subsection{Warp}
 
 Once astrometric and photometric calibrations have been performed,
@@ -316 +604 @@
   available} from the image extraction tools \note{in DR2}.
 
-\section{Stack}
+\subsection{Stack}
 
 The skycell images generated by the Warp process are added together to
@@ -348 +636 @@
 transients from a given season.
 
-\section{Stack Photometry}
+\subsection{Stack Photometry}
 
 The stack images are generated in the Stack stage of the IPP, but the
@@ -389 +677 @@
 is used for the Camera and Stack calibration stages.
 
-\section{Forced Warp Photometry}
+\subsection{Forced Warp Photometry}
 
 Traditionally, projects which use multiple exposures to increase the
@@ -462 +750 @@
 measurement as the signal-to-noise increases by $\sqrt{N}$.  
 
-\section{Forced Galaxy Models}
+\subsection{Forced Galaxy Models}
 
 The convolved galaxy models are also re-measured on the warp images by
@@ -515 +803 @@
   and objects}.
 
-\section{Difference Images}
+\subsection{Difference Images}
 
 Two of the primary science drivers for the Pan-STARRS system are the
@@ -548 +836 @@
 diffs'.  
 
-\begin{verbatim}
-DVO Ingest
-Calibration
-IPP to PSPS
-PSPS Load & Merge
-\end{verbatim}
+\subsection{Addstar : DVO Ingest}
+
+\subsection{Calibration Operations}
+
+\subsection{IPP to PSPS}
+
+\subsection{PSPS Load \& Merge}
+
+\section{IPP Hardware Systems}
+
+\subsection{Kihei Processing Cluster} 
+
+\subsection{Los Alamos National Labs} 
+
+\subsection{UH Cray Cluster} 
 
 \end{document}

trunk/doc/release.2015/ps1.calibration/calibration.tex

@@ -503 +503 @@
 the data from the exposure are loaded into the DVO database.
 
-\section{DVO Description}
-
-The Pan-STARRS IPP uses an internal database system, distinct from the
-publically visible database system, to determine the association
-between multiple detections of the same astronomical object and as
-part of the astrometric and photometric calibration process.  This
-database system, called the ``Desktop Virtual Observatory'' (DVO) was
-developed originally for the LONEOS project, and used as part of the
-CFHT Elixir system (Magnier \& Cuillandre REF).  The capabilities of
-this databasing system have been somewhat expanded for the Pan-STARRS
-context.  
-
-One of the main purposes of the DVO system is to define the
-relationship between individual detections of an astronomical object
-and the definition of that object.  Before describing the database
-schema, we will discuss the process by which detections are associated
-with objects.  New detections are generally added to the database in a
-group associated with, for example, an image from the GPC1 camera.  As
-new detections are loaded, they are compared to the objects already
-stored in the database.  If an object is already found in the database
-within the match radius, the new detection is associated to that
-object. If more than one object exists within the database, the
-detection is associated with the closest object.  
-
-Detections in DVO have a special piece of metadata called the
-\code{photcode} which identifies the source of the measurement.  A
-\code{photcode} has a name which in general consists of the name of
-the camera or telescope (e.g., GPC1 or 2MASS), the name (or short-hand
-name) of the filter used for the measurement (e.g., $g$), and an
-identifier for the detector, if not unique (e.g., XY01 for GPC1).
-Along with each name, there is a numerical value for the photcode.  A
-table within the DVO system, \code{Photcode}, lists the photcoes and
-defines a number of additional pieces of information for each
-photcode.  These include the nominal zero point and airmass slope, as
-well as color trends to transform a measurement in the specific
-photcode to a common system.  There are 3 classes of photcodes defined
-within the DVO system.  Those photcodes associated with detections
-from an image loaded into the database system are called \code{DEP}
-photcodes.  There are also photcodes associated with the average
-photometry values, called SEC photcodes.  There are also those
-measurements which come from external data sources for which DVO does
-not have any information to determine a calibration (e.g.,
-instrumental magnitudes and detector coordinates).  These are
-measurements are reference values and are assigned REF photcodes.
-
-In the implementation of DVO used for the PV3 calibration analysis,
-the database tables are stored on disk using binary FITS tables.  Each
-type of database table is stored as a separate file, or a collection
-of files for table which are spatially partitioned.  The binary FITS
-tables may be optionally compressed using the (to date) experimental
-FITS binary table compression strategy outlined by REF.  In this
-compression scheme, using a strategy similar to that used for FITS
-image compression (REF), the data stored in the binary table is
-compressed and stored in the 'HEAP' section of the FITS table.  In
-brief, each column in the FITS table is compressed as one (or more)
-blocks.  The standard fields which describe the data column format
-(e.g., TFORM1) are replaced with columns which describe the location
-and size of the compressed data in the HEAP section; the information
-about the uncompressed data is moved to a field with 'Z' prepended
-(e.g., ZFORM1) and an additional field is added to define the
-compression algorithm (e.g., ZCTYP1).  The column names (e.g., TTYPE1)
-and units (e.g., TUNIT1) are retained in their original form.  The
-compression algorithm can treat the entire column as a single block of
-data, or it may be broken into a number of chunks, each compressed in
-turn (this must be the same for all columns).  Additional header
-information is added to describe the block sizes and infomation needed
-to describe the HEAP data section.  The compression algorithms
-currently defined consist of the GZIP, RICE, PLIO, and HCOMPRESS
-(REFS).  For GZIP, the compression algorithm may transpose the byte
-order before compression: for floating point data of a similiar
-dynamic range, this choice may allow for better compression as each
-byte in the 4 or 8 byte floating point value is more likely to be
-similar to the same byte in other rows than to the other bytes of the
-same row value.  This option is called \code{GZIP_2} in the FITS
-standard, as opposed to the standard order, \code{GZIP_1}.  The DVO
-system can be set to specify the compression options for each column
-in the tables.  In practice, we have chosen a default in which
-floating point numbers used \code{GZIP_2}, character strings use
-\code{GZIP_1}, integers use \code{RICE}.  
-
-\subsubsection{Sky Partition}
-
-DVO includes two major classes of database tables: those containing
-information directly about astronomical objects in the sky and those
-containing other supporting information.  The object-related tables
-are partitioned on the basis of position in the sky: objects within a
-region bounded by lines of constant RA,DEC are contained in a specific
-file.  The boundaries and the associated partition names are stored in
-one of the supporting tables, \code{SkyTable}.  This table contains
-the definitions of the boundaries for each sky region (\code{R_MIN},
-\code{R_MAX}, \code{D_MIN}, \code{D_MAX}), the name of the sky region,
-an ID (\code{INDEX}, equal to the sequence number of the region in the
-table), and index entries to enable navigation within the table.  The
-regions are defined in a hierarchical sense, with a series of levels
-each containing a finer mesh of regions covering the sky.  
-
-In the default used by the PV3 DVO, the partitioning scheme is based
-on the one used by the Hubble Space Telescope Guide Star Catalog
-files.  Level 0 is a single region covering the full sky.  Level 1
-divides the sky in Declination into bands 7.5\degree\ high.  Level 2
-subdivides these Declination bands in the RA direction, with spacing
-related to the stellar density.  Level 3 divides these RA chunks into
-4 - 8 smaller partitions.  This level exactly matches the HST GSC
-layout, and uses the same naming convention to identify the
-partitions: n0000/0000, etc. \note{more on the names?}.  Level 4
-further divides these regions by a factor of 16.  In the
-\code{SkyTable}, a region at one level has a pointer to its parent
-region (the one which contains it) and a sequence pointing to its
-children (regions it contains).  The \code{SkyTable} enables fast
-lookups of the on-disk partitions which map to a specific coordinate
-on the sky.  In general, a single DVO will have the full sky
-represented with tables at a single level, though it is possible for
-mixed levels to be used, this mode is not well tested.  For the PV3
-master database, the partitioning at the 5th level results in \approx
-150,000 regions to cover the full sky, of which \approx 110,000 are
-used for the PV3 $3\pi$ data.  The densest portions of the bulge
-contain at most \approx 300k astronomical objects in the database
-files, with an associated maximum of 30M measurements in these files.
-With the compression scheme described above, this makes the largest
-database files \approx 3GB, which can be loaded into memory in 30
-seconds on our partition machines.
-
-The DVO software system allows the tables which are partitioned across
-the sky to also be distributed across multiple computers, which we
-call partition hosts.  A single file defines the names of these
-partition hosts and the location of the database partition on the
-disks of that machine.  The \code{SkyTable} contains elements to
-define by ID the parition host to which a partitioned set of tables
-has been assigned.  Operations which query the database, or perform
-other operations on the database, are aware of the partitioning scheme
-and will launch their operations as remote processes on the machines
-which contain the data they need.  For example, a query for data from
-a small region will launch sub-query operations on the machines which
-contain the data overlapping the region of interest.  These remote
-query operations will select the database information which matches
-the query request (i.e., applying restrictions as defined) and return
-to the master process the results.  The results from the various
-partition hosts are then merged into a single result by the master
-process.  This parallelization is critical to querying and
-manipulating the enormous database on a reasonable timescale.
-
-\subsection{Tables which describe objects} 
-
-Two tables carry the most important information about the astronomical
-objects in the database: Average and SecFilt.  These two tables
-specify the main average properties of the astronomical object.  The
-Average table includes the astrometric information ($\alpha, \delta,
-\mu \alpha, \mu \delta, \pi$) and associated errors, data quality
-flags for each object, links to the other tables, and a number of IDs,
-with one row for each astronomical object.  \note{go into complete
-  detail here on the IDs?}.  The SecFilt table\footnote{The name
-  SecFilt is a bit of a historical misnomer: originally, DVO was
-  designed for a monochromatic survey and data for a single
-  photometric band was maintained in the Average table.  Later, DVO
-  was adapted to a multifilter system and additional filters were
-  added to the SecFilt (Secondary Filter) table.  Eventually, the
-  schema was normalized and all photometric data placed in SecFilt,
-  with the Primary filter concept being dropped, but the name has
-  since stuck.} contains average photometric information for a
-collection of filters.  A given DVO instance has a specified set of
-filters for which average photometry is stored in the SecFilt table.
-The number and choice of filters for the SecFilt may be modified by
-the database administrator fairly easily, but the process of updating
-the database is somewhat expensive (\approx 24 hours for the current
-PV3 instance).  Thus the choice is semi-static for a given DVO
-instance.  In the case of the PV3 DVO instance, 9 average bandpasses
-are defined: {\it g, r, i, z, y, J, H, K, w}.  The SecFilt table
-contains one row for each filter for each object, thus the PV3 DVO
-contains 9 times as many rows as the Average table.  The order of the
-table is fixed in relation to the Average table: row $i$ of Average
-defines the object with photometry contained in rows $9i \rightarrow 9i +
-8$ ($i$ is zero counting).  
-
-The individual measurements of the astronomical objects are carried in
-the table \code{Measure}.  This table lists the values measured by
-\code{psphot} for each chip, warp, or stack image.  This includes the
-instrumental magnitudes for the PSF, aperture, and Kron photometry;
-raw position (Xccd, Yccd) and second moments (Mxx, Myy, Mxy), along
-with shape parameters of the PSF model at the position of the object
-(FWx, FWy, theta).  This table also includes metadata such as the
-exposure time, the date \& time of the observation, airmass, azimuth,
-and information specifying the filter \note{describe the photcodes}.
-The \code{Measure} table also carried the calibration magnitude offsts
-($M_{\rm cal}$ and $M_{\rm flat}$ discussed below) and the
-astrometrically calibrated position.  Astrometric offsets for several
-systematic corrections discussed below are also defined for each
-measurement.  Since stacks and forced warp photometry may have
-non-significant values, the table is somewhat de-normalized in that it
-also carried instrumental flux values for the PSF, aperture, and Kron
-photometry.  
-
-In the \code{Measure} table, there are three fields which provide two
-independent links from the specific measurement to the associated
-object: \code{Measure.catID} specifies the spatial partition to which
-the measurement belongs; \code{Measure.objID} specifies to which entry
-in the \code{Average} table the measurement belongs.  These two 32 bit
-fields can thus be combined into a single 64 bit unique ID for all
-objects in the database.  In addition, the field \code{Measure.averef}
-specifies the row number in the \code{Average} table of the associated
-object.  The \code{Measure} table may be unsorted, in which case it is
-slow to find the measurements associated with a specific object (a
-full table scan is required).  After the table is sorted, the
-\code{Measure} rows for a given object are grouped together.  In the
-case, the fields \code{Average.measureOffset} and
-\code{Average.Nmeasure} define an index for the code to jump to the
-list of measurements for a single object.  The field
-\code{Measure.imageID} defines the link from the measurement to the
-image which supplied the measurement.
-
-\note{some discussion of the db construction, addstar, dvomerge, etc?}
-
-For the warp images, we also measure the weak lensing KSB parameters
-related to the shear and smear tensors.  These measurements are stored
-in the \code{Lensing} table, along with the radial aperture fluxes for
-radii numbers 5, 6, \& 7 (XX, XX, XX arcsec).  This table contains one
-row for every warp row.  Similarly to the \code{Measure} table, the fields
-\code{objID}, \code{catID}, and \code{averef} define links from the
-\code{Lensing} table to the \code{Average} table.  In a similar
-fashion, the fields \code{Average.lensingOffset} and
-\code{Average.Nlensing} are the index into the sorted \code{Lensing}
-table entries.  \note{discuss failure of the Lensing to Measure
-  indexing}
-
-The values stored in the \code{Lensing} table are used to calculate
-average values for each of these types of measurements in each
-filter.  The \code{Lensobj} table stores the averaged KSB and radial
-aperture photometry for each of the 5 filters \grizy.  This table
-contains one entry per object per filter.  The table is not generally
-stored unsorted as it is calculated after the full database is
-populated.  The link from \code{Average} to \code{Lensobj} is defined
-by the fields \code{Average.offsetLensobj} and
-\code{Average.Nlensobj}.  Each \code{Lensobj} row also includes the
-photcode (filter) for which the average lensing (and radial aperture)
-properties have been calculated. 
-
-The \code{Galphot} table stores the results of the forced galaxy
-fitting analysis for each object that has been measured.  This table
-contains one row per filter and model type (Sersic, Exponential,
-DeVaucouleur) if forced galaxy models have been calculate for the
-object.  \note{need to expand on this somewhat}
-
-The \code{Starpar} table carries measurements provide by Greg Green \&
-Eddie Schlafly from their analysis of the SED of objects in the PS1
-$3\pi$ data, using the \note{PV1?} version of the analysis (Green et
-al REF).  In this work, the goal was a 3D model of the dust in the
-Galaxy based on Pan-STARRS (\note{and WISE \& 2MASS?}) photometry.  As
-part of this analysis, the authors fit the SEDs of all \note{stellar?}
-sources with stellar models including free parameters of extinction,
-distance modulus, metallicity, and absolute r-band magnitude.  While
-these photometric distance modulus measurements are not extremely
-precise (see below), they provide a constraint on the distance is used
-in our analysis of the astrometry (see Section~\ref{sec:astrometry}).
-
-\subsection{Other Tables} 
-
-Data from GPC1 (and other cameras processed by the IPP) are loaded
-into DVO in units \code{smf} files generated by the Camera calibration
-stage.  As described above, these files contain all measurements from
-a complete exposure, with measurements from each chip grouped into
-separate FITS tables.  When these measurements are loaded into the
-\code{Measure} and similar tables, a subset of the information from
-the chip header is used to populated a row in the DVO \code{Image}
-table.  This table contains one row for each chip known to DVO, with
-information such as the filter (\code{photcode}), the exposure time,
-the airmass, the astrometric calibration terms, the photometric
-zero point, etc.  For GPC1 and other mosaic cameras, an additional row
-is defined to carry the projection and camera distortion elements of
-the astrometry model.  As chips are loaded into this table, they are
-assigned an internal ID (a running sequence in the table).  Images may
-also be assigned an external ID: in the case of the GPC1 images, this
-ID is defined by the processing mysql database and is guaranteed to be
-unique within the processing system. 
-
-Other tables are used to track information used by the calibration
-system.  This includes the complete set of flat-field corrections
-determined by the photometry calibration analysis (see
-Section~\ref{sec:relphot}) and the astrometric flat-field corrections
-determined by the astrometry calibration analysis (see Section~\ref{sec:relastro})
-
 \section{Photometry Calibration}