Index: trunk/doc/release.2015/ps1.datasystem/datasystem.tex
===================================================================
--- trunk/doc/release.2015/ps1.datasystem/datasystem.tex	(revision 39964)
+++ trunk/doc/release.2015/ps1.datasystem/datasystem.tex	(revision 39973)
@@ -1372,15 +1372,15 @@
 Traditionally, projects which use multiple exposures to increase the
 depth and sensitivity of the observations have generated something
-equivalent to the stack images produced by the IPP analysis (c.f, CFHT
-Legacy survey, COSMOS, etc).  In theory, the photometry of the stack
-images produces the `best' photometry catalog, with best sensitivity
-and the best data quality at all magnitudes.  In practice, the stack
-images have some significant limitations due to the difficulty of
-modelling the PSF variations.  This difficulty is particularly severe
-for the Pan-STARRS $3\pi$ survey stacks due to the combination of the
-substantial mask fraction of the individual exposures, the large
-instrinsic image quality variations within a single exposure, and the
-wide range of image quality conditions under which data were obtained
-and used to generate the $3\pi$ PV3 stacks.
+equivalent to the \ippstage{stack} images produced by the IPP analysis
+(c.f, CFHT Legacy survey, COSMOS, etc).  In theory, the photometry of
+the \ippstage{stack} images produces the ``best'' photometry catalog,
+with best sensitivity and the best data quality at all magnitudes.  In
+practice, these images have some significant limitations due to the
+difficulty of modelling the PSF variations.  This difficulty is
+particularly severe for the Pan-STARRS $3\pi$ survey stacks due to the
+combination of the substantial mask fraction of the individual input
+exposures, the large instrinsic image quality variations within a
+single exposure, and the wide range of image quality conditions under
+which data were obtained and used to generate the $3\pi$ PV3 stacks.
 
 For any specific stack, the point spread function at a particular
@@ -1389,24 +1389,20 @@
 that point.  Because of the high mask fraction, the exposures which
 contributed to pixels at one location may be somewhat different just a
-few tens of pixels away.  Because of the intrinsic variations in the
-PSF across an exposure and because of the variations from exposure to
-exposure, the distribution of point spread functions of the images
-used at one position may be quite different from those at a nearby
-location.  In the end, the stack images have a effective point spread
-function which is not just variable, but changing significantly on
-small scales in a highly textured fashion.  \note{duplicates previous paragraph?}
+few tens of pixels away.  In the end, the \ippstage{stack} images have
+a effective point spread function which is not just variable, but
+changing significantly on small scales in a highly textured fashion.
 
 Any measurement which relies on a good knowledge of the PSF at the
 location of an object either needs to determine the PSF variations
-present in the stack, or the measurement will be somewhat degraded.
-The highly textured PSF variations make this a very challenging
-problem: not only would such a PSF model require an unusually fine-grained
-PSF model, there would likely not be enough PSF stars in an given
-stack to determine the model at the resolution required.  The IPP
-photometry analysis code uses a PSF model with 2D variations using a
-grid of at most $6\times 6$ samples per skycell, a number reasonably
-well-matched to the density of stars at most moderate Galactic
-latitudes.  This scale is far too large to track the fine-grained
-changes apparent in the stack images.
+present in the \ippstage{stack} image, or the measurement will be
+somewhat degraded.  The highly textured PSF variations make this a
+very challenging problem: not only would such a PSF model require an
+unusually fine-grained PSF model, there would likely not be enough PSF
+stars in a given \ippstage{stack} image to determine the model at the
+resolution required.  The IPP photometry analysis code uses a PSF
+model with 2D variations using a grid of at most $6\times 6$ samples
+per skycell, a number reasonably well-matched to the density of stars
+at most moderate Galactic latitudes.  This scale is far too large to
+track the fine-grained changes apparent in the stack images.
 
 Thus PSF photometry as well as convolved galaxy models in the stack
@@ -1421,76 +1417,105 @@
 The PV3 $3\pi$ analysis solves this problem by using the sources
 detected in the stack images and performing forced photometry on the
-individual warp images used to generate the stack.  This analysis is
-performed on all warps for a single filter as a single job, though
-this is more of a bookkeeping aid as it is not necessary for the
-analysis of the different warps to know about the results of the other
-warps.
-
-In the forced warp photometry, the positions of sources are loaded
-from the stack outputs.  PSF stars are pre-identified and a PSF model
-generated for each warp based on those stars, using the same stars for
-all warps to the extent possible (PSF stars which are excessively
-masked on a particular image are not used to model the PSF).  The PSF
-model is fitted to all of the known source positions in the warp
-images.  Aperture magnitudes, Kron magnitudes, and moments are also
-measured at this stage for each warp.  Note that the flux measurement
-for a faint, but significant, source from the stack image may be at a
-low significance ($< 5\sigma$) in any individual warp image; the flux
-may even be negative for specific warps.  When combined together,
-these low-significance measurements will result in a signficant
-measurement as the signal-to-noise increases by $\sqrt{N}$.  
+individual warp images used to generate the stack.  This
+\ippstage{fullforce} analysis is performed on all warps for a single
+skycell and filter as a single unit, as this matches the arrangement
+of the input source catalog from the \ippstage{skycal} stage.  When
+processing is queued for this stage, an entry is added to the
+\ippdbtable{fullForceRun} primary database table linking to the
+specific \ippdbcolumn{skycal\_id} entry that will be used as the
+catalog for the photometry.  The \ippdbcolumn{warp\_id} values for the
+input \ippstage{warp} stage images that contributed to the
+\ippstage{stack} associated with that \ippdbcolumn{skycal\_id} are
+then added to the \ippdbtable{fullForceInput} table, linked to the
+primary table by the \ippdbcolumn{ff\_id} identifier.  The individual
+jobs for each warp are then run, which passes the \ippstage{warp}
+stage image products along with the \ippstage{skycal} catalog to the
+\ippprog{psphotFullForce} program.
+
+In this program, the positions of sources are loaded from the input
+catalog.  PSF stars are pre-identified \note{how?} and a PSF model
+generated for each \ippstage{warp} image based on those stars, using
+the same stars for all warps to the extent possible (PSF stars which
+are excessively masked on a particular image are not used to model the
+PSF).  \note{this doesn't seem correct, as each warp is run
+  independently.}  The PSF model is fitted to all of the known source
+positions in the warp images.  Aperture magnitudes, Kron magnitudes,
+and moments are also measured at this stage for each warp.  Note that
+the flux measurement for a faint, but significant, source from the
+stack image may be at a low significance (less than the $5\sigma$
+criterion used when the photometry is not run in this forced mode) in
+any individual warp image; the flux may even be negative for specific
+warps.  When combined together, these low-significance measurements
+will result in a signficant measurement as the signal-to-noise
+increases by $\sqrt{N}$.
+
+Upon completion of the forced photometry (for point sources as well as
+galaxies, discussed below), an entry is added to the
+\ippdbtable{fullForceResult} table with the processing statistics for
+that combination of \ippdbcolumn{ff\_id} and \ippdbcolumn{warp\_id}.
+Once all of the entries in the \ippdbtable{fullForceInput} table have
+finished, a summary operation is run to generate an appropriate
+average value for each measurement, by combining the measurements from
+each of the inputs.  The output catalogs listed in the
+\ippdbtable{fullForceResult} table are passed to the
+\ippprog{psphotFullForceSummary} to do this averaging.  \note{describe
+  what is done} When this completes, an entry is added to the
+\ippdbtable{fullForceSummary}, and the \ippdbtable{fullForceRun} entry
+is marked as completed.
 
 \subsubsection{Forced Galaxy Models}
-
-The convolved galaxy models are also re-measured on the warp images by
-the forced photometry analysis stage.  In this analysis, the galaxy
-models determined by the stack photometry analysis are used to seed
-the analysis in the individual warps.  The purpose of this analysis is
-the same as the forced PSF photometry: the PSF of the stack is poorly
-determined due to the masking and PSF variations in the inputs.
-Without a good PSF model, the PSF-convolved galaxy models are of
-limited accuracy.  
-
-In the forced galaxy model analysis, we assume that the galaxy
-position and position angle, along with the Sersic index if
-appropriate, have been sufficiently well determined in the stack
-analysis.  In this case, the goal is to determine the best values for
-the major and minor axis of the elliptical contour and at the same
-time the best normalization corresponding to the best elliptical shape
-(and thus the best galaxy magnitude value).
-
-For each warp image, the stack value for the major and minor axis are
-used as the center of a $7\times 7$ grid search of the major and minor
-axis parameter values.  The grid spacing is defined as a function of
-the signal-to-noise of the galaxy in the stack image so that bright
-galaxies are measured with a much finer grid spacing that faint
-galaxies \note{need to quantify this}.  For each grid point, the major
-and minor axis values at that point are determined for the model.  The
-model is then generated and convolved with the PSF model for the warp
-image at that point.  The resulting model is then compared to the warp
-pixel data values and the best fit normalization value is defined.
-The normalization and the $\chi^2$ value for each grid point is
-recorded.  
+\note{CZW: is this the appropriate place for this section?}
+
+The convolved galaxy models are also re-measured on the
+\ippstage{warp} images by the \ippstage{fullforce} stage analysis.  In
+this analysis, the galaxy models determined by the
+\ippstage{staticsky} photometry analysis are used to seed the analysis
+in the individual \ippstage{warp} images.  The purpose of this
+analysis is the same as the \ippstage{fullforce} PSF photometry: the
+PSF of the \ippstage{stack} image is poorly determined due to the
+masking and PSF variations in the inputs.  Without a good PSF model,
+the PSF-convolved galaxy models are of limited accuracy.
+
+In the \ippstage{fullforce} galaxy model analysis, we assume that the
+galaxy position and position angle, along with the Sersic index if
+appropriate, have been sufficiently well determined in the
+\ippstage{staticsky} analysis.  In this case, the goal is to determine
+the best values for the major and minor axis of the elliptical contour
+and at the same time the best normalization corresponding to the best
+elliptical shape, and thus the best galaxy magnitude value.
+
+For each \ippstage{warp} image, the \ippstage{staticsky} value for the
+major and minor axis are used as the center of a $7\times{} 7$ grid
+search of the major and minor axis parameter values.  The grid spacing
+is defined as a function of the signal-to-noise of the galaxy in the
+stack image so that bright galaxies are measured with a much finer
+grid spacing that faint galaxies \note{need to quantify this}.  For
+each grid point, the major and minor axis values at that point are
+determined for the model.  The model is then generated and convolved
+with the PSF model for the \ippstage{warp} image at that point.  The
+resulting model is then compared to the \ippstage{warp} pixel data
+values and the best fit normalization value is defined.  The
+normalization and the $\chi^2$ value for each grid point is recorded.
 
 For a given galaxy, the result is a collection of $\chi^2$ values for
-each of the grid points spanning all warp images.  A single $\chi^2$
-grid can then be made from all warps by combining each grid point
-across the warps.  The combined $\chi^2$ for a single grid point is
-simply the sum of all $\chi^2$ values at that point.  If, for a single
-warp image, the galaxy model is excessively masked, then that image
-will be dropped for all grid points for that galaxy.  The reduced
-$\chi^2$ values can be determined by tracking the total number of warp
-pixels used across all warps to generate the combined $\chi^2$ values.
-From the combined grid of $\chi^2$ values, the point in the grid with
-the minimum $\chi^2$ is found.  Quadratic interpolation is used to
+each of the grid points spanning all \ippstage{warp} images.  A single
+$\chi^2$ grid can then be made by combining each grid point across the
+inputs.  The combined $\chi^2$ for a single grid point is simply the
+sum of all $\chi^2$ values at that point.  If, for a single \ippstage{warp}
+image, the galaxy model is excessively masked, then that image will be
+dropped for all grid points for that galaxy.  The reduced $\chi^2$
+values can be determined by tracking the total number of pixels
+used across all inputs to generate the combined $\chi^2$ values.  From
+the combined grid of $\chi^2$ values, the point in the grid with the
+minimum $\chi^2$ is found.  Quadratic interpolation is used to
 determine the major, minor axis values for the interpolated minimum
 $\chi^2$ value.  The errors on these two parameters is then found by
 determining the contour at which the \note{reduced?} $\chi^2$
-increases by 1.  
-
-Thus the Forced Galaxy Model analysis uses the PSF information from
-each warp to determine a best set of convovled galaxy models for each
-object in the stack images.  \note{discuss the subset of galaxy models
-  and objects}.
+increases by 1.
+
+Thus the \ippstage{fullforce} galaxy analysis uses the PSF information
+from each \ippstage{warp} to determine a best set of convovled galaxy
+models for each object in the \ippstage{skycal} catalog.
+\note{discuss the subset of galaxy models and objects}.
 
 \subsection{Difference Images}
@@ -1510,23 +1535,86 @@
 
 In the \ippstage{diff} stage, the IPP generates diffferece images for
-appropriately specified pairs of images.  It is possible for the difference image to
-be generated from a pair of warp images, from a warp and a stack of
-some variety, or from a pair of stacks.  During the PS1 survey, pairs
-of exposures, call TTI pairs (see~\note{Survey Strategy}), were
-obtained for each pointing within a $\approx$ 1 hour period in the
-same filter, and to the extent possible with the same orientation and
-boresite position.  The standard PS1 nightly processing generated
-difference images from the resulting warp pairs (`warp-warp diffs').
-
-The nightly stacks generated for the Medium Deep fields were combined
-with a template reference stack image to generate `stack-stack diffs'
-for these fields each night.  
-
-For the PV3 $3\Pi$ processing, the entire collection of warps for the
-survey were combined with the $3\pi$ stacks to generate `warp-stack
-diffs'.  
+appropriately specified pairs of images.  It is possible for the
+difference image to be generated from a pair of \ippstage{warp} stage
+images, from a \ippstage{warp} and a \ippstage{stack} of some variety,
+or from a pair of \ippstage{stack} stage images.  During the PS1
+survey, pairs of exposures, call TTI pairs (see~\note{Survey
+  Strategy}), were obtained for each pointing within a $\approx$ 1
+hour period in the same filter, and to the extent possible with the
+same orientation and boresite position.  The standard PS1 nightly
+processing generated difference images from the resulting pairs of
+\ippstage{warp} images.  The nightly processing generated
+\ippstage{stack} images for the Medium Deep fields, and these were
+combined with a template reference \ippstage{stack} image to generate
+``stack-stack diffs'' each night they were observed.  For the PV3
+$3\pi$ processing, the entire collection of \ippstage{warp} stage
+images for the survey were combined with images generated by the
+\ippstage{stack} processing to generate ``warp-stack diffs''.
+
+When a \ippstage{diff} processing is defined, an entry is added to the
+\ippdbtable{diffRun} table, and the appropriate input images are added
+to the \ippdbtable{diffInputSkyfile} table, with one entry for each
+skycell that are covered by the images.  For a \ippstage{diff}
+generated from two \ippstage{warp} stage products, the input images
+have their \ippdbcolumn{warp\_id} values recorded in the
+\ippdbcolumn{warp1} and \ippdbcolumn{warp2} for each skycell that
+overlaps.  If two \ippstage{stack} stages are to be used in the
+difference, their \ippdbcolumn{stack\_id} entries are recorded in the
+\ippdbcolumn{stack1} and \ippdbcolumn{stack2} fields.  As each
+\ippstage{stack} only covers a single skycell, the \ippstage{diff} is
+usually defined indirectly, using other information from the
+\ippdbtable{stackRun} table to select appropriate
+\ippdbcolumn{stack\_id} values.  Similarly, \ippstage{diff} processing
+is defined for the mixed case by creating entries that populate one of
+\ippdbcolumn{warp1} and \ippdbcolumn{stack1} and populating one of
+\ippdbcolumn{warp2} and \ippdbcolumn{stack2}.  In all cases, the
+minuend of the subtraction to be performed is the ``1'' entry, and the
+subtrahend is the ``2'' entry.
+
+Jobs are created based on the entries of
+\ippdbtable{diffInputSkyfile}, with the appropriate images and
+catalogs passed to the \ippprog{ppSub} program.  This does the
+subtraction, as well as the photometry of any sources detected in the
+\ippstage{diff} image.  The algorithm used for PSF matching is
+described in \citet{waters2017}.  Upon completion of these jobs,
+statistics about the processing are written to an entry in the
+\ippdbtable{diffSkyfile} table.  An \ippmisc{advance} checks for the
+completion of all of the components listed in
+\ippdbtable{diffInputSkyfile}, and marks the \ippdbtable{diffRun}
+entry as such.
 
 \subsection{Addstar : DVO Ingest}
 \label{subsec: addstar}
+\note{CZW: This should be reviewed.}
+
+Upon completion of the processing of each stage, the results of the
+photometry analysis are isolated in a large number of individual
+catalogs, with little connection between the separate measurements of
+astronomical sources.  Unifying these measurements in a DVO database
+is the purpose of the \ippstage{addstar} processing.  The catalogs for
+the \ippstage{camera}, \ippstage{staticsky}, \ippstage{skycal},
+\ippstage{fullforce}, and \ippstage{diff} are processed in this
+fashion, although not every measurement in each catalog are included
+in the final DVO that is constructed.
+
+The construction of this final DVO is performed in a hierarchical
+process.  The individual catalogs are added to a \ippmisc{minidvo},
+which is simply a DVO database defined over some subset of possible
+inputs.  These \ippmisc{minidvos} are then merged into larger
+databases to construct the final completely catalog.  \note{describe
+  database tables}
+
+Each catalog that is to be added to DVO has an entry created in the
+\ippdbtable{addRun} database table.  This entry notes which
+\ippdbcolumn{stage} is the source of the catalog, and links to the
+appropriate database table with the \ippdbcolumn{stage\_id} field.  As
+some stages, such as the \ippstage{diff} stage, create more than a
+single catalog, multiple entries with the \ippdbcolumn{stage\_id} are
+created, with the \ippdbcolumn{stage\_extra1} field containing an
+index to the individual components.  The catalog specified by the
+entry is added to the target \ippmisc{minidvo} by the
+\ippprog{addstar} program, \note{describe what's done?}.  When this
+completes, an entry containing the statistics of the job is added to
+the \ippdbtable{addProcessedExp} table.
 
 \subsection{Calibration Operations}