Index: trunk/doc/release.2015/ps1.datasystem/datasystem.tex
===================================================================
--- trunk/doc/release.2015/ps1.datasystem/datasystem.tex	(revision 39999)
+++ trunk/doc/release.2015/ps1.datasystem/datasystem.tex	(revision 40001)
@@ -176,12 +176,117 @@
     submission and refereeing process.}}
 
-\section{IPP Software Subsystems}
-\label{sec: subsystems}
-
-The IPP relies on a number of common libraries and programs to handle
-various tasks that are shared between multiple stages of the
-processing.  These subsystems are described in this section, to
-provide an introduction to these essential components that underlie
-the rest of the pipeline.
+\section{Overview of Pan-STARRS Data Processing}
+
+The Pan-STARRS Data Analysis system contains many features to support
+the wide range of activities: archiving and management of the raw and
+processed image files; real-time nightly processing of images for
+transient and moving object science; large-scale re-processing and
+calibration to produce measurements for the science collaboration and
+the wider public; \note{manual/specialized} image processing to
+facilitate research and development of the analysis system itself;
+distribution of the resulting data products to various consumers in a
+variety of formats and modes. 
+
+The Pan-STARRS Data Analysis system is divided internally into several major
+components:
+\begin{itemize}
+\item Summit : both the camera and observatory summit systems perform
+  data analysis tasks needed to support the on-going observations.
+  In this article, we focus on those aspects used by the off-summit
+  analysis stages.
+\item Image Processing Pipeline (IPP) : this portion of the data
+  analysis system takes the data from raw pixels on the summit
+  computers to calibrated measurements of astronomical objects in an
+  internal databasing system.
+\item Moving Object Processing System (MOPS) : this system is
+  responsible for linking individual detections of solar-system
+  objects together and determining the orbits.
+\item PSPS : this system ingests the calibrated measurements from the
+  IPP, MOPS, and others and generates a high-availability database
+  with web-based interactions for public consumption.
+\end{itemize}
+The above set of analysis stages take place at the IfA within the
+scope of responsibility of the Pan-STARRS Observatory.  Within the
+wider Pan-STARRS colloboration(s), additional data analysis operations
+are performed to support science results.  These collaboration-wide
+analysis operations range from those which are tightly-coupled to the
+Pan-STARRS Observatory system, such as the analysis of the transient
+discovery teams and the public archive database at MAST, to those
+which perform offline analysis for eventual ingest back into the
+Pan-STARRS databases and archive.  The latter category includes the
+ubercal photometric analysis, the photo-z analysis, and the QSO / RR
+Lyra search efforts.  In addition, collaborations within the wider
+Pan-STARRS community have implemented a variety of science-level
+analyses of their own to support their science goals (e.g., M31
+Cepheid search).
+
+Figure~\ref{fig:analysis.elements} illustrates the many elements of
+the Pan-STARRS data analysis system.  This figure focuses on the data
+analysis steps which occur within the Pan-STARRS observatory, with an
+emphasis on the analysis, calibration, and database ingest stages.
+The MOPS is described in detail by \cite{MOPS}, while the summit
+systems are described by \note{REF?}.
+
+Data analysis to support nighly science operations is driven by two
+main goals: 1) rapid detection of the moving and transient sources to
+enable recovery or follow-up with other telescopes. 2) regular
+analysis of the images to monitor data quality and for use in
+longer-timescale science projects.  Not all of the analysis elements
+listed in Figure~\ref{fig:analysis.elements} are used by the nightly
+analysis system.  Each of the data analysis stages are discussed in
+detail below.  In short, each image is processed independently to
+correct for instrumental signatures and to detect the astronomical
+sources (chip); astrometric and photometric calibrations are
+determined (camera), and finally images are geometric transformed to a
+common pixel representation (warp).  Warped images may either be added
+together (stack) or used in an image subtraction (diff).  For nightly
+science operations, images for certain fields such as the Medium Deep
+survey fields (see \cite{}), are stacked together in nightly chunks,
+providing deeper detection capability on short timescales.  Depending
+on the survey mode, difference images are generated for the nightly
+stack images (vs a deep stack template) or for individual warp images.
+In the later case, the warp images may be difference against another
+warp from the same night or against a reference stack from the
+appropriate part of the sky.
+
+\note{need earlier mention of 3pi, MD, etc}
+
+Pan-STARRS has performed several large-scale reprocessings of both the
+Medium Deep and 3pi Survey data.  For the 3pi Survey data, we identify
+these large-scale reprocessings as PV1, PV2, and PV3 (we also define
+the nightly science analysis of the data as PV0).  For these
+reprocessing stages, the standard steps of chip through warp, plus
+stack and diff are performed, starting from raw data, using a single
+homogenous version of the data analysis procedures.  (PV2 was a
+special case in which we started from the camera level products of
+PV1).  In addition to the analysis stages which are common with the
+nightly processing, these large-scale reprocessing stages include
+additional processing: a more detailed photometric analysis is
+performed on the stacks, including morphological analysis appropriate
+to galaxies.  The results of the stack photometry analysis are used to
+drive a forced-photometry analysis of the warp images.  The data
+products from the camera, stack photometry, and forced-warp photometry
+analysis stages are ingested into the internal calibration database
+(DVO, the Desktop Virtual Observatory) and used for photometric and
+astrometric calibrations.
+
+During the PS1 Science Consortium operations, data products were
+provided to the consortium members from many different stages of the
+analysis process.  Data access by the PS1 Science Consortium members
+was managed through a variety of mechanisms depending on the data
+volume and type of data products desired.
+Figure~\ref{fig:analysis.elements} illustrates some of these
+connections.  Access to small samples of imaging data was provided on
+demand via the Postage Stamp server; access to large sets of
+pre-defined raw and reduced data products was provided via the
+Distribution and Publication systems.  The interal calibration DVO
+databases were provided at several stages via a separate DVO
+distribution mechanism.  For the first two large-scale reprocessings
+(PV1 \& PV2), the data were ingested into the PSPS database system and
+made available to the PS1SC community through a web portal based at
+the IfA as well as the MAST portal.
+
+\section{IPP Data Processing Stages}
+\label{sec: stages}
 
 \subsection{Processing Database}
@@ -260,4 +365,664 @@
 processing is able to keep the data flowing even in the face of
 occasional network glitches or hardware crashes.
+
+\subsection{Summit copy}
+\label{subsec: summit copy}
+
+As exposures are taken by the PS1 telescope \& GPC1 camera system, the
+data from the 60 OTA devices are read out by the camera software
+wsystem and written to disk on a collection of computers at the summit
+in the PS1 facility called ``pixel servers.'' After the images are
+written to disk, a summary listing of the information about the
+exposure and the chip images are added to the summit datastore.
+
+During night-time operations, while the summit datastore is being
+populated, the IPP subsystem called \ippstage{summitcopy} monitors the
+datastores listed in the \ippdbtable{pzDatastore} table of the
+database in order to discover new exposures ready for download.  Once
+a new exposure has been listed on the datastore, \ippstage{summitcopy}
+adds an entry of the exposure to a table in the processing database
+(\ippdbtable{summitExp}), indexed by an identifier that simply
+increments the number of exposures announced by the summit, the
+\ippdbcolumn{summit\_id}.  This tells the \ippstage{summitcopy} system
+to look for the list of chips, which are then added to another table
+(\ippdbtable{summitImfile}).  This system then attempts to download
+the chips (registering the results of those operations into the
+\ippdbtable{pzDownloadExp} and \ippdbtable{pzDownloadImfile} tables)
+from the summit pixel servers via an http request.  As the image files
+are downloaded, their MD5 checksum values are calculated and compared
+with the value reported by the summit datastore.  Download failures
+are rare and marked with a non-zero \ippdbcolumn{fault}, allowing for
+a manual recovery, rather than automatically rejecting the failed
+chips.  Once all the components of the exposure have been downloaded,
+they are further entered into the \ippdbtable{newExp} and
+\ippdbtable{newImfile} tables, which index the exposures by
+\ippdbcolumn{exp\_id}.  This switch in index indicates that the
+exposure has successfully been copied from the summit to the IPP
+cluster, and that further processing is no longer dependent on outside
+resources.
+
+\subsection{Image Registration}
+\label{subsec: registration}
+
+Once the chips for an exposure have all been downloaded, the exposure
+is ready to be registered.  In this context, `registration' refers to
+the process of adding them to the database listing of known, raw
+exposures (not to be confused with 'registration' in the sense of
+pixel re-alignment).  The result of the registration analysis is an
+entry for each exposure in the \ippdbtable{rawExp} table, and one for
+each chip in the \ippdbtable{rawImfile} table.  These tables are
+critical for downstream processing to identify what exposures are
+available for processing in any other stage.  At the registration
+stage, a large amount of descriptive metadata for each chip is added
+to the \ippdbtable{rawImfile} table, the majority of which is
+extracted from the chip FITS file headers (e.g., RA, DEC, FILTER) and
+some of which is determined by a quick analysis of the pixels (e.g.,
+mean pixel values, standard deviation).  The chip-level information is
+merged into a set of exposure-level metadata and added to the
+\ippdbtable{rawExp} table entry.  The exposure-level metadata may be
+the same as any one of the chip, in a case where the values are
+duplicated across the chip files (e.g., the name of the telescope or
+the date \& time of the exposure), or it may be a calculation based on
+the values from each chip (e.g., average of the average pixel values).
+
+Unlike much of the rest of the IPP stage, the raw exposures may only
+have a single entry in the registration tables of the processing
+database tables (\ippdbtable{rawExp} and \ippdbtable{rawImfile}).
+
+For GPC1, the image registration stage is also the stage at which the
+\ippprog{burntool} analysis is run.  This analysis is more completely
+described in \citet{waters2017}.  In brief, the \ippprog{burntool}
+program identifies bright sources on the image, and identifies
+persistence trails that result from the incomplete transfer of charge.
+As this charge can leak out in subsequent exposures, the burntool
+analysis is run sequentially on the exposures, based on the
+observation date and time listed in the headers, with the results
+stored in an text table.  As a result of the sequential nature of this
+analysis, the registration of exposures is blocked until the
+\ippprog{burntool} has been run on the previous exposures.
+
+Once the registration process has finished, new science exposures that
+have an \ippdbcolumn{obs\_mode} value that indicates they are part of
+a particular science survey are automatically launched into the
+science analysis by defining entries for the \ippstage{chip}
+processing stage, as described above.  This analysis can be relaunched
+multiple times, such as for the large scale PV3 reprocessing.
+However, this automatic process ensures the shortest time between
+observation and analysis, which is particularly important in the
+search for transient sources.
+
+\subsection{Chip Processing}
+\label{subsec: chip}
+
+The science analysis of an exposure begins with the \ippstage{chip}
+stage, which operates on the individual OTA image files.  This
+analysis step has two main goals: detrending the image to remove the
+instrumental signature from the pixel values, and the detection of
+astronomical sources in the objects.  Based on the entry the
+\ippdbtable{chipRun} primary table defining the processing details
+(with the \ippdbcolumn{state} column indicating it needs processing),
+and the associated information listed in the \ippdbtable{rawImfile},
+jobs can be spawned for each component OTA.  The \ippprog{pantasks}
+environment managing the jobs attempts to target the processing host
+to one that should host the OTA, to reduce number of operations done
+on remote data.  In practice, this targeted processing has not had as
+large of an effect as was originally intended, as the data volume has
+reduced the ability of any one node to reliably contain a particular
+OTA.  The targeted processing has probably reduced the network load
+somewhat but it has not been as critical of a requirement as
+originally expected.
+
+%% In the \ippstage{chip} stage,
+%% the individual OTA image files are processed independently in parallel
+%% within the data processing cluster.  \note{move this to kihei
+%%   discussion?} Within the processing computer cluster, most of the
+%% data storage resources are in the form of computers with large raids
+%% as well as substantial processing capability.  The processing system
+%% attempts to locate one copy of specific raw registered data on
+%% pre-defined computers that have been set as storage targets for that
+%% OTA.  The processing system is aware of this data localization and
+%% attempts to target the processing for each OTA to the machine on which
+%% the data for that detector is stored.  The output products are then
+%% primarily saved back to the same machine.  This `targetted' processing
+%% was an early design choice to minimize the system wide network load
+%% during processing.  In practice, as computer disks filled up at
+%% different rates, the data has not been localized to a very high
+%% degree.  
+
+The actual image processing is performed by the \ippprog{ppImage}
+program.  This program reads the raw data into memory and applies the
+detrend corrections \citep[see][]{waters2017} to each cell in the OTA
+(which are stored as different extensions in the FITS file format),
+and then mosaics the cells into a single contiguous \ippstage{chip}
+stage image.  This step also creates in memory additional images to
+hold the mask data, which indicates which pixels may not be valid, and
+the variance image, constructed as the Poissonian noise on the number
+of electrons detected based on the original pixel value and the
+detector gain.  A background model is then fit across the image and
+subtracted to remove the expected contribution from the sky
+\citep[see][]{waters2017} for details.
+
+With the image calibration procedure finished, object identification
+and photometry can be performed.  Although this can be done using a
+stand alone program, \ippprog{psphot}, the underlying functions are
+contained in a library that allows \ippprog{ppImage} to directly do
+this analysis, removing the need to write out and re-read the image
+data.  The details of the detection and characterization of the
+sources in the image are provided in \citet{magnier2017b}.  
+
+The results of the image processing are then written to disk,
+including the science, mask, and variance images, the background model
+subtracted, the PSF model used in the photometry process, and a FITS
+catalog of detected sources.  Additional binned images of the full OTA
+are also saved, providing $16\times{}16$ and $256\times{}256$ pixel
+binning scales for quick visualization.  The processing log and a
+selection of summary metadata describing the processing results are
+also written to disk.  This metadata is used to populate a row in the
+\ippdbtable{chipProcessedImfile} table (linked to the
+\ippdbtable{chipRun} entry by a shared \ippdbcolumn{chip\_id} value)
+to indicate that the processing of this OTA is complete.
+
+As each OTA is processed independently of the others across a number
+of computers, the \ippprog{pantasks} managing the jobs periodically
+runs an \ippmisc{advance} task that checks that the number of rows in
+\ippdbtable{chipProcessedImfile} with \ippdbcolumn{fault} equal to
+zero matches the associated number of rows in \ippdbtable{rawImfile}.
+If this condition is met, than all processing for that exposure is
+finished, and the \ippdbcolumn{state} field is set to ``full''.  If
+the \ippdbtable{chipRun}.\ippdbcolumn{end\_stage} field is set to
+\ippstage{chip}, then no further action is taken.  However, this field
+is usually set to a subsequent stage (most often \ippstage{warp}),
+then an entry for this exposure is added to the \ippdbtable{camRun}
+table, and processing continues.
+
+%% The \ippstage{chip} processing stage consists of: reading the raw image into
+%% memory, applying the detrending steps \citep[see][]{waters2017},
+%% stiching the individual OTA cells into a single chip image, detection
+%% and characterization of the sources in the image
+%% \citep[see][]{magnier2017b}, and output of the various data products.
+%% These include the detrended chip image, variance image, and mask
+%% image, as well as the FITS catalog of detected sources.  The PSF model
+%% and background model are also saved, along with a processing log.  A
+%% selection of summary metadata describing the processing results are
+%% saved and written to the processing database along with the completion
+%% status of the process.  Finally, binned chip images are generated (on
+%% two scales, binned by 16 and 256 pixels) for use in the visualization
+%% system of the processing monitor tool. \note{describe elsewhere?}
+
+%% The database structure for the \stage{chip} stage mimics that of raw
+%% data, with a \ippdbtable{chipRun} characterizing the processing of a
+%% single exposure, mapping to a set of \ippdbtable{chipProcessedImfile}
+%% entries for each OTA via a common \ippdbcolumn{chip\_id}.  
+
+\subsection{Camera Calibration}
+\label{subsec: camera}
+
+After sources have been detected and measured for each of the chips,
+the next stage is to perform a basic calibration of the full exposure
+in the \ippstage{camera} stage.  This runs as a single job for the
+entire exposure, passing the collection of FITS table catalogs
+generated from each OTA in the \ippstage{chip} stage to the
+\ippprog{psastro} program.  Although the full catalog is loaded, the
+calibration primarily concerns the positions ($x_{\rm ccd}, y_{\rm
+  ccd}$) and the instrumental PSF magnitudes.  The header information
+in these catalogs is used to determine the coordinates of the
+telescope boresite (RA, DEC, position angle).  These three coordinates
+are used, along with a pre-determined model of the OTA layout within
+the camera, to generate an initial guess for the astrometry of each
+chip.  Reference star coordinates and magnitudes are loaded from a
+reference catalog for a region corresponding to the boundaries of the
+exposure, padded by a large fraction (25\%) of the exposure diameter
+to help guarantee a solution in the case of a modest pointing error.
+The guess astrometry is used to match the reference catalog to the
+observed stellar positions in the focal plane coordinate system.  Once
+an acceptable match is found, the astrometric calibration of the
+individual chips is performed, including a fit to a single model for
+the distortion introduced by the camera optics.  After the astrometic
+analysis is completed, the photometric calibration is determined using
+the final match to the reference catalog.  At this stage,
+pre-determined color terms may be included to convert the reference
+photometry to an appropriate photometric system.  For PS1, this is
+used to generate synthetic w-band photometry for areas where no
+PS1-based calibrated w-band photometry is available.  For more
+details, see \cite{magnier2017c}.  The result of these calibrations is
+stored as a single multi-extension FITS table containing the results
+from each OTA as a separate extension.
+
+In addition to the astrometric and photometric calibrations, the
+\ippstage{camera} stage also generates the dynamic masks for the
+images.  These include masking for optical ghosts, glints, saturated
+stars, diffraction spikes, and electronic crosstalk.  The mask images
+generated by the \ippstage{chip} stage are updated with these dynamic
+masks and a new set of files are saved for the downstream analysis
+stages.  The \ippstage{camera} stage also merges the binned chip
+images (see~\ref{sec:chip}) into single jpeg images of the entire
+focal plane.  These jpeg images can then be displayed by the process
+monitoring system to visualize the data processing.
+
+Again, summary metadata is saved to disk as well, and the results
+listed therein are used to populate a row in the
+\ippdbtable{camProcessedExp} database table.  As the full exposure is
+processed all at once, this update also updates the associated
+\ippdbtable{camRun} entry, linked by the \ippdbcolumn{cam\_id}.  As
+with the \ippstage{chip} stage, the
+\ippdbtable{camRun}.\ippdbcolumn{end\_stage} is for a subsequent
+stage, an appropriate entry is added to the \ippdbtable{fakeRun}
+table.
+
+\subsection{Fake Analysis}
+\label{subsec: fake}
+
+The \ippstage{fake} stage was originally designed to do false source
+injection and recovery, in order to determine the detection efficiency
+of sources on the exposure.  However, early in the design of the IPP,
+this task was moved to the rest of the photometry analysis done at the
+\ippstage{chip} stage.  Removing the stage would require significant
+changes to the database schema.  As a result, this conveniently named
+stage generally does no actual data processing, and consists mainly of
+database operations to move the exposure on to the \ippstage{warp}
+stage.  The operations mimic the \ippstage{chip} stage, with
+individual jobs run for each OTA that update rows in the
+\ippdbtable{fakeProcessedImfile}, and an \ippmisc{advance} task that
+updates the \ippdbtable{fakeRun} table and promotes the exposure to
+the next stage by adding a row to the \ippdbtable{warpRun} table.
+
+\subsection{Image Warping}
+\label{subsec: warp}
+
+The \ippstage{warp} stage moves the data from a given exposure beyond
+away from being camera specific and towards a uniform sky oriented
+arrangement.  There are a number of ``tessellations'' defined and used
+by the IPP to define the extent and scaling of images on this uniform
+arrangement.  A tessellation can be defined for a limited region, such
+as M31 or other fields of particular interest that can be well
+described by a single tangent plane projection, or for larger regions
+which have multiple projection centers.  For the $3\Pi$ survey, the
+\ippmisc{RINGS.V3} tessellation was used that used projection centers
+spaced every four degrees in both RA and DEC, with $0\farcs{}25$
+pixels.  These projections are further broken down into ``skycells''
+that form a $10\times{}10$ grid within the projection, with an overlap
+region of 60" between adjacent skycells to ensure that objects are not
+split on all images. 
+
+These tessellations are stored in the DVO format, with
+\ippdbtable{SkyTable} entries defining the projection centers and
+image boundaries for all the skycells.  The first step of the
+\ippstage{warp} stage is determining which skycells overlap with the
+input exposure.  These overlaps are determined by the
+\ippprog{dvoImageOverlaps} program, which compares the astrometrically
+calibrated catalog from the \ippstage{camera} stage to the
+\ippdbtable{SkyTable} entries.  The output of this command is used to
+populate the \ippdbtable{warpSkyCellMap} table in the database, which
+contains a row for each skycell and OTA that overlap.  This results in
+more rows than there are OTAs, as each skycell can contain
+contributions from multiple OTAs.
+
+Once this mapping has been defined, jobs to construct each skycell are
+run, passing the \ippstage{camera} stage catalog and the
+\ippstage{chip} stage images (including the variance images and the
+updated masks) to the \ippprog{pswarp} program.  For details on the
+warping algorithm, see \cite{waters2017}.  The output of this program
+are the geometrically transformed images containing all input pixels
+warped to the common skycell pixel grid, which can subsequently be
+used for stacking and difference image analysis.  The image, mask, and
+variance generated at this stage will be available from the image
+extraction tools at the MAST archive at STScI as part of the DR2 data
+release.  A catalog is also generated containing the locations of
+sources from the input catalog that fall within area of the
+\ippstage{warp}.
+
+When the jobs have completed, an entry for the skycell is added to the
+\ippdbtable{warpSkyfile} database table, linked to the
+\ippdbtable{warpRun} entry by a common \ippdbcolumn{warp\_id}.  An
+\ippmisc{advance} task again checks that all potential skycells have
+been generated.  At this point, the direct promotion of exposures from
+one stage to the next stops, as the logic for matching exposures for
+combination is more complicated than simply adding a single entry (as
+discussed above).
+
+\subsection{Stack Combination}
+\label{subsec: stack}
+
+The skycell images generated by the \ippstage{warp} process are added
+together to make deeper, higher signal-to-noise images in the
+\ippstage{stack} stage.  These stacked images also fill in coverage
+gaps between different exposures, resulting in an image of the sky
+with more uniform coverage than a single exposure.
+
+In the IPP processing, stacks may be made with various options for the
+input images.  During nightly science processing, the 8 exposures per
+filter for each Medium Deep field are combined into a set of stacks
+for that field.  These so-called `nightly stacks' are used by the
+transient survey projects to detect faint supernovae, among other
+transient events.  For the PV3 $3\pi$ analysis, all images in each
+filter from the observations for this survey were stacked together to
+generate a single set of images with $\sim 10 - 20\times$ the exposure
+of the individual survey exposures.  
+
+For the PV3 processing of the Medium Deep fields, stacks have been
+generated for the nightly groups and for the full depth using all
+exposures, producing ``deep stacks''.  In addition, a 'best seeing'
+set of stacks have been produced \note{using image quality cuts to be
+  described}.  We have also generated out-of-season stacks for the
+Medium Deep fields, in which all image not from a particular observing
+season for a field are combined into a stack.  These later stacks are
+useful as deep templates when studying long-term transient events in
+the Medium Deep fields as they are not (or less) contaminated by the
+flux of the transients from a given season.
+
+When a given set of \ippstage{stack} stage are defined, exposures with
+existing \ippstage{warp} entries that match the filter, position, and
+other criteria such as seeing are grouped by their skycell.  An entry
+is then added for each skycell in the \ippdbtable{stackRun} table,
+with the \ippdbcolumn{warp\_id} entries for the exposures added to the
+\ippdbtable{stackInputSkyfile} table, linked to the
+\ippdbtable{stackRun} entry by the \ippdbcolumn{stack\_id} field.
+This defines the mapping for which exposures contribute to the
+\ippstage{stack}.  This breaks exposures into single skycells, but as
+adjacent \ippstage{stack} skycells may contain inputs from different
+exposures, there is no simple way to group the processing at the
+\ippstage{stack} stage into exposures.
+
+The \ippstage{stack} jobs pass the information about the input images
+and catalogs to the \ippprog{ppStack} program, which performs the
+image combinations.  See~\cite{waters2017} for details on the stack
+combination algorithm.  In addition to the standard image, mask, and
+variance produced at other stage, additional images are constructed
+with information about the contributions to each pixel.  A number
+image contains the number of input exposures used for each pixel,
+along with an exposure time map, and a weighted exposure time map that
+scales the exposure time based on the relative variance of each input.
+These images for the $3\Pi$ analysis are currently available from the
+MAST image extraction tools at STSci.
+
+Upon completing the generation of these images, a row is added to the
+\ippdbtable{stackSumSkyfile} table with statistics about
+\ippstage{stack} processing.  As this completes all processing for the
+entry, no \ippmisc{advance} job is required.
+
+\subsection{Stack Photometry}
+\label{subsec: staticsky}
+
+Although images are generated in the \ippstage{stack} stage of the
+IPP, the source detection and extraction analysis of those images is
+deferred to the \ippstage{staticsky} stage.  This separation is
+maintained because the photometry analysis of the \ippstage{stack}
+images is performed on all 5 filters simultaneously.  By deferring
+this analysis, the processing system may also decouple the generation
+of the pixels from the source detection.  This makes the sequencing of
+analysis somewhat easier and less subject to blocks due to a failure
+in the stacking analysis.  Similar to the \ippstage{stack} stage, an
+entry is created in the \ippdbtable{staticskyRun} table, linked to a
+series of rows in the \ippdbtable{staticskyInput} table by a common
+\ippdbcolumn{sky\_id}, each of which also contains the appropriate
+\ippdbcolumn{stack\_id} entries for the skycell under consideration.
+
+The input images are passed to the \ippprog{psphotStack} program,
+which does the analysis.  The stack photometry algorithms are
+described in detail in \cite{magnier2017b}.  In short, sources are
+detected in all 5 filter images down to the $5\sigma$ significance.
+The collection of detected sources is merged into a single master
+list.  If a source is detected in at least two bands, or only in
+\yps{} band, then a PSF model is fitted to the pixels of the other
+bands in which the source was not detected.  This forced photometry
+results in lower significance measurements of the flux at the
+positions of objects which are thought to be real sources, by virtue
+of triggering a detection in at least two bands.  The relaxed limit
+for \yps{} band is included to allow for searches of \yps{} dropout
+objects: it is known that faint, high-redshift quasars may be detected
+in \yps{} band only.  Sources detected only in \yps{} band are
+therefore more likely to have a higher false-positive rate than the
+other stack sources.
+
+The stack photometry output files consist of a set of FITS table
+catalogs, with one file for each filter.  Within these files, there
+are multiple table extensions that include: the measurements of
+sources based on the PSF model; aperture like parameters such as the
+Petrosian flux and radius; the convolved galaxy model fits; and the
+radial aperture measurements.  \note{is this list complete?}  Once the
+photometry is complete, a row is added to the
+\ippdbtable{staticskyResult} table with basic statistics from the
+analysis.
+
+The stack photometry output catalogs are re-calibrated for both
+photometry and astrometry in a process very similar to the
+\ippstage{camera} calibration stage.  In the case of this
+\ippstage{skycal} stage, each skycell is processed independently.
+Because of this independence, when queued for processing, the entries
+in the \ippdbtable{skycalRun} table contain the \ippdbcolumn{sky\_id}
+and \ippdbcolumn{stack\_id} entries of the parent data directly.  As
+in the \ippstage{camera} stage, the \ippprog{psastro} program reads in
+the stack photometry catalog, and produces a calibrated output.  A
+different processing recipe is supplied to \ippprog{psastro}, which
+controls for the different data.  The same reference catalog is used
+for the \ippstage{camera} and \ippstage{stack} calibration stages.
+Upon completion, the analysis statistics are written to the
+\ippdbtable{skycalResult} table. \note{Any difference in output formats?}
+
+\subsection{Forced Warp Photometry}
+\label{subsec: fullforce}
+
+Traditionally, projects which use multiple exposures to increase the
+depth and sensitivity of the observations have generated something
+equivalent to the \ippstage{stack} images produced by the IPP analysis
+(c.f, CFHT Legacy survey, COSMOS, etc).  In theory, the photometry of
+the \ippstage{stack} images produces the ``best'' photometry catalog,
+with best sensitivity and the best data quality at all magnitudes.  In
+practice, these images have some significant limitations due to the
+difficulty of modelling the PSF variations.  This difficulty is
+particularly severe for the Pan-STARRS $3\pi$ survey stacks due to the
+combination of the substantial mask fraction of the individual input
+exposures, the large instrinsic image quality variations within a
+single exposure, and the wide range of image quality conditions under
+which data were obtained and used to generate the $3\pi$ PV3 stacks.
+
+For any specific stack, the point spread function at a particular
+location is the result of the combination of the point spread
+functions for those individual exposures which went into the stack at
+that point.  Because of the high mask fraction, the exposures which
+contributed to pixels at one location may be somewhat different just a
+few tens of pixels away.  In the end, the \ippstage{stack} images have
+a effective point spread function which is not just variable, but
+changing significantly on small scales in a highly textured fashion.
+
+Any measurement which relies on a good knowledge of the PSF at the
+location of an object either needs to determine the PSF variations
+present in the \ippstage{stack} image, or the measurement will be
+somewhat degraded.  The highly textured PSF variations make this a
+very challenging problem: not only would such a PSF model require an
+unusually fine-grained PSF model, there would likely not be enough PSF
+stars in a given \ippstage{stack} image to determine the model at the
+resolution required.  The IPP photometry analysis code uses a PSF
+model with 2D variations using a grid of at most $6\times 6$ samples
+per skycell, a number reasonably well-matched to the density of stars
+at most moderate Galactic latitudes.  This scale is far too large to
+track the fine-grained changes apparent in the stack images.
+
+Thus PSF photometry as well as convolved galaxy models in the stack
+are degraded by the PSF variations.  Aperture-like measurements are in
+general not as affected by the PSF variations, as long as the aperture
+in question is large compared to the FWHM of the PSF.
+
+%% The IPP team initially explored the option of convolving each input
+%% warp to a single target PSF chosen to match the worst of the input
+%% images for a given stack.  
+
+The PV3 $3\pi$ analysis solves this problem by using the sources
+detected in the stack images and performing forced photometry on the
+individual warp images used to generate the stack.  This
+\ippstage{fullforce} analysis is performed on all warps for a single
+skycell and filter as a single unit, as this matches the arrangement
+of the input source catalog from the \ippstage{skycal} stage.  When
+processing is queued for this stage, an entry is added to the
+\ippdbtable{fullForceRun} primary database table linking to the
+specific \ippdbcolumn{skycal\_id} entry that will be used as the
+catalog for the photometry.  The \ippdbcolumn{warp\_id} values for the
+input \ippstage{warp} stage images that contributed to the
+\ippstage{stack} associated with that \ippdbcolumn{skycal\_id} are
+then added to the \ippdbtable{fullForceInput} table, linked to the
+primary table by the \ippdbcolumn{ff\_id} identifier.  The individual
+jobs for each warp are then run, which passes the \ippstage{warp}
+stage image products along with the \ippstage{skycal} catalog to the
+\ippprog{psphotFullForce} program.
+
+In this program, the positions of sources are loaded from the input
+catalog.  PSF stars are pre-identified \note{how?} and a PSF model
+generated for each \ippstage{warp} image based on those stars, using
+the same stars for all warps to the extent possible (PSF stars which
+are excessively masked on a particular image are not used to model the
+PSF).  \note{this doesn't seem correct, as each warp is run
+  independently.}  The PSF model is fitted to all of the known source
+positions in the warp images.  Aperture magnitudes, Kron magnitudes,
+and moments are also measured at this stage for each warp.  Note that
+the flux measurement for a faint, but significant, source from the
+stack image may be at a low significance (less than the $5\sigma$
+criterion used when the photometry is not run in this forced mode) in
+any individual warp image; the flux may even be negative for specific
+warps.  When combined together, these low-significance measurements
+will result in a signficant measurement as the signal-to-noise
+increases by $\sqrt{N}$.
+
+Upon completion of the forced photometry (for point sources as well as
+galaxies, discussed below), an entry is added to the
+\ippdbtable{fullForceResult} table with the processing statistics for
+that combination of \ippdbcolumn{ff\_id} and \ippdbcolumn{warp\_id}.
+Once all of the entries in the \ippdbtable{fullForceInput} table have
+finished, a summary operation is run to generate an appropriate
+average value for each measurement, by combining the measurements from
+each of the inputs.  The output catalogs listed in the
+\ippdbtable{fullForceResult} table are passed to the
+\ippprog{psphotFullForceSummary} to do this averaging.  \note{describe
+  what is done} When this completes, an entry is added to the
+\ippdbtable{fullForceSummary}, and the \ippdbtable{fullForceRun} entry
+is marked as completed.
+
+\subsubsection{Forced Galaxy Models}
+\note{CZW: is this the appropriate place for this section?}
+
+The convolved galaxy models are also re-measured on the
+\ippstage{warp} images by the \ippstage{fullforce} stage analysis.  In
+this analysis, the galaxy models determined by the
+\ippstage{staticsky} photometry analysis are used to seed the analysis
+in the individual \ippstage{warp} images.  The purpose of this
+analysis is the same as the \ippstage{fullforce} PSF photometry: the
+PSF of the \ippstage{stack} image is poorly determined due to the
+masking and PSF variations in the inputs.  Without a good PSF model,
+the PSF-convolved galaxy models are of limited accuracy.
+
+In the \ippstage{fullforce} galaxy model analysis, we assume that the
+galaxy position and position angle, along with the Sersic index if
+appropriate, have been sufficiently well determined in the
+\ippstage{staticsky} analysis.  In this case, the goal is to determine
+the best values for the major and minor axis of the elliptical contour
+and at the same time the best normalization corresponding to the best
+elliptical shape, and thus the best galaxy magnitude value.
+
+For each \ippstage{warp} image, the \ippstage{staticsky} value for the
+major and minor axis are used as the center of a $7\times{} 7$ grid
+search of the major and minor axis parameter values.  The grid spacing
+is defined as a function of the signal-to-noise of the galaxy in the
+stack image so that bright galaxies are measured with a much finer
+grid spacing that faint galaxies \note{need to quantify this}.  For
+each grid point, the major and minor axis values at that point are
+determined for the model.  The model is then generated and convolved
+with the PSF model for the \ippstage{warp} image at that point.  The
+resulting model is then compared to the \ippstage{warp} pixel data
+values and the best fit normalization value is defined.  The
+normalization and the $\chi^2$ value for each grid point is recorded.
+
+For a given galaxy, the result is a collection of $\chi^2$ values for
+each of the grid points spanning all \ippstage{warp} images.  A single
+$\chi^2$ grid can then be made by combining each grid point across the
+inputs.  The combined $\chi^2$ for a single grid point is simply the
+sum of all $\chi^2$ values at that point.  If, for a single \ippstage{warp}
+image, the galaxy model is excessively masked, then that image will be
+dropped for all grid points for that galaxy.  The reduced $\chi^2$
+values can be determined by tracking the total number of pixels
+used across all inputs to generate the combined $\chi^2$ values.  From
+the combined grid of $\chi^2$ values, the point in the grid with the
+minimum $\chi^2$ is found.  Quadratic interpolation is used to
+determine the major, minor axis values for the interpolated minimum
+$\chi^2$ value.  The errors on these two parameters is then found by
+determining the contour at which the \note{reduced?} $\chi^2$
+increases by 1.
+
+Thus the \ippstage{fullforce} galaxy analysis uses the PSF information
+from each \ippstage{warp} to determine a best set of convovled galaxy
+models for each object in the \ippstage{skycal} catalog.
+\note{discuss the subset of galaxy models and objects}.
+
+\subsection{Difference Images}
+\label{subsec: diff}
+Two of the primary science drivers for the Pan-STARRS system are the
+search hazardous asteroids and the search for Type Ia supernovae to
+measure the history of the expansion of the universe.  Both of these
+projects require the discovery of faint, transient source in the
+images.  For the hazardous asteroids, and solar system studies in
+general, the sources are transient because they are moving between
+observations; supernovae are stationary but transient in brightness.
+In both cases, the discovery of these sources can be enhanced by
+subtracting a static reference image from the image taken at a certain
+epoch.  The quality of such a difference image can be enhanced by
+convolving one or both of the images so that the PSFs in the two
+images are matched.  \note{discuss Alard-Lupton}. 
+
+In the \ippstage{diff} stage, the IPP generates diffferece images for
+appropriately specified pairs of images.  It is possible for the
+difference image to be generated from a pair of \ippstage{warp} stage
+images, from a \ippstage{warp} and a \ippstage{stack} of some variety,
+or from a pair of \ippstage{stack} stage images.  During the PS1
+survey, pairs of exposures, call TTI pairs (see~\note{Survey
+  Strategy}), were obtained for each pointing within a $\approx$ 1
+hour period in the same filter, and to the extent possible with the
+same orientation and boresite position.  The standard PS1 nightly
+processing generated difference images from the resulting pairs of
+\ippstage{warp} images.  The nightly processing generated
+\ippstage{stack} images for the Medium Deep fields, and these were
+combined with a template reference \ippstage{stack} image to generate
+``stack-stack diffs'' each night they were observed.  For the PV3
+$3\pi$ processing, the entire collection of \ippstage{warp} stage
+images for the survey were combined with images generated by the
+\ippstage{stack} processing to generate ``warp-stack diffs''.
+
+When a \ippstage{diff} processing is defined, an entry is added to the
+\ippdbtable{diffRun} table, and the appropriate input images are added
+to the \ippdbtable{diffInputSkyfile} table, with one entry for each
+skycell that are covered by the images.  For a \ippstage{diff}
+generated from two \ippstage{warp} stage products, the input images
+have their \ippdbcolumn{warp\_id} values recorded in the
+\ippdbcolumn{warp1} and \ippdbcolumn{warp2} for each skycell that
+overlaps.  If two \ippstage{stack} stages are to be used in the
+difference, their \ippdbcolumn{stack\_id} entries are recorded in the
+\ippdbcolumn{stack1} and \ippdbcolumn{stack2} fields.  As each
+\ippstage{stack} only covers a single skycell, the \ippstage{diff} is
+usually defined indirectly, using other information from the
+\ippdbtable{stackRun} table to select appropriate
+\ippdbcolumn{stack\_id} values.  Similarly, \ippstage{diff} processing
+is defined for the mixed case by creating entries that populate one of
+\ippdbcolumn{warp1} and \ippdbcolumn{stack1} and populating one of
+\ippdbcolumn{warp2} and \ippdbcolumn{stack2}.  In all cases, the
+minuend of the subtraction to be performed is the ``1'' entry, and the
+subtrahend is the ``2'' entry.
+
+Jobs are created based on the entries of
+\ippdbtable{diffInputSkyfile}, with the appropriate images and
+catalogs passed to the \ippprog{ppSub} program.  This does the
+subtraction, as well as the photometry of any sources detected in the
+\ippstage{diff} image.  The algorithm used for PSF matching is
+described in \citet{waters2017}.  Upon completion of these jobs,
+statistics about the processing are written to an entry in the
+\ippdbtable{diffSkyfile} table.  An \ippmisc{advance} checks for the
+completion of all of the components listed in
+\ippdbtable{diffInputSkyfile}, and marks the \ippdbtable{diffRun}
+entry as such.
+
+\section{IPP Software Subsystems}
+\label{sec: subsystems}
+
+The IPP relies on a number of common libraries and programs to handle
+various tasks that are shared between multiple stages of the
+processing.  These subsystems are described in this section, to
+provide an introduction to these essential components that underlie
+the rest of the pipeline.
 
 \subsection{Nebulous}
@@ -929,659 +1694,4 @@
 \note{This likely needs cleaning up and more information.}
 
-\section{IPP Data Processing Stages}
-\label{sec: stages}
-
-
-\subsection{Summit copy}
-\label{subsec: summit copy}
-
-As exposures are taken by the PS1 telescope \& GPC1 camera system, the
-data from the 60 OTA devices are read out by the camera software
-wsystem and written to disk on a collection of computers at the summit
-in the PS1 facility called ``pixel servers.'' After the images are
-written to disk, a summary listing of the information about the
-exposure and the chip images are added to the summit datastore.
-
-During night-time operations, while the summit datastore is being
-populated, the IPP subsystem called \ippstage{summitcopy} monitors the
-datastores listed in the \ippdbtable{pzDatastore} table of the
-database in order to discover new exposures ready for download.  Once
-a new exposure has been listed on the datastore, \ippstage{summitcopy}
-adds an entry of the exposure to a table in the processing database
-(\ippdbtable{summitExp}), indexed by an identifier that simply
-increments the number of exposures announced by the summit, the
-\ippdbcolumn{summit\_id}.  This tells the \ippstage{summitcopy} system
-to look for the list of chips, which are then added to another table
-(\ippdbtable{summitImfile}).  This system then attempts to download
-the chips (registering the results of those operations into the
-\ippdbtable{pzDownloadExp} and \ippdbtable{pzDownloadImfile} tables)
-from the summit pixel servers via an http request.  As the image files
-are downloaded, their MD5 checksum values are calculated and compared
-with the value reported by the summit datastore.  Download failures
-are rare and marked with a non-zero \ippdbcolumn{fault}, allowing for
-a manual recovery, rather than automatically rejecting the failed
-chips.  Once all the components of the exposure have been downloaded,
-they are further entered into the \ippdbtable{newExp} and
-\ippdbtable{newImfile} tables, which index the exposures by
-\ippdbcolumn{exp\_id}.  This switch in index indicates that the
-exposure has successfully been copied from the summit to the IPP
-cluster, and that further processing is no longer dependent on outside
-resources.
-
-\subsection{Image Registration}
-\label{subsec: registration}
-
-Once the chips for an exposure have all been downloaded, the exposure
-is ready to be registered.  In this context, `registration' refers to
-the process of adding them to the database listing of known, raw
-exposures (not to be confused with 'registration' in the sense of
-pixel re-alignment).  The result of the registration analysis is an
-entry for each exposure in the \ippdbtable{rawExp} table, and one for
-each chip in the \ippdbtable{rawImfile} table.  These tables are
-critical for downstream processing to identify what exposures are
-available for processing in any other stage.  At the registration
-stage, a large amount of descriptive metadata for each chip is added
-to the \ippdbtable{rawImfile} table, the majority of which is
-extracted from the chip FITS file headers (e.g., RA, DEC, FILTER) and
-some of which is determined by a quick analysis of the pixels (e.g.,
-mean pixel values, standard deviation).  The chip-level information is
-merged into a set of exposure-level metadata and added to the
-\ippdbtable{rawExp} table entry.  The exposure-level metadata may be
-the same as any one of the chip, in a case where the values are
-duplicated across the chip files (e.g., the name of the telescope or
-the date \& time of the exposure), or it may be a calculation based on
-the values from each chip (e.g., average of the average pixel values).
-
-Unlike much of the rest of the IPP stage, the raw exposures may only
-have a single entry in the registration tables of the processing
-database tables (\ippdbtable{rawExp} and \ippdbtable{rawImfile}).
-
-For GPC1, the image registration stage is also the stage at which the
-\ippprog{burntool} analysis is run.  This analysis is more completely
-described in \citet{waters2017}.  In brief, the \ippprog{burntool}
-program identifies bright sources on the image, and identifies
-persistence trails that result from the incomplete transfer of charge.
-As this charge can leak out in subsequent exposures, the burntool
-analysis is run sequentially on the exposures, based on the
-observation date and time listed in the headers, with the results
-stored in an text table.  As a result of the sequential nature of this
-analysis, the registration of exposures is blocked until the
-\ippprog{burntool} has been run on the previous exposures.
-
-Once the registration process has finished, new science exposures that
-have an \ippdbcolumn{obs\_mode} value that indicates they are part of
-a particular science survey are automatically launched into the
-science analysis by defining entries for the \ippstage{chip}
-processing stage, as described above.  This analysis can be relaunched
-multiple times, such as for the large scale PV3 reprocessing.
-However, this automatic process ensures the shortest time between
-observation and analysis, which is particularly important in the
-search for transient sources.
-
-\subsection{Chip Processing}
-\label{subsec: chip}
-
-The science analysis of an exposure begins with the \ippstage{chip}
-stage, which operates on the individual OTA image files.  This
-analysis step has two main goals: detrending the image to remove the
-instrumental signature from the pixel values, and the detection of
-astronomical sources in the objects.  Based on the entry the
-\ippdbtable{chipRun} primary table defining the processing details
-(with the \ippdbcolumn{state} column indicating it needs processing),
-and the associated information listed in the \ippdbtable{rawImfile},
-jobs can be spawned for each component OTA.  The \ippprog{pantasks}
-environment managing the jobs attempts to target the processing host
-to one that should host the OTA, to reduce number of operations done
-on remote data.  In practice, this targeted processing has not had as
-large of an effect as was originally intended, as the data volume has
-reduced the ability of any one node to reliably contain a particular
-OTA.  The targeted processing has probably reduced the network load
-somewhat but it has not been as critical of a requirement as
-originally expected.
-
-%% In the \ippstage{chip} stage,
-%% the individual OTA image files are processed independently in parallel
-%% within the data processing cluster.  \note{move this to kihei
-%%   discussion?} Within the processing computer cluster, most of the
-%% data storage resources are in the form of computers with large raids
-%% as well as substantial processing capability.  The processing system
-%% attempts to locate one copy of specific raw registered data on
-%% pre-defined computers that have been set as storage targets for that
-%% OTA.  The processing system is aware of this data localization and
-%% attempts to target the processing for each OTA to the machine on which
-%% the data for that detector is stored.  The output products are then
-%% primarily saved back to the same machine.  This `targetted' processing
-%% was an early design choice to minimize the system wide network load
-%% during processing.  In practice, as computer disks filled up at
-%% different rates, the data has not been localized to a very high
-%% degree.  
-
-The actual image processing is performed by the \ippprog{ppImage}
-program.  This program reads the raw data into memory and applies the
-detrend corrections \citep[see][]{waters2017} to each cell in the OTA
-(which are stored as different extensions in the FITS file format),
-and then mosaics the cells into a single contiguous \ippstage{chip}
-stage image.  This step also creates in memory additional images to
-hold the mask data, which indicates which pixels may not be valid, and
-the variance image, constructed as the Poissonian noise on the number
-of electrons detected based on the original pixel value and the
-detector gain.  A background model is then fit across the image and
-subtracted to remove the expected contribution from the sky
-\citep[see][]{waters2017} for details.
-
-With the image calibration procedure finished, object identification
-and photometry can be performed.  Although this can be done using a
-stand alone program, \ippprog{psphot}, the underlying functions are
-contained in a library that allows \ippprog{ppImage} to directly do
-this analysis, removing the need to write out and re-read the image
-data.  The details of the detection and characterization of the
-sources in the image are provided in \citet{magnier2017b}.  
-
-The results of the image processing are then written to disk,
-including the science, mask, and variance images, the background model
-subtracted, the PSF model used in the photometry process, and a FITS
-catalog of detected sources.  Additional binned images of the full OTA
-are also saved, providing $16\times{}16$ and $256\times{}256$ pixel
-binning scales for quick visualization.  The processing log and a
-selection of summary metadata describing the processing results are
-also written to disk.  This metadata is used to populate a row in the
-\ippdbtable{chipProcessedImfile} table (linked to the
-\ippdbtable{chipRun} entry by a shared \ippdbcolumn{chip\_id} value)
-to indicate that the processing of this OTA is complete.
-
-As each OTA is processed independently of the others across a number
-of computers, the \ippprog{pantasks} managing the jobs periodically
-runs an \ippmisc{advance} task that checks that the number of rows in
-\ippdbtable{chipProcessedImfile} with \ippdbcolumn{fault} equal to
-zero matches the associated number of rows in \ippdbtable{rawImfile}.
-If this condition is met, than all processing for that exposure is
-finished, and the \ippdbcolumn{state} field is set to ``full''.  If
-the \ippdbtable{chipRun}.\ippdbcolumn{end\_stage} field is set to
-\ippstage{chip}, then no further action is taken.  However, this field
-is usually set to a subsequent stage (most often \ippstage{warp}),
-then an entry for this exposure is added to the \ippdbtable{camRun}
-table, and processing continues.
-
-%% The \ippstage{chip} processing stage consists of: reading the raw image into
-%% memory, applying the detrending steps \citep[see][]{waters2017},
-%% stiching the individual OTA cells into a single chip image, detection
-%% and characterization of the sources in the image
-%% \citep[see][]{magnier2017b}, and output of the various data products.
-%% These include the detrended chip image, variance image, and mask
-%% image, as well as the FITS catalog of detected sources.  The PSF model
-%% and background model are also saved, along with a processing log.  A
-%% selection of summary metadata describing the processing results are
-%% saved and written to the processing database along with the completion
-%% status of the process.  Finally, binned chip images are generated (on
-%% two scales, binned by 16 and 256 pixels) for use in the visualization
-%% system of the processing monitor tool. \note{describe elsewhere?}
-
-%% The database structure for the \stage{chip} stage mimics that of raw
-%% data, with a \ippdbtable{chipRun} characterizing the processing of a
-%% single exposure, mapping to a set of \ippdbtable{chipProcessedImfile}
-%% entries for each OTA via a common \ippdbcolumn{chip\_id}.  
-
-\subsection{Camera Calibration}
-\label{subsec: camera}
-
-After sources have been detected and measured for each of the chips,
-the next stage is to perform a basic calibration of the full exposure
-in the \ippstage{camera} stage.  This runs as a single job for the
-entire exposure, passing the collection of FITS table catalogs
-generated from each OTA in the \ippstage{chip} stage to the
-\ippprog{psastro} program.  Although the full catalog is loaded, the
-calibration primarily concerns the positions ($x_{\rm ccd}, y_{\rm
-  ccd}$) and the instrumental PSF magnitudes.  The header information
-in these catalogs is used to determine the coordinates of the
-telescope boresite (RA, DEC, position angle).  These three coordinates
-are used, along with a pre-determined model of the OTA layout within
-the camera, to generate an initial guess for the astrometry of each
-chip.  Reference star coordinates and magnitudes are loaded from a
-reference catalog for a region corresponding to the boundaries of the
-exposure, padded by a large fraction (25\%) of the exposure diameter
-to help guarantee a solution in the case of a modest pointing error.
-The guess astrometry is used to match the reference catalog to the
-observed stellar positions in the focal plane coordinate system.  Once
-an acceptable match is found, the astrometric calibration of the
-individual chips is performed, including a fit to a single model for
-the distortion introduced by the camera optics.  After the astrometic
-analysis is completed, the photometric calibration is determined using
-the final match to the reference catalog.  At this stage,
-pre-determined color terms may be included to convert the reference
-photometry to an appropriate photometric system.  For PS1, this is
-used to generate synthetic w-band photometry for areas where no
-PS1-based calibrated w-band photometry is available.  For more
-details, see \cite{magnier2017c}.  The result of these calibrations is
-stored as a single multi-extension FITS table containing the results
-from each OTA as a separate extension.
-
-In addition to the astrometric and photometric calibrations, the
-\ippstage{camera} stage also generates the dynamic masks for the
-images.  These include masking for optical ghosts, glints, saturated
-stars, diffraction spikes, and electronic crosstalk.  The mask images
-generated by the \ippstage{chip} stage are updated with these dynamic
-masks and a new set of files are saved for the downstream analysis
-stages.  The \ippstage{camera} stage also merges the binned chip
-images (see~\ref{sec:chip}) into single jpeg images of the entire
-focal plane.  These jpeg images can then be displayed by the process
-monitoring system to visualize the data processing.
-
-Again, summary metadata is saved to disk as well, and the results
-listed therein are used to populate a row in the
-\ippdbtable{camProcessedExp} database table.  As the full exposure is
-processed all at once, this update also updates the associated
-\ippdbtable{camRun} entry, linked by the \ippdbcolumn{cam\_id}.  As
-with the \ippstage{chip} stage, the
-\ippdbtable{camRun}.\ippdbcolumn{end\_stage} is for a subsequent
-stage, an appropriate entry is added to the \ippdbtable{fakeRun}
-table.
-
-\subsection{Fake Analysis}
-\label{subsec: fake}
-
-The \ippstage{fake} stage was originally designed to do false source
-injection and recovery, in order to determine the detection efficiency
-of sources on the exposure.  However, early in the design of the IPP,
-this task was moved to the rest of the photometry analysis done at the
-\ippstage{chip} stage.  Removing the stage would require significant
-changes to the database schema.  As a result, this conveniently named
-stage generally does no actual data processing, and consists mainly of
-database operations to move the exposure on to the \ippstage{warp}
-stage.  The operations mimic the \ippstage{chip} stage, with
-individual jobs run for each OTA that update rows in the
-\ippdbtable{fakeProcessedImfile}, and an \ippmisc{advance} task that
-updates the \ippdbtable{fakeRun} table and promotes the exposure to
-the next stage by adding a row to the \ippdbtable{warpRun} table.
-
-\subsection{Image Warping}
-\label{subsec: warp}
-
-The \ippstage{warp} stage moves the data from a given exposure beyond
-away from being camera specific and towards a uniform sky oriented
-arrangement.  There are a number of ``tessellations'' defined and used
-by the IPP to define the extent and scaling of images on this uniform
-arrangement.  A tessellation can be defined for a limited region, such
-as M31 or other fields of particular interest that can be well
-described by a single tangent plane projection, or for larger regions
-which have multiple projection centers.  For the $3\Pi$ survey, the
-\ippmisc{RINGS.V3} tessellation was used that used projection centers
-spaced every four degrees in both RA and DEC, with $0\farcs{}25$
-pixels.  These projections are further broken down into ``skycells''
-that form a $10\times{}10$ grid within the projection, with an overlap
-region of 60" between adjacent skycells to ensure that objects are not
-split on all images. 
-
-These tessellations are stored in the DVO format, with
-\ippdbtable{SkyTable} entries defining the projection centers and
-image boundaries for all the skycells.  The first step of the
-\ippstage{warp} stage is determining which skycells overlap with the
-input exposure.  These overlaps are determined by the
-\ippprog{dvoImageOverlaps} program, which compares the astrometrically
-calibrated catalog from the \ippstage{camera} stage to the
-\ippdbtable{SkyTable} entries.  The output of this command is used to
-populate the \ippdbtable{warpSkyCellMap} table in the database, which
-contains a row for each skycell and OTA that overlap.  This results in
-more rows than there are OTAs, as each skycell can contain
-contributions from multiple OTAs.
-
-Once this mapping has been defined, jobs to construct each skycell are
-run, passing the \ippstage{camera} stage catalog and the
-\ippstage{chip} stage images (including the variance images and the
-updated masks) to the \ippprog{pswarp} program.  For details on the
-warping algorithm, see \cite{waters2017}.  The output of this program
-are the geometrically transformed images containing all input pixels
-warped to the common skycell pixel grid, which can subsequently be
-used for stacking and difference image analysis.  The image, mask, and
-variance generated at this stage will be available from the image
-extraction tools at the MAST archive at STScI as part of the DR2 data
-release.  A catalog is also generated containing the locations of
-sources from the input catalog that fall within area of the
-\ippstage{warp}.
-
-When the jobs have completed, an entry for the skycell is added to the
-\ippdbtable{warpSkyfile} database table, linked to the
-\ippdbtable{warpRun} entry by a common \ippdbcolumn{warp\_id}.  An
-\ippmisc{advance} task again checks that all potential skycells have
-been generated.  At this point, the direct promotion of exposures from
-one stage to the next stops, as the logic for matching exposures for
-combination is more complicated than simply adding a single entry (as
-discussed above).
-
-\subsection{Stack Combination}
-\label{subsec: stack}
-
-The skycell images generated by the \ippstage{warp} process are added
-together to make deeper, higher signal-to-noise images in the
-\ippstage{stack} stage.  These stacked images also fill in coverage
-gaps between different exposures, resulting in an image of the sky
-with more uniform coverage than a single exposure.
-
-In the IPP processing, stacks may be made with various options for the
-input images.  During nightly science processing, the 8 exposures per
-filter for each Medium Deep field are combined into a set of stacks
-for that field.  These so-called `nightly stacks' are used by the
-transient survey projects to detect faint supernovae, among other
-transient events.  For the PV3 $3\pi$ analysis, all images in each
-filter from the observations for this survey were stacked together to
-generate a single set of images with $\sim 10 - 20\times$ the exposure
-of the individual survey exposures.  
-
-For the PV3 processing of the Medium Deep fields, stacks have been
-generated for the nightly groups and for the full depth using all
-exposures, producing ``deep stacks''.  In addition, a 'best seeing'
-set of stacks have been produced \note{using image quality cuts to be
-  described}.  We have also generated out-of-season stacks for the
-Medium Deep fields, in which all image not from a particular observing
-season for a field are combined into a stack.  These later stacks are
-useful as deep templates when studying long-term transient events in
-the Medium Deep fields as they are not (or less) contaminated by the
-flux of the transients from a given season.
-
-When a given set of \ippstage{stack} stage are defined, exposures with
-existing \ippstage{warp} entries that match the filter, position, and
-other criteria such as seeing are grouped by their skycell.  An entry
-is then added for each skycell in the \ippdbtable{stackRun} table,
-with the \ippdbcolumn{warp\_id} entries for the exposures added to the
-\ippdbtable{stackInputSkyfile} table, linked to the
-\ippdbtable{stackRun} entry by the \ippdbcolumn{stack\_id} field.
-This defines the mapping for which exposures contribute to the
-\ippstage{stack}.  This breaks exposures into single skycells, but as
-adjacent \ippstage{stack} skycells may contain inputs from different
-exposures, there is no simple way to group the processing at the
-\ippstage{stack} stage into exposures.
-
-The \ippstage{stack} jobs pass the information about the input images
-and catalogs to the \ippprog{ppStack} program, which performs the
-image combinations.  See~\cite{waters2017} for details on the stack
-combination algorithm.  In addition to the standard image, mask, and
-variance produced at other stage, additional images are constructed
-with information about the contributions to each pixel.  A number
-image contains the number of input exposures used for each pixel,
-along with an exposure time map, and a weighted exposure time map that
-scales the exposure time based on the relative variance of each input.
-These images for the $3\Pi$ analysis are currently available from the
-MAST image extraction tools at STSci.
-
-Upon completing the generation of these images, a row is added to the
-\ippdbtable{stackSumSkyfile} table with statistics about
-\ippstage{stack} processing.  As this completes all processing for the
-entry, no \ippmisc{advance} job is required.
-
-\subsection{Stack Photometry}
-\label{subsec: staticsky}
-
-Although images are generated in the \ippstage{stack} stage of the
-IPP, the source detection and extraction analysis of those images is
-deferred to the \ippstage{staticsky} stage.  This separation is
-maintained because the photometry analysis of the \ippstage{stack}
-images is performed on all 5 filters simultaneously.  By deferring
-this analysis, the processing system may also decouple the generation
-of the pixels from the source detection.  This makes the sequencing of
-analysis somewhat easier and less subject to blocks due to a failure
-in the stacking analysis.  Similar to the \ippstage{stack} stage, an
-entry is created in the \ippdbtable{staticskyRun} table, linked to a
-series of rows in the \ippdbtable{staticskyInput} table by a common
-\ippdbcolumn{sky\_id}, each of which also contains the appropriate
-\ippdbcolumn{stack\_id} entries for the skycell under consideration.
-
-The input images are passed to the \ippprog{psphotStack} program,
-which does the analysis.  The stack photometry algorithms are
-described in detail in \cite{magnier2017b}.  In short, sources are
-detected in all 5 filter images down to the $5\sigma$ significance.
-The collection of detected sources is merged into a single master
-list.  If a source is detected in at least two bands, or only in
-\yps{} band, then a PSF model is fitted to the pixels of the other
-bands in which the source was not detected.  This forced photometry
-results in lower significance measurements of the flux at the
-positions of objects which are thought to be real sources, by virtue
-of triggering a detection in at least two bands.  The relaxed limit
-for \yps{} band is included to allow for searches of \yps{} dropout
-objects: it is known that faint, high-redshift quasars may be detected
-in \yps{} band only.  Sources detected only in \yps{} band are
-therefore more likely to have a higher false-positive rate than the
-other stack sources.
-
-The stack photometry output files consist of a set of FITS table
-catalogs, with one file for each filter.  Within these files, there
-are multiple table extensions that include: the measurements of
-sources based on the PSF model; aperture like parameters such as the
-Petrosian flux and radius; the convolved galaxy model fits; and the
-radial aperture measurements.  \note{is this list complete?}  Once the
-photometry is complete, a row is added to the
-\ippdbtable{staticskyResult} table with basic statistics from the
-analysis.
-
-The stack photometry output catalogs are re-calibrated for both
-photometry and astrometry in a process very similar to the
-\ippstage{camera} calibration stage.  In the case of this
-\ippstage{skycal} stage, each skycell is processed independently.
-Because of this independence, when queued for processing, the entries
-in the \ippdbtable{skycalRun} table contain the \ippdbcolumn{sky\_id}
-and \ippdbcolumn{stack\_id} entries of the parent data directly.  As
-in the \ippstage{camera} stage, the \ippprog{psastro} program reads in
-the stack photometry catalog, and produces a calibrated output.  A
-different processing recipe is supplied to \ippprog{psastro}, which
-controls for the different data.  The same reference catalog is used
-for the \ippstage{camera} and \ippstage{stack} calibration stages.
-Upon completion, the analysis statistics are written to the
-\ippdbtable{skycalResult} table. \note{Any difference in output formats?}
-
-\subsection{Forced Warp Photometry}
-\label{subsec: fullforce}
-
-Traditionally, projects which use multiple exposures to increase the
-depth and sensitivity of the observations have generated something
-equivalent to the \ippstage{stack} images produced by the IPP analysis
-(c.f, CFHT Legacy survey, COSMOS, etc).  In theory, the photometry of
-the \ippstage{stack} images produces the ``best'' photometry catalog,
-with best sensitivity and the best data quality at all magnitudes.  In
-practice, these images have some significant limitations due to the
-difficulty of modelling the PSF variations.  This difficulty is
-particularly severe for the Pan-STARRS $3\pi$ survey stacks due to the
-combination of the substantial mask fraction of the individual input
-exposures, the large instrinsic image quality variations within a
-single exposure, and the wide range of image quality conditions under
-which data were obtained and used to generate the $3\pi$ PV3 stacks.
-
-For any specific stack, the point spread function at a particular
-location is the result of the combination of the point spread
-functions for those individual exposures which went into the stack at
-that point.  Because of the high mask fraction, the exposures which
-contributed to pixels at one location may be somewhat different just a
-few tens of pixels away.  In the end, the \ippstage{stack} images have
-a effective point spread function which is not just variable, but
-changing significantly on small scales in a highly textured fashion.
-
-Any measurement which relies on a good knowledge of the PSF at the
-location of an object either needs to determine the PSF variations
-present in the \ippstage{stack} image, or the measurement will be
-somewhat degraded.  The highly textured PSF variations make this a
-very challenging problem: not only would such a PSF model require an
-unusually fine-grained PSF model, there would likely not be enough PSF
-stars in a given \ippstage{stack} image to determine the model at the
-resolution required.  The IPP photometry analysis code uses a PSF
-model with 2D variations using a grid of at most $6\times 6$ samples
-per skycell, a number reasonably well-matched to the density of stars
-at most moderate Galactic latitudes.  This scale is far too large to
-track the fine-grained changes apparent in the stack images.
-
-Thus PSF photometry as well as convolved galaxy models in the stack
-are degraded by the PSF variations.  Aperture-like measurements are in
-general not as affected by the PSF variations, as long as the aperture
-in question is large compared to the FWHM of the PSF.
-
-%% The IPP team initially explored the option of convolving each input
-%% warp to a single target PSF chosen to match the worst of the input
-%% images for a given stack.  
-
-The PV3 $3\pi$ analysis solves this problem by using the sources
-detected in the stack images and performing forced photometry on the
-individual warp images used to generate the stack.  This
-\ippstage{fullforce} analysis is performed on all warps for a single
-skycell and filter as a single unit, as this matches the arrangement
-of the input source catalog from the \ippstage{skycal} stage.  When
-processing is queued for this stage, an entry is added to the
-\ippdbtable{fullForceRun} primary database table linking to the
-specific \ippdbcolumn{skycal\_id} entry that will be used as the
-catalog for the photometry.  The \ippdbcolumn{warp\_id} values for the
-input \ippstage{warp} stage images that contributed to the
-\ippstage{stack} associated with that \ippdbcolumn{skycal\_id} are
-then added to the \ippdbtable{fullForceInput} table, linked to the
-primary table by the \ippdbcolumn{ff\_id} identifier.  The individual
-jobs for each warp are then run, which passes the \ippstage{warp}
-stage image products along with the \ippstage{skycal} catalog to the
-\ippprog{psphotFullForce} program.
-
-In this program, the positions of sources are loaded from the input
-catalog.  PSF stars are pre-identified \note{how?} and a PSF model
-generated for each \ippstage{warp} image based on those stars, using
-the same stars for all warps to the extent possible (PSF stars which
-are excessively masked on a particular image are not used to model the
-PSF).  \note{this doesn't seem correct, as each warp is run
-  independently.}  The PSF model is fitted to all of the known source
-positions in the warp images.  Aperture magnitudes, Kron magnitudes,
-and moments are also measured at this stage for each warp.  Note that
-the flux measurement for a faint, but significant, source from the
-stack image may be at a low significance (less than the $5\sigma$
-criterion used when the photometry is not run in this forced mode) in
-any individual warp image; the flux may even be negative for specific
-warps.  When combined together, these low-significance measurements
-will result in a signficant measurement as the signal-to-noise
-increases by $\sqrt{N}$.
-
-Upon completion of the forced photometry (for point sources as well as
-galaxies, discussed below), an entry is added to the
-\ippdbtable{fullForceResult} table with the processing statistics for
-that combination of \ippdbcolumn{ff\_id} and \ippdbcolumn{warp\_id}.
-Once all of the entries in the \ippdbtable{fullForceInput} table have
-finished, a summary operation is run to generate an appropriate
-average value for each measurement, by combining the measurements from
-each of the inputs.  The output catalogs listed in the
-\ippdbtable{fullForceResult} table are passed to the
-\ippprog{psphotFullForceSummary} to do this averaging.  \note{describe
-  what is done} When this completes, an entry is added to the
-\ippdbtable{fullForceSummary}, and the \ippdbtable{fullForceRun} entry
-is marked as completed.
-
-\subsubsection{Forced Galaxy Models}
-\note{CZW: is this the appropriate place for this section?}
-
-The convolved galaxy models are also re-measured on the
-\ippstage{warp} images by the \ippstage{fullforce} stage analysis.  In
-this analysis, the galaxy models determined by the
-\ippstage{staticsky} photometry analysis are used to seed the analysis
-in the individual \ippstage{warp} images.  The purpose of this
-analysis is the same as the \ippstage{fullforce} PSF photometry: the
-PSF of the \ippstage{stack} image is poorly determined due to the
-masking and PSF variations in the inputs.  Without a good PSF model,
-the PSF-convolved galaxy models are of limited accuracy.
-
-In the \ippstage{fullforce} galaxy model analysis, we assume that the
-galaxy position and position angle, along with the Sersic index if
-appropriate, have been sufficiently well determined in the
-\ippstage{staticsky} analysis.  In this case, the goal is to determine
-the best values for the major and minor axis of the elliptical contour
-and at the same time the best normalization corresponding to the best
-elliptical shape, and thus the best galaxy magnitude value.
-
-For each \ippstage{warp} image, the \ippstage{staticsky} value for the
-major and minor axis are used as the center of a $7\times{} 7$ grid
-search of the major and minor axis parameter values.  The grid spacing
-is defined as a function of the signal-to-noise of the galaxy in the
-stack image so that bright galaxies are measured with a much finer
-grid spacing that faint galaxies \note{need to quantify this}.  For
-each grid point, the major and minor axis values at that point are
-determined for the model.  The model is then generated and convolved
-with the PSF model for the \ippstage{warp} image at that point.  The
-resulting model is then compared to the \ippstage{warp} pixel data
-values and the best fit normalization value is defined.  The
-normalization and the $\chi^2$ value for each grid point is recorded.
-
-For a given galaxy, the result is a collection of $\chi^2$ values for
-each of the grid points spanning all \ippstage{warp} images.  A single
-$\chi^2$ grid can then be made by combining each grid point across the
-inputs.  The combined $\chi^2$ for a single grid point is simply the
-sum of all $\chi^2$ values at that point.  If, for a single \ippstage{warp}
-image, the galaxy model is excessively masked, then that image will be
-dropped for all grid points for that galaxy.  The reduced $\chi^2$
-values can be determined by tracking the total number of pixels
-used across all inputs to generate the combined $\chi^2$ values.  From
-the combined grid of $\chi^2$ values, the point in the grid with the
-minimum $\chi^2$ is found.  Quadratic interpolation is used to
-determine the major, minor axis values for the interpolated minimum
-$\chi^2$ value.  The errors on these two parameters is then found by
-determining the contour at which the \note{reduced?} $\chi^2$
-increases by 1.
-
-Thus the \ippstage{fullforce} galaxy analysis uses the PSF information
-from each \ippstage{warp} to determine a best set of convovled galaxy
-models for each object in the \ippstage{skycal} catalog.
-\note{discuss the subset of galaxy models and objects}.
-
-\subsection{Difference Images}
-\label{subsec: diff}
-Two of the primary science drivers for the Pan-STARRS system are the
-search hazardous asteroids and the search for Type Ia supernovae to
-measure the history of the expansion of the universe.  Both of these
-projects require the discovery of faint, transient source in the
-images.  For the hazardous asteroids, and solar system studies in
-general, the sources are transient because they are moving between
-observations; supernovae are stationary but transient in brightness.
-In both cases, the discovery of these sources can be enhanced by
-subtracting a static reference image from the image taken at a certain
-epoch.  The quality of such a difference image can be enhanced by
-convolving one or both of the images so that the PSFs in the two
-images are matched.  \note{discuss Alard-Lupton}. 
-
-In the \ippstage{diff} stage, the IPP generates diffferece images for
-appropriately specified pairs of images.  It is possible for the
-difference image to be generated from a pair of \ippstage{warp} stage
-images, from a \ippstage{warp} and a \ippstage{stack} of some variety,
-or from a pair of \ippstage{stack} stage images.  During the PS1
-survey, pairs of exposures, call TTI pairs (see~\note{Survey
-  Strategy}), were obtained for each pointing within a $\approx$ 1
-hour period in the same filter, and to the extent possible with the
-same orientation and boresite position.  The standard PS1 nightly
-processing generated difference images from the resulting pairs of
-\ippstage{warp} images.  The nightly processing generated
-\ippstage{stack} images for the Medium Deep fields, and these were
-combined with a template reference \ippstage{stack} image to generate
-``stack-stack diffs'' each night they were observed.  For the PV3
-$3\pi$ processing, the entire collection of \ippstage{warp} stage
-images for the survey were combined with images generated by the
-\ippstage{stack} processing to generate ``warp-stack diffs''.
-
-When a \ippstage{diff} processing is defined, an entry is added to the
-\ippdbtable{diffRun} table, and the appropriate input images are added
-to the \ippdbtable{diffInputSkyfile} table, with one entry for each
-skycell that are covered by the images.  For a \ippstage{diff}
-generated from two \ippstage{warp} stage products, the input images
-have their \ippdbcolumn{warp\_id} values recorded in the
-\ippdbcolumn{warp1} and \ippdbcolumn{warp2} for each skycell that
-overlaps.  If two \ippstage{stack} stages are to be used in the
-difference, their \ippdbcolumn{stack\_id} entries are recorded in the
-\ippdbcolumn{stack1} and \ippdbcolumn{stack2} fields.  As each
-\ippstage{stack} only covers a single skycell, the \ippstage{diff} is
-usually defined indirectly, using other information from the
-\ippdbtable{stackRun} table to select appropriate
-\ippdbcolumn{stack\_id} values.  Similarly, \ippstage{diff} processing
-is defined for the mixed case by creating entries that populate one of
-\ippdbcolumn{warp1} and \ippdbcolumn{stack1} and populating one of
-\ippdbcolumn{warp2} and \ippdbcolumn{stack2}.  In all cases, the
-minuend of the subtraction to be performed is the ``1'' entry, and the
-subtrahend is the ``2'' entry.
-
-Jobs are created based on the entries of
-\ippdbtable{diffInputSkyfile}, with the appropriate images and
-catalogs passed to the \ippprog{ppSub} program.  This does the
-subtraction, as well as the photometry of any sources detected in the
-\ippstage{diff} image.  The algorithm used for PSF matching is
-described in \citet{waters2017}.  Upon completion of these jobs,
-statistics about the processing are written to an entry in the
-\ippdbtable{diffSkyfile} table.  An \ippmisc{advance} checks for the
-completion of all of the components listed in
-\ippdbtable{diffInputSkyfile}, and marks the \ippdbtable{diffRun}
-entry as such.
-
 \subsection{Addstar : DVO Ingest}
 \label{subsec: addstar}