Index: trunk/doc/release.2015/ps1.datasystem/datasystem.tex
===================================================================
--- trunk/doc/release.2015/ps1.datasystem/datasystem.tex	(revision 39894)
+++ trunk/doc/release.2015/ps1.datasystem/datasystem.tex	(revision 39958)
@@ -87,12 +87,13 @@
 \keywords{Surveys:\PSONE }
 
-\section{INTRODUCTION}\label{sec:intro}
+\section{Introduction}
+\label{sec: intro}
 
 This is the second in a series of seven papers describing the
 Pan-STARRS1 Surveys, the data reduction techiques and the resulting
 data products.  This paper (Paper II) describes how the various data
-processing stages are organised and implemented in the Imaging
+processing stages are organised and implemented in the Image
 Processing Pipeline (IPP), including details of the the processing
-database which is a critical element in the IPP infrastructure .
+database which is a critical element in the IPP infrastructure.
 
 %Chambers et al. 2017 (Paper I)
@@ -142,4 +143,29 @@
 described in detail in \cite{2012ApJ...750...99T}.
 
+The Pan-STARRS Image Processing Pipeline consists of a suite of
+software programs and data systems that are designed to reduce
+astronomical images, with a focus on parallelization necessary to
+speed the processing of the large images produced by the GPC1 camera.
+Part of this parallelization is derived from the fact that this camera
+consists of 60 independent orthogonal transfer array (OTA) devices,
+and can therefore be processed simultaneously.  Although there are
+multiple stages that operate on an entire exposure at once, the
+majority of stages operate only on smaller segments of a full exposure
+to allow the processing tasks to be spread over the machines in the
+processing cluster.
+
+This paper presents a description of the IPP data handling system.
+Section \ref{sec: subsystems} describes the major IPP subsystems that
+underlie the main pipeline, providing a set of common interfaces and
+tools used at multiple stages.  The main processing stages of the
+pipeline are described in Section \ref{sec: stages}, although all
+exposures may not necessarily pass through each of these stages.  The
+hardware systems that have done the processing for the PV3 data
+release are listed in Section \ref{sec: hardware}, with some details
+on the scale of computing needed to reduce this large number of
+exposures.  Finally, Section \ref{sec: discussion} presents a
+discussion of some of the lessons learned in the creation of the IPP,
+and its utility in reducing data from other cameras and telescopes.
+
 {\color{red} {\em Note: These papers are being placed on arXiv.org to
     provide crucial support information at the time of the public
@@ -151,6 +177,14 @@
 
 \section{IPP Software Subsystems}
+\label{sec: subsystems}
+
+The IPP relies on a number of common libraries and programs to handle
+various tasks that are shared between multiple stages of the
+processing.  These subsystems are described in this section, to
+provide an introduction to these essential components that underlie
+the rest of the pipeline.
 
 \subsection{Processing Database}
+\label{subsec: database}
 
 A critical element in the Pan-STARRS IPP infrastructure is the
@@ -163,5 +197,5 @@
 back to the database.  In this way, the database records this history
 of the processing, and also provides the information needed to
-successive processing stages to begin their own tasks.  
+successive processing stages to begin their own tasks.
 
 The processing database is colloquially referred to as the `gpc1'
@@ -169,58 +203,64 @@
 processing of images and data products related to the PS1 GPC1 camera.
 This same database engine also has instances for other cameras
-processed by the IPP, e.g., GPC2, the test cameras TC1, TC3, the
-Imaging Sky Probe (ISP), etc.
+processed by the IPP, e.g., GPC2, the test cameras TC1, TC3, and the
+Imaging Sky Probe (ISP).
 
 Within the processing database, the various processing stages are
 represented as a set of tables.  In general, there is a top level
-table which defines the conceptual list of processing items either to
-be done, in progress, or completed.  An associated table lists the
-details of elements which have been processed.  For example, one
-critical stage is the Chip processing stage, discussed below, in which
-the individual chips from an exposure are detrended and sources are
-detected.  Within the gpc1 database, there is a top-level table called
-`chipRun' in which each exposure has a single entry.  Associated with
-this table is the `chipProcessedImfile' table, which contains one row
-for each of the (up to 60) chips associated with the exposure.  The
-top-level tables, such as chipRun, are populated once the system has
-decided that a specific item (e.g., an exposure) should be processed
-at that stage.  Initially, the entry is given a state of `run',
-denoting that the exposure is ready to be processed.  The low-level
-table entries, such as the chipProcessedImfile entries, are only
-populated once the element (e.g., the chip) has been processed by the
-analysis system.  Once all elements for a given stage, e.g., chips in
-this case, are completed, then the status of the top-level table entry
-(chipRun) are switched from `run' to `done'.
-
-If the analysis of an element (e.g., chip) completed successfully,
-then the corresponding table row (e.g., chipProcessedImfile) is listed
-with a fault of 0.  If the analysis failed, then a non-zero fault is
-recorded.  An analysis which results in a fault is one in which the
-failure is thought to be temporary.  For example, if a computer had a
-network glitch and was unable to write out some of the output files,
-this would be an ephemeral failure which was not a failing of the
-data, but merely the processing system.  On the other hand, if the
-analysis failed because of a problem with the input data, this is
-noted by setting a non-zero value in a different table field,
-`quality'.  For example, if the chip analysis failed to discover any
-stars because the image was completely saturated, the analysis can
-complete successfully (fault = 0), but the `quality' field will be set
-to a non-zero value.  The various processing stages are able to select
-only the good (quality = 0) elements of a prior stage when choosing
-items for processing.  For example, the Camera calibration stage will
-only use data from chips with good quality data, dropping the failed
-chips from the rest of the analysis.  On the other hand, a fault in
-one of the elements for a given stage will block any dependent stages
-from processing that item.  In this way, if a glitch occurs and a chip
-from an exposure failed to be written to disk in the Chip stage, the
-system will not partially process the exposure with the rest of the
-chips.  Since many of the faults which occur are ephemeral, the
-processing stages are set up to occasional clear and re-try the
-faulted entries.  Thus, automatic processing is able to keep the data
-flowing even in the face of occasional network glitches or hardware
-crashes.
+primary table which defines the conceptual list of processing items
+either to be done, in progress, or completed.  An associated secondary
+table lists the details of elements which have been processed.  Table
+\ref{tab: database schema} contains an outline of the database schema,
+showing the relations between tables organized by processing stage.
+As an example, one critical stage is the \ippstage{chip} processing
+stage, discussed below, in which the individual chips from an exposure
+are detrended and sources are detected.  Within the gpc1 database, the
+primary table called \ippdbtable{chipRun} in which each exposure has a
+single entry.  Associated with this table is the
+\ippdbtable{chipProcessedImfile} table, which contains one row for
+each of the (up to 60) chips associated with the exposure.  The
+primary tables, such as \ippdbtable{chipRun}, are populated once the
+system has decided that a specific item (e.g., an exposure) should be
+processed at that stage.  Initially, the entry is given a state of
+``run'', denoting that the exposure is ready to be processed.  The
+low-level table entries, such as the \ippdbtable{chipProcessedImfile}
+entries, are only populated once the element (e.g., the chip) has been
+processed by the analysis system.  Once all elements for a given
+stage, e.g., chips in this case, are completed, then the status of the
+top-level table entry (\ippdbtable{chipRun}) are switched from ``run''
+to ``full''.
+
+If the analysis of an element (e.g., the individual OTA chip)
+completed successfully, then the corresponding table row (e.g.,
+\ippdbtable{chipProcessedImfile}) is listed with a \ippdbcolumn{fault}
+of 0.  If the analysis failed, then a non-zero \ippdbcolumn{fault} is
+recorded.  An analysis which results in a \ippdbcolumn{fault} is one
+in which the failure is thought to be temporary.  For example, if the
+processing computer had a network interruption and was unable to write
+some of the output files, this would be an ephemeral failure which was
+not a failing of the data, but merely the processing system.  On the
+other hand, if the analysis failed because of a problem with the input
+data, this is noted by setting a non-zero value in a different table
+field, \ippdbcolumn{quality}.  For example, if the chip analysis
+failed to discover any stars because the image was completely
+saturated, the analysis can complete successfully (\ippdbcolumn{fault}
+= 0), but the \ippdbcolumn{quality} field will be set to a non-zero
+value.  The various processing stages are able to select only the good
+(\ippdbcolumn{quality} = 0) elements of a prior stage when choosing
+items for processing.  For example, the \ippstage{camera} calibration
+stage will only use data from chips with good \ippdbcolumn{quality}
+data, dropping the failed chips from the rest of the analysis.  On the
+other hand, a \ippdbcolumn{fault} in one of the elements for a given
+stage will block any dependent stages from processing that item.  In
+this way, if such a temporary failure occurs, the system will not
+process an exposure through subsequent stages without the component
+that has failed temporarily.  Since many of the \ippdbcolumn{fault}s
+which occur are ephemeral, the processing stages are set up to
+occasional clear and re-try the faulted entries.  Thus, automatic
+processing is able to keep the data flowing even in the face of
+occasional network glitches or hardware crashes.
 
 \subsection{Nebulous}
-
+\label{subsec: nebulous}
 Storing the large volume of data that is generated by the GPC1 camera
 was recognized early in the Pan-STARRS project as a major concern.
@@ -229,30 +269,78 @@
 the files, with a Simple Object Access Protocol interface between the
 database and the other IPP programs.  The actual files are stored on
-NFS mounted partitions on a series of storage nodes in the IPP
-cluster.
-
-The original design of \ippprog{Nebulous} intended to aid in the
+NFS mounted partitions on a series of storage nodes in the IPP cluster
+that can be accessed throughout the cluster.  This distribution of
+files is useful to balance the disk I/O, as this parallelizes the
+load.
+
+The original design of \ippprog{Nebulous} was intended to aid in the
 targetted processing of data, by having specific image data (such as
 all the images from one OTA device) located on a single storage node.
 This would allow any jobs involving that data to be assigned to the
-storage node, eliminating network IO.  Important data could be
-duplicated to a further data nodes, with the alternate locations
+storage node, eliminating network I/O.  Important data could be
+duplicated to additional data nodes, with the alternate locations
 stored in the database.  In practice, however, hardware failures and
 increases in hard drive storage volumes and network bandwidth have
-reduced the amount that the IPP processing is targeted.
+reduced the degree to which the IPP processing is targeted.
 
 When a program creates a new file in \ippprog{Nebulous}, it supplies
-an URI of the form \texttt{neb://HOST.VOLUME/PATH/FILENAME}.  The host
+an URI of the form \code{neb://HOST.VOLUME/PATH/FILENAME}.  The host
 and volume specifiers are optional, and allow a file to be created on
-a specific node.  The path and filename appear as a standard full file
-location, and are used internally as the ``external id''.  A
-storage\_object is then created in the database for this id, and an
-instance of the file created on the specified node (or at random from
-available nodes if left empty).  
-
-
-\subsection{Pantasks \& Parallel Processing}
+a specific node.  The path and filename appear similar to a standard
+full file location, and are used internally as the ``external id''.  A
+storage object entry is then created in the database for this id, and
+an instance of the file created on the specified node (or at random
+from available nodes if left empty).  This instance is created in a
+deterministic filename location.  The external id is hashed using the
+SHA-1 function, and the first four hexadecimal digits of this hash are
+separated into two two-digit strings and used as the top and second
+level directory location for the disk file.  The disk file is created
+using the database instance id, and a transformed version of the
+external id, which has colons replacing any forward slash characters,
+to convert the external id path into a filename.  For the example URI
+above, this results in a file located on disk in a location like
+\code{/data/HOST.VOLUME/nebulous/d5/d8/9244993440.PATH:FILENAME}.
+This file naming structure is useful, as it duplicates database
+contents on disk.
+
+The storage volumes that contain the data on disk are defined in the
+\ippprog{Nebulous} database in a number of ways.  First, the locations
+and mount points for the actual NFS storage are listed in the
+\ippdbtable{volume} table.  This table contains columns indicating if
+the volume should be used for reading (\ippdbcolumn{available}) and
+writing (\ippdbcolumn{allocate}), allowing these properties to be
+manually set, which is useful in scheduling downtime for hardware
+issues.  Another column, \ippdbcolumn{xattr}, is used to control the
+behavior of this volume, with specific values used to denote desired
+behavior.  For instance, the value of $1$ is used to indicate that a
+volume should only be used as a backup volume (which accepts only
+replicated copies), and the value of $5$ is used to indicate that the
+volume is permanently unavailable, and should be ignored.
+
+In addition to this permanent table describing the volumes, a
+\ippdbtable{mountedvol} table also exists that only lists those
+volumes that are currently visible and accessible from the
+\ippprog{Nebulous} database server.  This table also lists the total
+and currently available disk space on each volume, allowing the
+\ippprog{Nebulous} load balancing routines to prioritize those volumes
+with large unused disk space before filling the volumes with only
+small amounts remaining.  This table is regenerated every ten to
+twenty minutes, after a scan of each of the volumes listed in the
+\ippdbtable{volume} table.
+
+The final table controlling the operations of the \ippprog{Nebulous}
+volumes is the \ippdbtable{cabinet} table, which organizes the
+individual volumes into ``cabinets,'' a concept loosly based on the
+physical arrangement of the storage servers in the data center.  These
+cabinets are used to prevent the replication of a storage object
+within a group of volumes where all instances of the object could be
+taken off line by a single failure.  As the data center cabinets share
+power supplies among the servers they contain, ensuring physical
+distance between replicated copies is important to guarantee that a
+temporary failure of one of these devices does not significantly
+impact processing.
 
 \subsection{DVO}
+\label{subsec: DVO}
 
 The Pan-STARRS IPP uses an internal database system, distinct from the
@@ -262,5 +350,5 @@
 database system, called the ``Desktop Virtual Observatory'' (DVO) was
 developed originally for the LONEOS project, and used as part of the
-CFHT Elixir system (Magnier \& Cuillandre REF).  The capabilities of
+CFHT Elixir system \citep{2004PASP..116..449M}.  The capabilities of
 this databasing system have been somewhat expanded for the Pan-STARRS
 context.  
@@ -268,34 +356,33 @@
 One of the main purposes of the DVO system is to define the
 relationship between individual detections of an astronomical object
-and the definition of that object.  Before describing the database
-schema, we will discuss the process by which detections are associated
-with objects.  New detections are generally added to the database in a
-group associated with, for example, an image from the GPC1 camera.  As
-new detections are loaded, they are compared to the objects already
-stored in the database.  If an object is already found in the database
-within the match radius, the new detection is associated to that
-object. If more than one object exists within the database, the
-detection is associated with the closest object.  
+and the definition of that object.  New detections are generally added
+to the database in a group associated with, for example, an image from
+the GPC1 camera.  As new detections are loaded, they are compared to
+the objects already stored in the database.  If an object is already
+found in the database within the match radius of \note{one arcsecond}, the new detection is
+associated to that object. If more than one object exists within the
+database, the detection is associated with the closest object.
 
 Detections in DVO have a special piece of metadata called the
-\code{photcode} which identifies the source of the measurement.  A
-\code{photcode} has a name which in general consists of the name of
-the camera or telescope (e.g., GPC1 or 2MASS), the name (or short-hand
-name) of the filter used for the measurement (e.g., $g$), and an
-identifier for the detector, if not unique (e.g., XY01 for GPC1).
-Along with each name, there is a numerical value for the photcode.  A
-table within the DVO system, \code{Photcode}, lists the photcoes and
-defines a number of additional pieces of information for each
-photcode.  These include the nominal zero point and airmass slope, as
-well as color trends to transform a measurement in the specific
-photcode to a common system.  There are 3 classes of photcodes defined
-within the DVO system.  Those photcodes associated with detections
-from an image loaded into the database system are called \code{DEP}
-photcodes.  There are also photcodes associated with the average
-photometry values, called SEC photcodes.  There are also those
-measurements which come from external data sources for which DVO does
-not have any information to determine a calibration (e.g.,
-instrumental magnitudes and detector coordinates).  These are
-measurements are reference values and are assigned REF photcodes.
+\ippdbcolumn{photcode} which identifies the source of the measurement.
+A \ippdbcolumn{photcode} has a name which in general consists of the
+name of the camera or telescope (e.g., GPC1 or 2MASS), the name (or
+short-hand name) of the filter used for the measurement (e.g.,
+\gps{}), and an identifier for the detector, if not unique (e.g., XY01
+for a GPC1 OTA).  Along with each name, there is a numerical value for
+the photcode.  A table within the DVO system, \ippdbtable{Photcode},
+lists the photcodes and defines a number of additional pieces of
+information for each photcode.  These include the nominal zero point
+and airmass slope, as well as color trends to transform a measurement
+in the specific photcode to a common system.  There are 3 classes of
+photcodes defined within the DVO system.  Those photcodes associated
+with detections from an image loaded into the database system are
+called \ippmisc{DEP} photcodes.  There are also photcodes associated with
+the average photometry values, called \ippmisc{SEC} photcodes.  There are
+also those measurements which come from external data sources for
+which DVO does not have any information to determine a calibration
+(e.g., instrumental magnitudes and detector coordinates).  These are
+measurements are reference values and are assigned \ippmisc{REF}
+photcodes.
 
 In the implementation of DVO used for the PV3 calibration analysis,
@@ -304,7 +391,7 @@
 of files for table which are spatially partitioned.  The binary FITS
 tables may be optionally compressed using the (to date) experimental
-FITS binary table compression strategy outlined by REF.  In this
+FITS binary table compression strategy outlined by \note{REF}.  In this
 compression scheme, using a strategy similar to that used for FITS
-image compression (REF), the data stored in the binary table is
+image compression (\note{REF}), the data stored in the binary table is
 compressed and stored in the 'HEAP' section of the FITS table.  In
 brief, each column in the FITS table is compressed as one (or more)
@@ -342,11 +429,12 @@
 region bounded by lines of constant RA,DEC are contained in a specific
 file.  The boundaries and the associated partition names are stored in
-one of the supporting tables, \code{SkyTable}.  This table contains
-the definitions of the boundaries for each sky region (\code{R_MIN},
-\code{R_MAX}, \code{D_MIN}, \code{D_MAX}), the name of the sky region,
-an ID (\code{INDEX}, equal to the sequence number of the region in the
+one of the supporting tables, \ippdbtable{SkyTable}.  This table
+contains the definitions of the boundaries for each sky region
+(\ippdbcolumn{R\_MIN}, \ippdbcolumn{R\_MAX}, \ippdbcolumn{D\_MIN},
+\ippdbcolumn{D\_MAX}), the name of the sky region, an ID
+(\ippdbcolumn{INDEX}, equal to the sequence number of the region in the
 table), and index entries to enable navigation within the table.  The
 regions are defined in a hierarchical sense, with a series of levels
-each containing a finer mesh of regions covering the sky.  
+each containing a finer mesh of regions covering the sky.
 
 In the default used by the PV3 DVO, the partitioning scheme is based
@@ -358,21 +446,22 @@
 4 - 8 smaller partitions.  This level exactly matches the HST GSC
 layout, and uses the same naming convention to identify the
-partitions: n0000/0000, etc. \note{more on the names?}.  Level 4
-further divides these regions by a factor of 16.  In the
-\code{SkyTable}, a region at one level has a pointer to its parent
-region (the one which contains it) and a sequence pointing to its
-children (regions it contains).  The \code{SkyTable} enables fast
-lookups of the on-disk partitions which map to a specific coordinate
-on the sky.  In general, a single DVO will have the full sky
-represented with tables at a single level, though it is possible for
-mixed levels to be used, this mode is not well tested.  For the PV3
-master database, the partitioning at the 5th level results in \approx
-150,000 regions to cover the full sky, of which \approx 110,000 are
-used for the PV3 $3\pi$ data.  The densest portions of the bulge
-contain at most \approx 300k astronomical objects in the database
-files, with an associated maximum of 30M measurements in these files.
-With the compression scheme described above, this makes the largest
-database files \approx 3GB, which can be loaded into memory in 30
-seconds on our partition machines.
+partitions: \code{n0000/0000}, etc. \note{more on the names?}.  Level
+4 further divides these regions by a factor of 16.  In the
+\ippdbtable{SkyTable}, a region at one level has a pointer to its
+parent region (the one which contains it) and a sequence pointing to
+its children (regions it contains).  The \ippdbtable{SkyTable} enables
+fast lookups of the on-disk partitions which map to a specific
+coordinate on the sky.  In general, a single DVO will have the full
+sky represented with tables at a single level, though it is possible
+for mixed levels to be used, this mode is not well tested.  For the
+PV3 master database, the partitioning at the \note{should this be
+  4th?} 5th level results in \approx 150,000 regions to cover the full
+sky, of which \approx 110,000 are used for the PV3 $3\pi$ data.  The
+densest portions of the bulge contain at most \approx 300,000
+astronomical objects in the database files, with an associated maximum
+of 30 million measurements in these files.  With the compression
+scheme described above, this makes the largest database files \approx
+3GB, which can be loaded into memory in 30 seconds on the processing
+machines that contain partition data.
 
 The DVO software system allows the tables which are partitioned across
@@ -380,5 +469,5 @@
 call partition hosts.  A single file defines the names of these
 partition hosts and the location of the database partition on the
-disks of that machine.  The \code{SkyTable} contains elements to
+disks of that machine.  The \ippdbtable{SkyTable} contains elements to
 define by ID the parition host to which a partitioned set of tables
 has been assigned.  Operations which query the database, or perform
@@ -395,109 +484,123 @@
 manipulating the enormous database on a reasonable timescale.
 
-\subsection{Tables which describe objects} 
+\subsubsection{Astronomical Objects}
 
 Two tables carry the most important information about the astronomical
-objects in the database: Average and SecFilt.  These two tables
-specify the main average properties of the astronomical object.  The
-Average table includes the astrometric information ($\alpha, \delta,
-\mu \alpha, \mu \delta, \pi$) and associated errors, data quality
-flags for each object, links to the other tables, and a number of IDs,
-with one row for each astronomical object.  \note{go into complete
-  detail here on the IDs?}.  The SecFilt table\footnote{The name
-  SecFilt is a bit of a historical misnomer: originally, DVO was
-  designed for a monochromatic survey and data for a single
-  photometric band was maintained in the Average table.  Later, DVO
-  was adapted to a multifilter system and additional filters were
-  added to the SecFilt (Secondary Filter) table.  Eventually, the
-  schema was normalized and all photometric data placed in SecFilt,
-  with the Primary filter concept being dropped, but the name has
-  since stuck.} contains average photometric information for a
-collection of filters.  A given DVO instance has a specified set of
-filters for which average photometry is stored in the SecFilt table.
-The number and choice of filters for the SecFilt may be modified by
-the database administrator fairly easily, but the process of updating
-the database is somewhat expensive (\approx 24 hours for the current
-PV3 instance).  Thus the choice is semi-static for a given DVO
-instance.  In the case of the PV3 DVO instance, 9 average bandpasses
-are defined: {\it g, r, i, z, y, J, H, K, w}.  The SecFilt table
-contains one row for each filter for each object, thus the PV3 DVO
-contains 9 times as many rows as the Average table.  The order of the
-table is fixed in relation to the Average table: row $i$ of Average
-defines the object with photometry contained in rows $9i \rightarrow 9i +
-8$ ($i$ is zero counting).  
+objects in the database: \ippdbtable{Average} and
+\ippdbtable{SecFilt}.  These two tables specify the main average
+properties of the astronomical object.  The \ippdbtable{Average} table includes the
+astrometric information ($\alpha, \delta, \mu \alpha, \mu \delta,
+\pi$) and associated errors, data quality flags for each object, links
+to the other tables, and a number of IDs, with one row for each
+astronomical object.  \note{go into complete detail here on the IDs?}.
+The \ippdbtable{SecFilt} table\footnote{The name \ippdbtable{SecFilt}
+  is a bit of a historical misnomer: originally, DVO was designed for
+  a monochromatic survey and data for a single photometric band was
+  maintained in the Average table.  Later, DVO was adapted to a
+  multifilter system and additional filters were added to the
+  \ippdbtable{SecFilt} (Secondary Filter) table.  Eventually, the
+  schema was normalized and all photometric data placed in
+  \ippdbtable{SecFilt}, with the Primary filter concept being dropped,
+  but the name has since stuck.} contains average photometric
+information for a collection of filters.  A given DVO instance has a
+specified set of filters for which average photometry is stored in the
+\ippdbtable{SecFilt} table.  The number and choice of filters for the
+\ippdbtable{SecFilt} may be modified by the database administrator
+fairly easily, but the process of updating the database is somewhat
+expensive (\approx 24 hours for the current PV3 instance).  Thus the
+choice is semi-static for a given DVO instance.  In the case of the
+PV3 DVO instance, 9 average bandpasses are defined: \gps{}, \rps{},
+\ips{}, \zps{}, \yps{}, {\it J, H, K}, and \wps{}.  The
+\ippdbtable{SecFilt} table contains one row for each filter for each
+object, thus the PV3 DVO contains 9 times as many rows as the
+\ippdbtable{Average} table.  The order of the table is fixed in
+relation to the \ippdbtable{Average} table: row $i$ of
+\ippdbtable{Average} defines the object with photometry contained in
+rows $9i \rightarrow 9i + 8$ ($i$ is zero counting).
 
 The individual measurements of the astronomical objects are carried in
-the table \code{Measure}.  This table lists the values measured by
-\code{psphot} for each chip, warp, or stack image.  This includes the
-instrumental magnitudes for the PSF, aperture, and Kron photometry;
-raw position (Xccd, Yccd) and second moments (Mxx, Myy, Mxy), along
-with shape parameters of the PSF model at the position of the object
-(FWx, FWy, theta).  This table also includes metadata such as the
-exposure time, the date \& time of the observation, airmass, azimuth,
-and information specifying the filter \note{describe the photcodes}.
-The \code{Measure} table also carried the calibration magnitude offsts
-($M_{\rm cal}$ and $M_{\rm flat}$ discussed below) and the
-astrometrically calibrated position.  Astrometric offsets for several
-systematic corrections discussed below are also defined for each
-measurement.  Since stacks and forced warp photometry may have
-non-significant values, the table is somewhat de-normalized in that it
-also carried instrumental flux values for the PSF, aperture, and Kron
-photometry.  
-
-In the \code{Measure} table, there are three fields which provide two
-independent links from the specific measurement to the associated
-object: \code{Measure.catID} specifies the spatial partition to which
-the measurement belongs; \code{Measure.objID} specifies to which entry
-in the \code{Average} table the measurement belongs.  These two 32 bit
-fields can thus be combined into a single 64 bit unique ID for all
-objects in the database.  In addition, the field \code{Measure.averef}
-specifies the row number in the \code{Average} table of the associated
-object.  The \code{Measure} table may be unsorted, in which case it is
-slow to find the measurements associated with a specific object (a
-full table scan is required).  After the table is sorted, the
-\code{Measure} rows for a given object are grouped together.  In the
-case, the fields \code{Average.measureOffset} and
-\code{Average.Nmeasure} define an index for the code to jump to the
-list of measurements for a single object.  The field
-\code{Measure.imageID} defines the link from the measurement to the
-image which supplied the measurement.
+the table \ippdbtable{Measure}.  This table lists the values measured
+by \ippprog{psphot} for each \ippstage{chip}, \ippstage{warp}, or
+\ippstage{stack} stage image.  This includes the instrumental magnitudes for
+the PSF, aperture, and Kron photometry; raw position
+(\ippdbcolumn{Xccd}, \ippdbcolumn{Yccd}) and second moments
+(\ippdbcolumn{Mxx}, \ippdbcolumn{Myy}, \ippdbcolumn{Mxy}), along with
+shape parameters of the PSF model at the position of the object
+(\ippdbcolumn{FWx}, \ippdbcolumn{FWy}, \ippdbcolumn{theta}).  Metadata
+about the exposure that the measurement was derived from is also
+include, such as the exposure time, the date \& time of the
+observation, airmass, azimuth, and \ippdbcolumn{photcode} information
+specifying the filter.  The \ippdbtable{Measure} table also carries
+the calibration magnitude offsts ($M_{\rm cal}$ and $M_{\rm flat}$,
+discussed below) and the astrometrically calibrated position.
+Astrometric offsets for several systematic corrections discussed below
+are also defined for each measurement.  Since stacks and forced warp
+photometry may have non-significant values, the table is somewhat
+de-normalized in that it also carries instrumental flux values for the
+PSF, aperture, and Kron photometry.
+
+In the \ippdbtable{Measure} table, there are three fields which
+provide two independent links from the specific measurement to the
+associated object: \ippdbtable{Measure}.\ippdbcolumn{catID} specifies
+the spatial partition to which the measurement belongs;
+\ippdbtable{Measure}.\ippdbcolumn{objID} specifies to which entry in
+the \ippdbtable{Average} table the measurement belongs.  These two 32
+bit fields can thus be combined into a single 64 bit ID unique for all
+objects in the database.  In addition, the field
+\ippdbtable{Measure}.\ippdbcolumn{averef} specifies the row number in
+the \ippdbtable{Average} table of the associated object.  The
+\ippdbtable{Measure} table may be unsorted, in which case it is slow
+to find the measurements associated with a specific object (a full
+table scan is required).  After the table is sorted, the
+\ippdbcolumn{Measure} rows for a given object are grouped together.
+In this case, the fields
+\ippdbtable{Average}.\ippdbcolumn{measureOffset} and
+\ippdbcolumn{Average}.\ippdbcolumn{Nmeasure} define an index for the
+code to jump to the list of measurements for a single object.  The
+field \ippdbtable{Measure}.\ippdbcolumn{imageID} defines the link from
+the measurement to the image which supplied the measurement.
 
 \note{some discussion of the db construction, addstar, dvomerge, etc?}
 
 For the warp images, we also measure the weak lensing KSB parameters
-related to the shear and smear tensors.  These measurements are stored
-in the \code{Lensing} table, along with the radial aperture fluxes for
-radii numbers 5, 6, \& 7 (XX, XX, XX arcsec).  This table contains one
-row for every warp row.  Similarly to the \code{Measure} table, the fields
-\code{objID}, \code{catID}, and \code{averef} define links from the
-\code{Lensing} table to the \code{Average} table.  In a similar
-fashion, the fields \code{Average.lensingOffset} and
-\code{Average.Nlensing} are the index into the sorted \code{Lensing}
-table entries.  \note{discuss failure of the Lensing to Measure
-  indexing}
-
-The values stored in the \code{Lensing} table are used to calculate
-average values for each of these types of measurements in each
-filter.  The \code{Lensobj} table stores the averaged KSB and radial
-aperture photometry for each of the 5 filters \grizy.  This table
-contains one entry per object per filter.  The table is not generally
-stored unsorted as it is calculated after the full database is
-populated.  The link from \code{Average} to \code{Lensobj} is defined
-by the fields \code{Average.offsetLensobj} and
-\code{Average.Nlensobj}.  Each \code{Lensobj} row also includes the
-photcode (filter) for which the average lensing (and radial aperture)
-properties have been calculated. 
-
-The \code{Galphot} table stores the results of the forced galaxy
+related to the shear and smear tensors \citep{1995ApJ...449..460K}.
+These measurements are stored in the \ippdbcolumn{Lensing} table,
+along with the radial aperture fluxes for radii numbers 5, 6, \& 7
+(respectively 3.0, 4.63, and 7.43 arcsec).  This table contains one
+row for every warp row. \note{warp row hasn't been defined anywhere.}
+Similarly to the \ippdbtable{Measure} table, the fields
+\ippdbcolumn{objID}, \ippdbcolumn{catID}, and \ippdbcolumn{averef}
+define links from the \ippdbtable{Lensing} table to the
+\ippdbtable{Average} table.  In a similar fashion, the fields
+\ippdbtable{Average}.\ippdbcolumn{lensingOffset} and
+\ippdbtable{Average}.\ippdbcolumn{Nlensing} are the index into the
+sorted \ippdbtable{Lensing} table entries.  \note{discuss failure of
+  the Lensing to Measure indexing}
+
+The values stored in the \ippdbtable{Lensing} table are used to
+calculate average values for each of these types of measurements in
+each filter.  The \ippdbtable{Lensobj} table stores the averaged KSB
+and radial aperture photometry for each of the 5 filters \grizy.  This
+table contains one entry per object per filter.  The table is not
+generally stored unsorted as it is calculated after the full database
+is populated.  The link from \ippdbtable{Average} to
+\ippdbtable{Lensobj} is defined by the fields
+\ippdbtable{Average}.\ippdbcolumn{offsetLensobj} and
+\ippdbtable{Average}.\ippdbcolumn{Nlensobj}.  Each
+\ippdbtable{Lensobj} row also includes the \ippdbcolumn{photcode} for which
+the average lensing (and radial aperture) properties have been
+calculated.
+
+The \ippdbtable{Galphot} table stores the results of the forced galaxy
 fitting analysis for each object that has been measured.  This table
-contains one row per filter and model type (Sersic, Exponential,
+contains one row per filter and model type (Sersic, Exponential, or
 DeVaucouleur) if forced galaxy models have been calculate for the
 object.  \note{need to expand on this somewhat}
 
-The \code{Starpar} table carries measurements provide by Greg Green \&
-Eddie Schlafly from their analysis of the SED of objects in the PS1
-$3\pi$ data, using the \note{PV1?} version of the analysis (Green et
-al REF).  In this work, the goal was a 3D model of the dust in the
-Galaxy based on Pan-STARRS (\note{and WISE \& 2MASS?}) photometry.  As
+The \ippdbtable{Starpar} table carries measurements provide by Greg
+Green \& Eddie Schlafly from their analysis of the SED of objects in
+the PS1 $3\pi$ data, using the \note{PV1?} version of the analysis
+\citep{2015ApJ...810...25G}.  In this work, the goal was a 3D model of
+the dust in the Galaxy based on Pan-STARRS and 2MASS photometry.  As
 part of this analysis, the authors fit the SEDs of all \note{stellar?}
 sources with stellar models including free parameters of extinction,
@@ -505,197 +608,652 @@
 these photometric distance modulus measurements are not extremely
 precise (see below), they provide a constraint on the distance is used
-in our analysis of the astrometry (see Section~\ref{sec:astrometry}).
-
-\subsection{Other Tables} 
+in our analysis of the astrometry \citep[][see]{magnier2017a}.
+
+\subsubsection{Other Tables} 
 
 Data from GPC1 (and other cameras processed by the IPP) are loaded
-into DVO in units \code{smf} files generated by the Camera calibration
-stage.  As described above, these files contain all measurements from
-a complete exposure, with measurements from each chip grouped into
-separate FITS tables.  When these measurements are loaded into the
-\code{Measure} and similar tables, a subset of the information from
-the chip header is used to populated a row in the DVO \code{Image}
-table.  This table contains one row for each chip known to DVO, with
-information such as the filter (\code{photcode}), the exposure time,
-the airmass, the astrometric calibration terms, the photometric
-zero point, etc.  For GPC1 and other mosaic cameras, an additional row
-is defined to carry the projection and camera distortion elements of
-the astrometry model.  As chips are loaded into this table, they are
-assigned an internal ID (a running sequence in the table).  Images may
-also be assigned an external ID: in the case of the GPC1 images, this
-ID is defined by the processing mysql database and is guaranteed to be
-unique within the processing system. 
+into DVO in units \code{smf} files generated by the \ippstage{camera}
+calibration stage (see section \ref{subsec: camera} below).  As
+described above, these files contain all measurements from a complete
+exposure, with measurements from each chip grouped into separate FITS
+tables.  When these measurements are loaded into the
+\ippdbtable{Measure} and similar tables, a subset of the information
+from the chip header is used to populated a row in the DVO
+\ippdbtable{Image} table.  This table contains one row for each chip
+known to DVO, with information such as the filter
+(\ippdbcolumn{photcode}), the exposure time, the airmass, the
+astrometric calibration terms, the photometric zero point, etc.  For
+GPC1 and other mosaic cameras, an additional row is defined to carry
+the projection and camera distortion elements of the astrometry model.
+As chips are loaded into this table, they are assigned an internal ID
+(a running sequence in the table).  Images may also be assigned an
+external ID: in the case of the GPC1 images, this ID is defined by the
+processing mysql database and is guaranteed to be unique within the
+processing system.
 
 Other tables are used to track information used by the calibration
 system.  This includes the complete set of flat-field corrections
-determined by the photometry calibration analysis (see
-Section~\ref{sec:relphot}) and the astrometric flat-field corrections
-determined by the astrometry calibration analysis (see Section~\ref{sec:relastro})
+determined by the photometry calibration analysis and the astrometric
+flat-field corrections determined by the astrometry calibration
+analysis \citep[][see]{magnier2017a}.
+
+\subsection{Datastore repositories}
+\label{subsec: datastore}
+
+Transferring data between the IPP and other parts of the Pan-STARRS
+system is generally accomplished via a ``datastore'', an http service
+that exposes data in a common form.  One of the main datastores used
+by the IPP is the one located at the summit.  This datastore exposes,
+a list of the exposures obtained since the start of the PS1
+operations.  Requests to this server may restrict to the latest by
+time.  Each row in the listing includes basic information about the
+exposure: an exposure identifier (e.g., o5432g0123o;
+see~\ref{GPC1.names} for details), the date and time of the exposure,
+the telescope commanded pointing, the filter and exposure time, and
+the observation comment for that exposure.  The row also provides a
+link to a listing of the chips associated with that exposure.  This
+listing includes a link to the individual chip FITS files as well as
+an md5 checksum.  Systems which are allowed access may download chip
+FITS files via http requests to the provided links.
+
+The IPP also uses datastores to provide access to its own data
+products.  The detections identified in the \ippstage{diff} stage
+images are organized by the \ippstage{publish} stage, which writes
+output files containing those detections to a datastore that is
+monitored by the Moving Object Processing System
+\citep[][MOPS]{2013PASP..125..357D}, which analyses the detections to
+identify asteroids.  Separate datastores are also used by the
+\ippstage{distribution} stage to provide access to data products to
+the Pan-STARRS Science Consortium members.  
+
+\subsection{Pantasks \& Parallel Processing}
+\label{subsec: pantasks}
+
+The actual processing of data is managed by the \ippprog{pantasks}
+program.  This program launches a series of connections to other
+computers in the IPP compute cluster, and launches instances of the
+\ippprog{pclient} program that wait for jobs to be submitted.  These
+jobs are generated from \ippmisc{ippTasks} command scripts that each
+contain one set of tasks, written in the opihi language shared with
+DVO.  These task scripts are generally sorted by the IPP stage.  The
+tasks listed in these scripts are mainly written in the form of a pair
+of \ippmisc{load} and \ippmisc{run} tasks.
+
+The \ippmisc{load} task for a particular stage queries the processing
+database via an appropriate \ippmisc{ippTool} (see section \ref{subsec:
+  ipptools} below) for a list of jobs that are waiting to be run.
+This task is executed on the host running the \ippprog{pantasks}
+server, and only one of each type of \ippmisc{load} task is permitted to
+run simultaneously, preventing race conditions.  To manage the
+processing, the primary database tables for each stage contain a
+\ippdbcolumn{label} field.  This field groups data into categories,
+and the appropriate list of \ippdbcolumn{label}s are supplied to the
+\ippprog{pantasks} program to be included in the queries to ensure
+that only jobs for the active \ippdbcolumn{label}s are executed.  The
+results from the database query are stored in \ippmisc{books} within the
+\ippprog{pantasks} environment, with each resulting result row saved
+to a separate entry within the \ippmisc{book}.  These \ippmisc{books} are
+similar to an associative array indexing the entries (\ippmisc{pages} to
+continue the analogy) to be accessed via a particular key.  Keys for
+most stages are a combination of the stage id and an identifier for
+the individual component for the job that will be executed.  The rest
+of the column results from the query are stored to the \ippmisc{page},
+with each field stored with the database column name.  An additional
+field is added, the \ippdbcolumn{pantasksState}, which is set to
+\ippmisc{INIT}, indicating that this \ippmisc{page} is a new addition to the
+\ippmisc{book}.  Once all results from the database query have been
+saved, the \ippmisc{load} task terminates, and waits to be rerun after
+some timeout (on the order of a minute).  Once all new \ippmisc{pages}
+have been added, the task then scans the \ippmisc{book} for any pages
+with \ippdbcolumn{pantasksState} set to \ippmisc{DONE}, and removes them
+from the book, as these represent jobs that have finished.
+
+Parallel to the \ippmisc{load} task, the \ippmisc{run} tasks are executed
+by the \ippprog{pclient} code on one of the client nodes.  These
+clients can access the \ippmisc{books} from \ippprog{pantasks}, and scan
+the \ippmisc{book} to determine if it contains \ippmisc{pages}, and if one
+of those pages contains an entry with the \ippdbcolumn{pantasksState}
+set to \ippmisc{INIT}.  If such a \ippmisc{page} is found, the task sets
+this field to \ippmisc{RUN} (ensuring no other \ippmisc{run} task will
+attempt to process that page as well), and uses the information stored
+on the page to construct a valid command line for the appropriate IPP
+script.  The script (described below) command line is then executed,
+and upon completion, pass the exit code back to \ippprog{pantasks},
+allowing for an accounting of success and failures.  The page is then
+set to \ippmisc{DONE}, indicating the job has completed, and the
+\ippmisc{page} can be removed from the \ippmisc{book} by the next call of
+the \ippmisc{load} task.  As the \ippmisc{run} tasks will involve processing
+that may not complete in a known period of time, there is a timeout
+limit set for the task, after which the \ippprog{pantasks} process can
+terminate the task.  This prevents unproductive jobs from building up.
+
+In addition to these tasks, most stages have \ippmisc{revert} tasks
+paired with the \ippmisc{run} task.  These tasks issue an appropriate
+\ippmisc{ippTool} command to remove entries from the stage's secondary
+database table that have the \ippdbcolumn{fault} field set to a value
+other than zero.  This step allows these failures to be cleared from
+the system, and schedule those jobs again for a retry.  Again, this
+query is limited by the set of active \ippdbcolumn{label}s, ensuring
+that only the data currently being processed is reverted.  Similarly,
+some stages have \ippmisc{advance} tasks that update the primary table to
+indicate that all of its components are complete.  For many of the
+early stages of the pipeline (the \ippstage{chip} through
+\ippstage{warp} stages), this \ippmisc{advance} task also adds an entry
+into the database table for the next stage of processing for the
+exposure being considered.  This step allows the data to process
+automatically from stage to stage without intervention.
+
+\subsubsection{Stage automation}
+\label{subsec: automation}
+\note{I'm not convinced this is the right place for it, but it felt like a natural extension of the ``advance''}.
+
+Beyond the warp stage, there is no longer a single ``next'' stage into
+which data can be queued.  Because of this, more robust methods are
+used to advance the data.  For processing data that is actively being
+observed at the summit, this is handled by a set of ``nightly
+science'' tasks and an associated \ippmisc{ippScript}.  The goal of
+these tasks is to ensure that exposures are correctly paired into sets
+of \ippstage{warp}-\ippstage{warp} difference images, and that nightly
+stacks are generated for MD fields.
+
+Queuing the diffs is done by first examining the set of all exposures
+that have been taken at the summit on the current night of observing,
+and querying information from each stage up through \ippstage{warp}
+stage.  These exposures are grouped by \ippdbcolumn{filter} and
+\ippdbcolumn{object}, which is a unique identifier for each telescope
+pointing on the sky.  Exposures in each group are then sorted by
+increasing observation date (\ippdbcolumn{dateobs}), and the
+information from each stage is used to ensure that the observations
+have completed all stages through \ippstage{warp} successfully.  If
+this is not the case, the script ignores that group until a future
+scan, to allow the data to finish processing.  If the lack of
+completion is due to a failure at a stage prior to \ippstage{warp},
+then that exposure is dropped from the group to allow the remainder to
+be considered for \ippstage{diff}s.  The remaining exposures are then
+paired sequentially, with the final exposure ignored in the case of an
+odd number of exposures (caused by either a failure in the processing
+of an exposure or due to the telescope closing for weather prior to
+completing all scheduled observations).  Once the observing process
+completes (signalled to the nightly science task and script by the
+presence in the database of entries for dark exposures that are taken
+each morning after the telescope closes), and the script has generated
+all \ippstage{diff} pairs that can be made with the above rule, a
+second pass is performed, this time with the exposures in each group
+sorted by decreasing observation date.  This change in ordering allows
+exposures that were excluded due to an odd number of exposures to be
+paired with the exposure closest in time (with the exposure that was
+previously first ignored).
+
+The nightly stacks are queued based on checking that a minimum number
+of complete \ippstage{warp} entries exist for each filter and field.
+For the nightly MD processing, this minimum number was set to 8
+exposures, as this is the number of exposures taken for each field.
+Once this number was reached, no more exposures are expected, so
+\ippstage{stack} database entries can be queued with the
+\ippstage{warp} entries.  Again, failures and weather can reduce the
+number of usable exposures, so the task generates \ippstage{stack}
+entries using whatever exposures are available if one has not yet been
+constructed by the time the morning dark exposures are registered into
+the database.
+
+Automating the nightly processing is important, as it ensures that
+data is processed as soon as it is downloaded from the summit,
+reducing the lag between an observation and the reduced data.  The
+other processing task that requires automation is the reprocessing of
+the entire $3\Pi$ survey, as the size of the dataset make it
+challenging to do manually.  To manage this, the ``large area
+processing'' (LAP) task and script are used.  The first stage of this
+processing is generating an entry in the \ippdbtable{lapSequence}
+table defining a new reprocessing.  After this, individual
+\ippdbtable{lapRun} entries can be queued that define a
+\ippdbcolumn{filter} and a \ippdbcolumn{projection\_cell} to be
+considered.  A \ippdbcolumn{projection\_cell} is a unit of sky defined
+to be a square four degrees on each side which has a single tangent
+plane projection \citep[][see]{watersc2017}.  Once this entry is
+defined, is is populated with exposures (stored in the
+\ippdbtable{lapExp} table in the database), with any exposure located
+within 5 degrees of the center of the projection cell included.  This
+radius ensures that any exposure that overlaps the projection cell
+will be included.  Once the exposures have been added, the other
+exposures within the same sequence are checked to see if a
+\ippstage{chip} stage entry has been generated, and if so, the
+\ippdbcolumn{chip\_id} for that entry is saved into the
+\ippdbtable{lapExp} as well.  This linkage ensures that each exposure
+is only processed once.  If no entry is found, a new \ippstage{chip}
+entry is queued for processing.  The task periodically checks the
+status of the exposures in each \ippdbtable{lapRun} entry, and if they
+have all completed the \ippstage{warp} stage, then a \ippstage{stack}
+is queued for each skycell contained within the
+\ippdbcolumn{projection\_cell}.
+
+\subsection{ippTools and ippScripts}
+\label{subsec: ipptools}
+
+As shown above, the \ippprog{pantasks} tasks rely on \ippmisc{ippTools}
+calls for database queries.  Each stage has an appropriate
+\ippmisc{ippTool}, allowing the database interaction to be governed by a
+fixed set of inserts and queries.  Isolating the database interaction
+in this way adds a layer of validation before queries are executed,
+and ensures that all database modifications are handled in a uniform
+fashion.  
+
+In addition to simple queries and updates to entries already in the
+database, the various \ippmisc{ippTools} programs can also be used to
+define new processing runs for a stage.  Again, using pre-defined
+queries wrapped in a program allow the options to be parsed to ensure
+that any new processing run definitions are appropriately restricted
+in scope, reducing the chance that mistakes will fill the database
+with many unwanted jobs.
+
+Connecting the \ippprog{pantasks} parallel processing environment to
+the actual IPP analysis programs are a series of \ippmisc{ippScripts}
+written in Perl.  These scripts are what are actually executed by
+\ippprog{pantasks}, with command line options provided based on the
+database query performed there by the \ippmisc{load} task.  These
+options are combined with configuration information stored in
+\ippmisc{ippconfig} recipe files.  
+
+The appropriate recipe is selected from the configuration information,
+based on the source camera of the data to be processed, and optionally
+modified by the \ippdbcolumn{reduction} field in the database.  These
+optional \ippdbcolumn{reduction} entries provide a way to group a
+non-standard set of processing options together across multiple
+stages, by selecting a recipe that is not the default.
+
+With the set of configuration options and database entries for the
+data to be processed, the \ippmisc{ippScript} checks the input files
+that will be used, and confirms that a valid copy of each is available
+from the \ippprog{Nebulous} system.  For stages that have a large
+number of inputs (such as the \ippstage{stack} stage, which requires
+images, masks, variance maps, and detection catalogs from each of the
+potentially large number of \ippstage{warp} stage inputs), the input
+files are organized into temporary input list files, formatted in an
+appropriate way for the analysis program that will process them.
+
+The script also sets up an output logfile for this processing run,
+ensuring that any status information from either the script itself or
+the underlying analysis is stored on disk.  The majority of this
+information is identical between calls to the script, but for rare
+failures of the analysis programs, retaining this information allows
+for such problems to be diagnosed and repaired.  
+
+The command line for the main analysis program is constructed based on
+the database values, the recipe options, and the input file names.
+The analysis program is then executed, and any failure reported back
+to the parent \ippprog{pantasks} process.  In the standard case of the
+analysis completing successfully, the script checks that the expected
+output products were generated, preventing hidden I/O errors from
+being a problem with subsequent processing of those output products.
+
+One output product that must exist is the \code{stats} file, which is
+generated by the analysis program and contains statistics about the
+processing, including such things as the image background level, the
+fraction of masked pixels, and the version numbers of the analysis
+program.  This stats file is then parsed by the
+\ippprog{ppStatsFromMetadata} program, which uses the information
+within to generate command options for the \ippmisc{ippTool} program
+to ensure that these statistics are included in the database row that
+is created in the secondary database table for the individual
+component processed.
+
+\subsection{psLib and psModules}
+\label{subsec: pslib}
+
+Underlying all of the analysis programs are the \ippmisc{psLib} and
+\ippmisc{psModules} C libraries.  The more fundamental \ippmisc{psLib}
+library defines the internal data structures that are used (arrays of
+arbitrary type, vectors, images, and hash tables among others), manage
+data access (particularly for FITS images and tables), and organize
+string and error handling in a uniform fashion.  This library also
+contains fundamental math operations, covering vector statistics,
+matrix operations, and function minimization.  Common image operations
+such as binning, interpolation, and convolution are also provided, as
+well as the methods to to write JPEG versions of the data for
+visualizations.  Finally, general coordinate transformations are
+provided between planes and projections of spheres.
+
+The functions provided by \ippmisc{psModules} have more focused scopes
+that are nevertheless still shared between multiple programs.  The
+isolation of source objects is included, providing the organization of
+detections that is used in the \ippprog{psphot} photometry analysis
+\citep{magnier2017c}.  The PSF matching required for \ippstage{stack}
+and \ippstage{diff} stage image combinations is as well.  The
+unification of configuration options between config files on disk and
+the options specified on the command line is handled by
+\ippmisc{psModules} functions, as is the contruction of data
+structures in memory to represent the astronomical camera based on the
+header information in the input file.  The functions to generate and
+apply the detrend corrections to the data are also provided by this
+library.
+
+\note{This likely needs cleaning up and more information.}
 
 \section{IPP Data Processing Stages}
-
-\subsection{Download from Summit}
-
-As exposures are taken by the PS1 telescope \& camera system, the 60
-OTA CCDs are read out by the camera software system and each chip is
-written to disk on computers at the summit in the PS1 facility.  The
-chip images are written to a collection of machines in the PS1
-facility called the `pixel servers'.  After the images are written to
-disk, a summary listing of the information about the exposure and the
-chip images are written to an http server system called the
-`datastore'.  The datastore exposes, via http, a list of the exposures
-obtained since the start of the PS1 operations.  Requests to this
-server may restrict to the latest by time.  Each row in the listing
-includes basic information about the exposure: an exposure identifier
-(e.g., o5432g0123o; see~\ref{GPC1.names} for details), the date and
-time of the exposure, \note{etc}.  The row also provides a link to a
-listing of the chips associated with that exposure.  This listing
-includes a link to the individual chip FITS files as well as an md5
-CHECKSUM.  Systems which are allowed access may download chip FITS
-files via http requests to the provided links.
-
-During night-time operations, while the telescope is observing the sky
-and the camera subsystem is saving images to the pixel servers and
-adding their information to the datastore list, the IPP subsystem
-called `summitcopy' monitors the datastore in order to discover new
-exposures ready for download.  Once a new exposure has been listed on
-the datastore, summitcopy adds an entry of the exposure to a table in
-the processing database (summitExp).  This tells the summitcopy to
-look for the list of chips, which are then added to another table
-(summitImfile).  The summitcopy system then attempts to download the
-chips from the summit pixel servers with an http request.  As the chip
-files are downloaded, their md5 checksum values are calculated and
-compared with the value reported by the camera subsystem / datastore.
-Download failures are rare and marked as a non-zero fault, allowing for a
-manual recovery, rather than automatically rejecting the failed
-chips.  
+\label{sec: stages}
+
+
+\subsection{Summit copy}
+\label{subsec: summit copy}
+
+As exposures are taken by the PS1 telescope \& GPC1 camera system, the
+data from the 60 OTA devices are read out by the camera software
+wsystem and written to disk on a collection of computers at the summit
+in the PS1 facility called ``pixel servers.'' After the images are
+written to disk, a summary listing of the information about the
+exposure and the chip images are added to the summit datastore.
+
+During night-time operations, while the summit datastore is being
+populated, the IPP subsystem called \ippstage{summitcopy} monitors the
+datastores listed in the \ippdbtable{pzDatastore} table of the
+database in order to discover new exposures ready for download.  Once
+a new exposure has been listed on the datastore, \ippstage{summitcopy}
+adds an entry of the exposure to a table in the processing database
+(\ippdbtable{summitExp}), indexed by an identifier that simply
+increments the number of exposures announced by the summit, the
+\ippdbcolumn{summit\_id}.  This tells the \ippstage{summitcopy} system
+to look for the list of chips, which are then added to another table
+(\ippdbtable{summitImfile}).  This system then attempts to download
+the chips (registering the results of those operations into the
+\ippdbtable{pzDownloadExp} and \ippdbtable{pzDownloadImfile} tables)
+from the summit pixel servers via an http request.  As the image files
+are downloaded, their MD5 checksum values are calculated and compared
+with the value reported by the summit datastore.  Download failures
+are rare and marked with a non-zero \ippdbcolumn{fault}, allowing for
+a manual recovery, rather than automatically rejecting the failed
+chips.  Once all the components of the exposure have been downloaded,
+they are further entered into the \ippdbtable{newExp} and
+\ippdbtable{newImfile} tables, which index the exposures by
+\ippdbcolumn{exp\_id}.  This switch in index indicates that the
+exposure has successfully been copied from the summit to the IPP
+cluster, and that further processing is no longer dependent on outside
+resources.
 
 \subsection{Image Registration}
-
-Once chips for an exposure have all been downloaded, the exposure is
-ready to be registered.  In this context, `registration' refers to the
-process of adding them to the database listing of known, raw exposures
-(not to be confused with 'registration' in the sense of pixel
-re-alignment).  The result of the registration analysis is an entry
-for each exposure in the rawExp table, and one for each chip in the
-rawImfile table.  These tables are critical for downstream processing
-to identify what exposures are available for processing in any other
-stage.  In the registration stage, a large amount of descriptive
-metadata for each chip is added to the rawImfile table, some of which
-is extracted from the chip FITS file headers (e.g., RA, DEC, FILTER)
-and some of which is determined by a quick analysis of the pixels
-(e.g., mean pixel values, standard deviation).  The chip-level
-information is merged into a set of exposure-level metadata and added
-to the rawExp table entry.  The exposure-level metadata may be the
-same as any one of the chip, in a case where the values are duplicated
-across the chip files (e.g., the name of the telescope or the date \&
-time of the exposure), or it may be a calculation based on the values
-from each chip (e.g., average of the average pixel values).
+\label{subsec: registration}
+
+Once the chips for an exposure have all been downloaded, the exposure
+is ready to be registered.  In this context, `registration' refers to
+the process of adding them to the database listing of known, raw
+exposures (not to be confused with 'registration' in the sense of
+pixel re-alignment).  The result of the registration analysis is an
+entry for each exposure in the \ippdbtable{rawExp} table, and one for
+each chip in the \ippdbtable{rawImfile} table.  These tables are
+critical for downstream processing to identify what exposures are
+available for processing in any other stage.  At the registration
+stage, a large amount of descriptive metadata for each chip is added
+to the \ippdbtable{rawImfile} table, the majority of which is
+extracted from the chip FITS file headers (e.g., RA, DEC, FILTER) and
+some of which is determined by a quick analysis of the pixels (e.g.,
+mean pixel values, standard deviation).  The chip-level information is
+merged into a set of exposure-level metadata and added to the
+\ippdbtable{rawExp} table entry.  The exposure-level metadata may be
+the same as any one of the chip, in a case where the values are
+duplicated across the chip files (e.g., the name of the telescope or
+the date \& time of the exposure), or it may be a calculation based on
+the values from each chip (e.g., average of the average pixel values).
 
 Unlike much of the rest of the IPP stage, the raw exposures may only
 have a single entry in the registration tables of the processing
-database tables (rawExp and rawImfile).
+database tables (\ippdbtable{rawExp} and \ippdbtable{rawImfile}).
+
+For GPC1, the image registration stage is also the stage at which the
+\ippprog{burntool} analysis is run.  This analysis is more completely
+described in \citet{waters2017}.  In brief, the \ippprog{burntool}
+program identifies bright sources on the image, and identifies
+persistence trails that result from the incomplete transfer of charge.
+As this charge can leak out in subsequent exposures, the burntool
+analysis is run sequentially on the exposures, based on the
+observation date and time listed in the headers, with the results
+stored in an text table.  As a result of the sequential nature of this
+analysis, the registration of exposures is blocked until the
+\ippprog{burntool} has been run on the previous exposures.
+
+Once the registration process has finished, new science exposures that
+have an \ippdbcolumn{obs\_mode} value that indicates they are part of
+a particular science survey are automatically launched into the
+science analysis by defining entries for the \ippstage{chip}
+processing stage, as described above.  This analysis can be relaunched
+multiple times, such as for the large scale PV3 reprocessing.
+However, this automatic process ensures the shortest time between
+observation and analysis, which is particularly important in the
+search for transient sources.
 
 \subsection{Chip Processing}
-
-The science analysis of an exposure begins with the processing of the
-individual chips, the Chip Processing stage.  This analysis step has
-two main goals: the removal of the instrumental signature from the
-pixel values (detrending) and the detection of the sources in the
-objects.  In the Chip stage, the individual chips are processed
-independently in parallel within the data processing cluster.  Within
-the processing computer cluster, most of the data storage resources
-are in the form of computers with large raids as well as substantial
-processing capability.  The processing system attempts to locate one
-copy of specific raw chips on pre-defined computers for each chip.
-The processing system is aware of this data localization and attempts
-to target the processing of a particular chip to the machine on which
-the data for that chip is stored.  The output products are then
-primarily saved back to the same machine.  This `targetted' processing
-was an early design choice to minimize the system wide network load
-during processing.  In practice, as computer disks filled up at
-different rates, the data has not been localized to a very high
-degree.  The targeted processing has probably reduced the network load
+\label{subsec: chip}
+
+The science analysis of an exposure begins with the \ippstage{chip}
+stage, which operates on the individual OTA image files.  This
+analysis step has two main goals: detrending the image to remove the
+instrumental signature from the pixel values, and the detection of
+astronomical sources in the objects.  Based on the entry the
+\ippdbtable{chipRun} primary table defining the processing details
+(with the \ippdbcolumn{state} column indicating it needs processing),
+and the associated information listed in the \ippdbtable{rawImfile},
+jobs can be spawned for each component OTA.  The \ippprog{pantasks}
+environment managing the jobs attempts to target the processing host
+to one that should host the OTA, to reduce number of operations done
+on remote data.  In practice, this targeted processing has not had as
+large of an effect as was originally intended, as the data volume has
+reduced the ability of any one node to reliably contain a particular
+OTA.  The targeted processing has probably reduced the network load
 somewhat but it has not been as critical of a requirement as
 originally expected.
 
-The Chip processing stage consists of: reading the raw image into
-memory, appyling the detrending steps \citep[see][]{waters2017},
-stiching the individual OTA cells into a single chip image, detection
-and characterization of the sources in the image
-\citep[see][]{magnier2017b}, and output of the various data products.
-These include the detrended chip image, variance image, and mask
-image, as well as the FITS catalog of detected sources.  The PSF model
-and background model are also saved, along with a processing log.  A
+%% In the \ippstage{chip} stage,
+%% the individual OTA image files are processed independently in parallel
+%% within the data processing cluster.  \note{move this to kihei
+%%   discussion?} Within the processing computer cluster, most of the
+%% data storage resources are in the form of computers with large raids
+%% as well as substantial processing capability.  The processing system
+%% attempts to locate one copy of specific raw registered data on
+%% pre-defined computers that have been set as storage targets for that
+%% OTA.  The processing system is aware of this data localization and
+%% attempts to target the processing for each OTA to the machine on which
+%% the data for that detector is stored.  The output products are then
+%% primarily saved back to the same machine.  This `targetted' processing
+%% was an early design choice to minimize the system wide network load
+%% during processing.  In practice, as computer disks filled up at
+%% different rates, the data has not been localized to a very high
+%% degree.  
+
+The actual image processing is performed by the \ippprog{ppImage}
+program.  This program reads the raw data into memory and applies the
+detrend corrections \citep[see][]{waters2017} to each cell in the OTA
+(which are stored as different extensions in the FITS file format),
+and then mosaics the cells into a single contiguous \ippstage{chip}
+stage image.  This step also creates in memory additional images to
+hold the mask data, which indicates which pixels may not be valid, and
+the variance image, constructed as the Poissonian noise on the number
+of electrons detected based on the original pixel value and the
+detector gain.  A background model is then fit across the image and
+subtracted to remove the expected contribution from the sky
+\citep[see][]{waters2017} for details.
+
+With the image calibration procedure finished, object identification
+and photometry can be performed.  Although this can be done using a
+stand alone program, \ippprog{psphot}, the underlying functions are
+contained in a library that allows \ippprog{ppImage} to directly do
+this analysis, removing the need to write out and re-read the image
+data.  The details of the detection and characterization of the
+sources in the image are provided in \citet{magnier2017b}.  
+
+The results of the image processing are then written to disk,
+including the science, mask, and variance images, the background model
+subtracted, the PSF model used in the photometry process, and a FITS
+catalog of detected sources.  Additional binned images of the full OTA
+are also saved, providing $16\times{}16$ and $256\times{}256$ pixel
+binning scales for quick visualization.  The processing log and a
 selection of summary metadata describing the processing results are
-saved and written to the processing database along with the completion
-status of the process.  Finally, binned chip images are generated (on
-two scales, binned by 16 and 256 pixels) for use in the visualization
-system of the processing monitor tool.
+also written to disk.  This metadata is used to populate a row in the
+\ippdbtable{chipProcessedImfile} table (linked to the
+\ippdbtable{chipRun} entry by a shared \ippdbcolumn{chip\_id} value)
+to indicate that the processing of this OTA is complete.
+
+As each OTA is processed independently of the others across a number
+of computers, the \ippprog{pantasks} managing the jobs periodically
+runs an \ippmisc{advance} task that checks that the number of rows in
+\ippdbtable{chipProcessedImfile} with \ippdbcolumn{fault} equal to
+zero matches the associated number of rows in \ippdbtable{rawImfile}.
+If this condition is met, than all processing for that exposure is
+finished, and the \ippdbcolumn{state} field is set to ``full''.  If
+the \ippdbtable{chipRun}.\ippdbcolumn{end\_stage} field is set to
+\ippstage{chip}, then no further action is taken.  However, this field
+is usually set to a subsequent stage (most often \ippstage{warp}),
+then an entry for this exposure is added to the \ippdbtable{camRun}
+table, and processing continues.
+
+%% The \ippstage{chip} processing stage consists of: reading the raw image into
+%% memory, applying the detrending steps \citep[see][]{waters2017},
+%% stiching the individual OTA cells into a single chip image, detection
+%% and characterization of the sources in the image
+%% \citep[see][]{magnier2017b}, and output of the various data products.
+%% These include the detrended chip image, variance image, and mask
+%% image, as well as the FITS catalog of detected sources.  The PSF model
+%% and background model are also saved, along with a processing log.  A
+%% selection of summary metadata describing the processing results are
+%% saved and written to the processing database along with the completion
+%% status of the process.  Finally, binned chip images are generated (on
+%% two scales, binned by 16 and 256 pixels) for use in the visualization
+%% system of the processing monitor tool. \note{describe elsewhere?}
+
+%% The database structure for the \stage{chip} stage mimics that of raw
+%% data, with a \ippdbtable{chipRun} characterizing the processing of a
+%% single exposure, mapping to a set of \ippdbtable{chipProcessedImfile}
+%% entries for each OTA via a common \ippdbcolumn{chip\_id}.  
 
 \subsection{Camera Calibration}
-
-After sources have been detected and measured for each of the chip,
-the next stage is to perform a basic calibration of the full exposure.
-This stage starts with the collection of FITS tables containing the
-instrumental measurements of the detected sources, primarily the
-positions ($x_{\rm ccd}, y_{\rm ccd}$) and the instrumental PSF
-magnitudes.  The data for all chips of an exposure are loaded by the
-analysis program.  The header information is used to determine the
-coordinates of the telescope boresite (RA, DEC, Position angle).
-These three coordinates are used, along with a model of the camera
-layout, to generate an initial guess for the astrometry of each chip.
-Reference star coordinates and magnitudes are loaded from a reference
-catalog for a region corresponding to the boundaries of the exposure,
-padded by a large fraction of the exposure diameter in case of a
-modest pointing error.  The guess astrometry is used to match the
-reference catalog to the observed stellar positions in the focal plane
-coordinate system.  Once an acceptable match is found, the astrometric
-calibration of the individual chips is performed, including a a fit to
-a single model for the distortion introduced by the camera optics.
-After the astrometic analysis is completed, the photometric
-calibration is determined using the final match to the reference
-catalog.  At this stage, pre-determined color terms may be included to
-convert the reference photometry to an appropriate photometric
-system.  For PS1, this is used to generate synthetic w-band photometry
-for areas where no PS1-based calibrated w-band photometry is
-available.  For more details, see \cite{magnier2017c}.
+\label{subsec: camera}
+
+After sources have been detected and measured for each of the chips,
+the next stage is to perform a basic calibration of the full exposure
+in the \ippstage{camera} stage.  This runs as a single job for the
+entire exposure, passing the collection of FITS table catalogs
+generated from each OTA in the \ippstage{chip} stage to the
+\ippprog{psastro} program.  Although the full catalog is loaded, the
+calibration primarily concerns the positions ($x_{\rm ccd}, y_{\rm
+  ccd}$) and the instrumental PSF magnitudes.  The header information
+in these catalogs is used to determine the coordinates of the
+telescope boresite (RA, DEC, position angle).  These three coordinates
+are used, along with a pre-determined model of the OTA layout within
+the camera, to generate an initial guess for the astrometry of each
+chip.  Reference star coordinates and magnitudes are loaded from a
+reference catalog for a region corresponding to the boundaries of the
+exposure, padded by a large fraction (25\%) of the exposure diameter
+to help guarantee a solution in the case of a modest pointing error.
+The guess astrometry is used to match the reference catalog to the
+observed stellar positions in the focal plane coordinate system.  Once
+an acceptable match is found, the astrometric calibration of the
+individual chips is performed, including a fit to a single model for
+the distortion introduced by the camera optics.  After the astrometic
+analysis is completed, the photometric calibration is determined using
+the final match to the reference catalog.  At this stage,
+pre-determined color terms may be included to convert the reference
+photometry to an appropriate photometric system.  For PS1, this is
+used to generate synthetic w-band photometry for areas where no
+PS1-based calibrated w-band photometry is available.  For more
+details, see \cite{magnier2017c}.  The result of these calibrations is
+stored as a single multi-extension FITS table containing the results
+from each OTA as a separate extension.
 
 In addition to the astrometric and photometric calibrations, the
-Camera stage also generates the dynamic masks for the images.  The dynamic
-masks include masking for optical ghosts, glints, saturated stars,
-diffraction spikes, and electronic crosstalk.  The mask images
-generated by the Chip stage are updated with these dynamic masks and a
-new set of files are saved for the downstream analysis stages.
-
-The Camera stage also merges the binned chip images
-(see~\ref{sec:chip}) into single jpeg images of the entire focal
-plane.  These jpeg images can then be displayed by the process
+\ippstage{camera} stage also generates the dynamic masks for the
+images.  These include masking for optical ghosts, glints, saturated
+stars, diffraction spikes, and electronic crosstalk.  The mask images
+generated by the \ippstage{chip} stage are updated with these dynamic
+masks and a new set of files are saved for the downstream analysis
+stages.  The \ippstage{camera} stage also merges the binned chip
+images (see~\ref{sec:chip}) into single jpeg images of the entire
+focal plane.  These jpeg images can then be displayed by the process
 monitoring system to visualize the data processing.
 
-\subsection{Warp}
-
-Once astrometric and photometric calibrations have been performed,
-images are geometrically transformed into a set of common pixel-grid
-images with simple projections from the sky.  These images, called
-skycells, can then be used in subsequent stacking and difference image
-analysis without concern about the astrometric transformation of an
-exposure.  This processing is called `warping'; the warp analysis
-stage is run on all exposures before they are processed further.  For
-details on the warping algorithm, see \cite{waters2017}.
-
-The output products from the Warp stage consist of the skycell images
-containing the signal, the variance, and the mask information.  These
-images have been shipped to STScI and \note{are available / will be
-  available} from the image extraction tools \note{in DR2}.
-
-\subsection{Stack}
-
-The skycell images generated by the Warp process are added together to
-make deeper, higher signal-to-noise images in the Stack stage.  The
-stacks also fill in coverage gaps between different exposures,
-resulting in an image of the sky with more uniform coverage than a
-single exposure.  See~\cite{waters2017} for details on the stack
-combination algorithm.
+Again, summary metadata is saved to disk as well, and the results
+listed therein are used to populate a row in the
+\ippdbtable{camProcessedExp} database table.  As the full exposure is
+processed all at once, this update also updates the associated
+\ippdbtable{camRun} entry, linked by the \ippdbcolumn{cam\_id}.  As
+with the \ippstage{chip} stage, the
+\ippdbtable{camRun}.\ippdbcolumn{end\_stage} is for a subsequent
+stage, an appropriate entry is added to the \ippdbtable{fakeRun}
+table.
+
+\subsection{Fake Analysis}
+\label{subsec: fake}
+
+The \ippstage{fake} stage was originally designed to do false source
+injection and recovery, in order to determine the detection efficiency
+of sources on the exposure.  However, early in the design of the IPP,
+this task was moved to the rest of the photometry analysis done at the
+\ippstage{chip} stage.  Removing the stage would require significant
+changes to the database schema.  As a result, this conveniently named
+stage generally does no actual data processing, and consists mainly of
+database operations to move the exposure on to the \ippstage{warp}
+stage.  The operations mimic the \ippstage{chip} stage, with
+individual jobs run for each OTA that update rows in the
+\ippdbtable{fakeProcessedImfile}, and an \ippmisc{advance} task that
+updates the \ippdbtable{fakeRun} table and promotes the exposure to
+the next stage by adding a row to the \ippdbtable{warpRun} table.
+
+\subsection{Image Warping}
+\label{subsec: warp}
+
+The \ippstage{warp} stage moves the data from a given exposure beyond
+away from being camera specific and towards a uniform sky oriented
+arrangement.  There are a number of ``tessellations'' defined and used
+by the IPP to define the extent and scaling of images on this uniform
+arrangement.  A tessellation can be defined for a limited region, such
+as M31 or other fields of particular interest that can be well
+described by a single tangent plane projection, or for larger regions
+which have multiple projection centers.  For the $3\Pi$ survey, the
+\ippmisc{RINGS.V3} tessellation was used that used projection centers
+spaced every four degrees in both RA and DEC, with $0\farcs{}25$
+pixels.  These projections are further broken down into ``skycells''
+that form a $10\times{}10$ grid within the projection, with an overlap
+region of 60" between adjacent skycells to ensure that objects are not
+split on all images. 
+
+These tessellations are stored in the DVO format, with
+\ippdbtable{SkyTable} entries defining the projection centers and
+image boundaries for all the skycells.  The first step of the
+\ippstage{warp} stage is determining which skycells overlap with the
+input exposure.  These overlaps are determined by the
+\ippprog{dvoImageOverlaps} program, which compares the astrometrically
+calibrated catalog from the \ippstage{camera} stage to the
+\ippdbtable{SkyTable} entries.  The output of this command is used to
+populate the \ippdbtable{warpSkyCellMap} table in the database, which
+contains a row for each skycell and OTA that overlap.  This results in
+more rows than there are OTAs, as each skycell can contain
+contributions from multiple OTAs.
+
+Once this mapping has been defined, jobs to construct each skycell are
+run, passing the \ippstage{camera} stage catalog and the
+\ippstage{chip} stage images (including the variance images and the
+updated masks) to the \ippprog{pswarp} program.  For details on the
+warping algorithm, see \cite{waters2017}.  The output of this program
+are the geometrically transformed images containing all input pixels
+warped to the common skycell pixel grid, which can subsequently be
+used for stacking and difference image analysis.  The image, mask, and
+variance generated at this stage will be available from the image
+extraction tools at the MAST archive at STScI as part of the DR2 data
+release.  A catalog is also generated containing the locations of
+sources from the input catalog that fall within area of the
+\ippstage{warp}.
+
+When the jobs have completed, an entry for the skycell is added to the
+\ippdbtable{warpSkyfile} database table, linked to the
+\ippdbtable{warpRun} entry by a common \ippdbcolumn{warp\_id}.  An
+\ippmisc{advance} task again checks that all potential skycells have
+been generated.  At this point, the direct promotion of exposures from
+one stage to the next stops, as the logic for matching exposures for
+combination is more complicated than simply adding a single entry (as
+discussed above).
+
+\subsection{Stack Combination}
+\label{subsec: stack}
+
+The skycell images generated by the \ippstage{warp} process are added
+together to make deeper, higher signal-to-noise images in the
+\ippstage{stack} stage.  These stacked images also fill in coverage
+gaps between different exposures, resulting in an image of the sky
+with more uniform coverage than a single exposure.
 
 In the IPP processing, stacks may be made with various options for the
@@ -703,79 +1261,126 @@
 filter for each Medium Deep field are combined into a set of stacks
 for that field.  These so-called `nightly stacks' are used by the
-transient survey projects to detect the faint supernovae, among other
-transient events.  For the PV3 $3\pi$ analysis, all filter images from
-the $3\pi$ survey observation were stacked together to generate a
-single set of images with $\sim 10 - 20\times$ the exposure of the
-individual survey exposures.  The signal, variance, and mask images
-resulting from these deep stacks are part of the DR1 release and are
-available from the image extraction tools.
+transient survey projects to detect faint supernovae, among other
+transient events.  For the PV3 $3\pi$ analysis, all images in each
+filter from the observations for this survey were stacked together to
+generate a single set of images with $\sim 10 - 20\times$ the exposure
+of the individual survey exposures.  
 
 For the PV3 processing of the Medium Deep fields, stacks have been
 generated for the nightly groups and for the full depth using all
-exposures (deep stacks).  In addition, a 'best seeing' set of stack
-have been produced \note{using image quality cuts to be described}.
-We have also generated out-of-season stacks for the Medium Deep
-fields, in which all image not from a particular observing season for
-a field are combined into a stack.  These later stacks are useful as
-deep templates when studying long-term transient events in the Medium
-Deep fields as they are not (or less) contaminated by the flux of the
-transients from a given season.
+exposures, producing ``deep stacks''.  In addition, a 'best seeing'
+set of stacks have been produced \note{using image quality cuts to be
+  described}.  We have also generated out-of-season stacks for the
+Medium Deep fields, in which all image not from a particular observing
+season for a field are combined into a stack.  These later stacks are
+useful as deep templates when studying long-term transient events in
+the Medium Deep fields as they are not (or less) contaminated by the
+flux of the transients from a given season.
+
+When a given set of \ippstage{stack} stage are defined, exposures with
+existing \ippstage{warp} entries that match the filter, position, and
+other criteria such as seeing are grouped by their skycell.  An entry
+is then added for each skycell in the \ippdbtable{stackRun} table,
+with the \ippdbcolumn{warp\_id} entries for the exposures added to the
+\ippdbtable{stackInputSkyfile} table, linked to the
+\ippdbtable{stackRun} entry by the \ippdbcolumn{stack\_id} field.
+This defines the mapping for which exposures contribute to the
+\ippstage{stack}.  This breaks exposures into single skycells, but as
+adjacent \ippstage{stack} skycells may contain inputs from different
+exposures, there is no simple way to group the processing at the
+\ippstage{stack} stage into exposures.
+
+The \ippstage{stack} jobs pass the information about the input images
+and catalogs to the \ippprog{ppStack} program, which performs the
+image combinations.  See~\cite{waters2017} for details on the stack
+combination algorithm.  In addition to the standard image, mask, and
+variance produced at other stage, additional images are constructed
+with information about the contributions to each pixel.  A number
+image contains the number of input exposures used for each pixel,
+along with an exposure time map, and a weighted exposure time map that
+scales the exposure time based on the relative variance of each input.
+These images for the $3\Pi$ analysis are currently available from the
+MAST image extraction tools at STSci.
+
+Upon completing the generation of these images, a row is added to the
+\ippdbtable{stackSumSkyfile} table with statistics about
+\ippstage{stack} processing.  As this completes all processing for the
+entry, no \ippmisc{advance} job is required.
 
 \subsection{Stack Photometry}
-
-The stack images are generated in the Stack stage of the IPP, but the
-source detection and extraction analysis of those images is deferred
-until a separate stage, the Stack Photometry stage.  This separation
-is maintained because the stack photometry analysis is performed on
-all 5 filter stack images at the same time.  By deferring the
-analysis, the processing system may decouple the generation of the
-pixels from the source detection.  This makes the sequencing of
+\label{subsec: staticsky}
+
+Although images are generated in the \ippstage{stack} stage of the
+IPP, the source detection and extraction analysis of those images is
+deferred to the \ippstage{staticsky} stage.  This separation is
+maintained because the photometry analysis of the \ippstage{stack}
+images is performed on all 5 filters simultaneously.  By deferring
+this analysis, the processing system may also decouple the generation
+of the pixels from the source detection.  This makes the sequencing of
 analysis somewhat easier and less subject to blocks due to a failure
-in the stacking analysis.
-
-The stack photometry algorithms are described in detail in
-\cite{magnier2017b}.  In short, sources are detected in all 5 filter
-images down to the $5\sigma$ significance.  The collection of detected
-sources is merged into a single master list.  If a source is detected
-in at least two bands, or only in $y$-band, then a PSF model is fitted
-to the pixels of the other bands in which the source was not detected.
-This forced photometry results in lower significance measurements of
-the flux at the positions of objects which are thought to be real
-sources, by virtue of triggering a detection in at least two bands.
-The relaxed limit for $y$-band is included to allow for searches of
-$y$-dropout objects: it is known that faint, high-redshift quasars may
-be detected in $y$-band only.  The casual user of the PV3 dataset
-should be wary of sources detected only in $y$-band as these are
-likely to have a higher false-positive rate than the other stack
-sources.
-
-The stack photometry output files consist of a set of FITS tables in a
-single file, with one file for each filter.  Within one of these
-files, the tables include: the measurements of sources based on the
-PSF model; aperture like parameters such as the Petrosian flux and
-radius; the convolved Galaxy model fits; the radial aperture
-measurements.  \note{is this list complete?}
+in the stacking analysis.  Similar to the \ippstage{stack} stage, an
+entry is created in the \ippdbtable{staticskyRun} table, linked to a
+series of rows in the \ippdbtable{staticskyInput} table by a common
+\ippdbcolumn{sky\_id}, each of which also contains the appropriate
+\ippdbcolumn{stack\_id} entries for the skycell under consideration.
+
+The input images are passed to the \ippprog{psphotStack} program,
+which does the analysis.  The stack photometry algorithms are
+described in detail in \cite{magnier2017b}.  In short, sources are
+detected in all 5 filter images down to the $5\sigma$ significance.
+The collection of detected sources is merged into a single master
+list.  If a source is detected in at least two bands, or only in
+\yps{} band, then a PSF model is fitted to the pixels of the other
+bands in which the source was not detected.  This forced photometry
+results in lower significance measurements of the flux at the
+positions of objects which are thought to be real sources, by virtue
+of triggering a detection in at least two bands.  The relaxed limit
+for \yps{} band is included to allow for searches of \yps{} dropout
+objects: it is known that faint, high-redshift quasars may be detected
+in \yps{} band only.  Sources detected only in \yps{} band are
+therefore more likely to have a higher false-positive rate than the
+other stack sources.
+
+The stack photometry output files consist of a set of FITS table
+catalogs, with one file for each filter.  Within these files, there
+are multiple table extensions that include: the measurements of
+sources based on the PSF model; aperture like parameters such as the
+Petrosian flux and radius; the convolved galaxy model fits; and the
+radial aperture measurements.  \note{is this list complete?}  Once the
+photometry is complete, a row is added to the
+\ippdbtable{staticskyResult} table with basic statistics from the
+analysis.
 
 The stack photometry output catalogs are re-calibrated for both
-photometry and astrometry in a process very similar to the Camera
-calibration stage.  In the case of the stack calibration, however,
-each skycell is processed independently.  The same reference catalog
-is used for the Camera and Stack calibration stages.
+photometry and astrometry in a process very similar to the
+\ippstage{camera} calibration stage.  In the case of this
+\ippstage{skycal} stage, each skycell is processed independently.
+Because of this independence, when queued for processing, the entries
+in the \ippdbtable{skycalRun} table contain the \ippdbcolumn{sky\_id}
+and \ippdbcolumn{stack\_id} entries of the parent data directly.  As
+in the \ippstage{camera} stage, the \ippprog{psastro} program reads in
+the stack photometry catalog, and produces a calibrated output.  A
+different processing recipe is supplied to \ippprog{psastro}, which
+controls for the different data.  The same reference catalog is used
+for the \ippstage{camera} and \ippstage{stack} calibration stages.
+Upon completion, the analysis statistics are written to the
+\ippdbtable{skycalResult} table. \note{Any difference in output formats?}
 
 \subsection{Forced Warp Photometry}
+\label{subsec: fullforce}
 
 Traditionally, projects which use multiple exposures to increase the
 depth and sensitivity of the observations have generated something
-equivalent to the stack images produced by the IPP analysis.  In
-theory, the photometry of the stack images produces the `best'
-photometry catalog, with best sensitivity and the best data quality at
-all magnitudes (c.f, CFHT Legacy survey, COSMOS, etc).  In practice,
-the stack images have some significant limitations due to the
-difficulty of modelling the PSF variations.  This difficulty is
-particularly severe for the Pan-STARRS $3\pi$ survey stacks due to the
-combination of the substantial mask fraction of the individual
-exposures, the large instrinsic image quality variations within a
-single exposure, and the wide range of image quality conditions under
-which data were obtained and used to generate the $3\pi$ PV3 stacks.
+equivalent to the stack images produced by the IPP analysis (c.f, CFHT
+Legacy survey, COSMOS, etc).  In theory, the photometry of the stack
+images produces the `best' photometry catalog, with best sensitivity
+and the best data quality at all magnitudes.  In practice, the stack
+images have some significant limitations due to the difficulty of
+modelling the PSF variations.  This difficulty is particularly severe
+for the Pan-STARRS $3\pi$ survey stacks due to the combination of the
+substantial mask fraction of the individual exposures, the large
+instrinsic image quality variations within a single exposure, and the
+wide range of image quality conditions under which data were obtained
+and used to generate the $3\pi$ PV3 stacks.
 
 For any specific stack, the point spread function at a particular
@@ -790,5 +1395,5 @@
 location.  In the end, the stack images have a effective point spread
 function which is not just variable, but changing significantly on
-small scales in a highly textured fashion.  
+small scales in a highly textured fashion.  \note{duplicates previous paragraph?}
 
 Any measurement which relies on a good knowledge of the PSF at the
@@ -796,5 +1401,5 @@
 present in the stack, or the measurement will be somewhat degraded.
 The highly textured PSF variations make this a very challenging
-problem: not would such a PSF model require an unusually fine-grained
+problem: not only would such a PSF model require an unusually fine-grained
 PSF model, there would likely not be enough PSF stars in an given
 stack to determine the model at the resolution required.  The IPP
@@ -805,5 +1410,5 @@
 changes apparent in the stack images.
 
-Thus PSF photometry as well as convolved Galaxy models in the stack
+Thus PSF photometry as well as convolved galaxy models in the stack
 are degraded by the PSF variations.  Aperture-like measurements are in
 general not as affected by the PSF variations, as long as the aperture
@@ -815,8 +1420,8 @@
 
 The PV3 $3\pi$ analysis solves this problem by using the sources
-detected in the Stack images and performing forced photometry on the
+detected in the stack images and performing forced photometry on the
 individual warp images used to generate the stack.  This analysis is
 performed on all warps for a single filter as a single job, though
-this is more of a bookkeepping aid as it is not necessary for the
+this is more of a bookkeeping aid as it is not necessary for the
 analysis of the different warps to know about the results of the other
 warps.
@@ -836,5 +1441,5 @@
 measurement as the signal-to-noise increases by $\sqrt{N}$.  
 
-\subsection{Forced Galaxy Models}
+\subsubsection{Forced Galaxy Models}
 
 The convolved galaxy models are also re-measured on the warp images by
@@ -855,5 +1460,5 @@
 (and thus the best galaxy magnitude value).
 
-For each warp image, the Stack value for the major and minor axis are
+For each warp image, the stack value for the major and minor axis are
 used as the center of a $7\times 7$ grid search of the major and minor
 axis parameter values.  The grid spacing is defined as a function of
@@ -890,5 +1495,5 @@
 
 \subsection{Difference Images}
-
+\label{subsec: diff}
 Two of the primary science drivers for the Pan-STARRS system are the
 search hazardous asteroids and the search for Type Ia supernovae to
@@ -904,6 +1509,6 @@
 images are matched.  \note{discuss Alard-Lupton}. 
 
-In the Difference Image stage, the IPP generates diffferece images for
-specified pairs of images.  It is possible for the difference image to
+In the \ippstage{diff} stage, the IPP generates diffferece images for
+appropriately specified pairs of images.  It is possible for the difference image to
 be generated from a pair of warp images, from a warp and a stack of
 some variety, or from a pair of stacks.  During the PS1 survey, pairs
@@ -918,21 +1523,69 @@
 for these fields each night.  
 
-For the PV3 processing, the entire collection of warps for the $3\pi$
+For the PV3 $3\Pi$ processing, the entire collection of warps for the
 survey were combined with the $3\pi$ stacks to generate `warp-stack
 diffs'.  
 
 \subsection{Addstar : DVO Ingest}
+\label{subsec: addstar}
 
 \subsection{Calibration Operations}
+\label{subsec: calibration}
 
 \subsection{IPP to PSPS}
+\label{subsec: ipp2psps}
+\note{Default to pointing to Flewelling et al 2017?}
 
 \subsection{PSPS Load \& Merge}
+\label{subsec: psps}
+\note{Default as well to pointing to Flewelling et al 2017?}
 
 \section{IPP Hardware Systems}
+\label{sec: hardware}
+
 
 \subsection{Kihei Processing Cluster} 
+\label{subsec: kihei}
+
+The majority of all Pan-STARRS processing has been performed on the
+dedicated IPP cluster, located in Kihei on Maui.  This cluster was
+originally located at the Maui High Performance Computing Center
+(MHPCC), a United States Air Force research laboratory center managed
+by the University of Hawaii.  This site was chosen based on the
+original development funding provided by the Air Force Research Labs
+\citep[see][for more details]{chambers2017}.  Once the Air Force
+funding stopped being a significant driver for Pan-STARRS, the cluster was
+physically moved from the MHPCC to a similar nearby computing center
+located at the Maui Research and Technology Center.
+
+The computing cluster is comprised of three main types of computers,
+with a variety of individual specifications due to the cluster being
+assembled from multiple generations of purchases.  The data storage
+nodes contain \note{check, as this contains B nodes} approximately 10
+petabytes of storage space that are used to store both the raw
+exposure data downloaded from the telescope as well as processed data
+products.  These nodes are also used to do processing, and have jobs
+targeted to them in an effort to reduce the network I/O demands
+(see~\ref{chip section} for more on this process).
+
+These storage nodes are not fully capable of completing all processing
+on the short timescale necessary for each night's worth of data.  To
+increase the processing capability, we have a large number
+\note{actual number?} of ``compute'' nodes, that have small amounts of
+local storage, but are able to add processing power.  In addition to
+the direct processing of image data, these nodes are also used to
+manage the \ippprog{Nebulous} file interface, as well as controlling
+the job scheduling for the processing.
+
+The final type of computer in the cluster are the database servers.
+These special purpose computers \note{have lots of memory and disk
+  space?  Is that it?} are used to store and manage both the IPP gpc1
+and \ippprog{Nebulous} databases.  In addition to the main master
+servers, we have duplicate servers used as database replicants, which
+allow for quick switching from the main to backup servers in case of a
+hardware issue that cannot be resolved immediately.
 
 \subsection{Los Alamos National Labs} 
+\label{subsec: LANL}
 
 In order to increase the processing rate for the $3\Pi$ PV3 data, we
@@ -1036,4 +1689,5 @@
 
 \subsection{UH Cray Cluster} 
+\label{subsec: UH Cray}
 
 In December 2014, the University of Hawaii installed a 178-compute
@@ -1060,4 +1714,6 @@
 994,890 runs processed there.
 
+\section{Discussion}
+\label{sec: discussion}
 
 \acknowledgments
@@ -1083,4 +1739,66 @@
 \input{datasystem.bbl}
 
+\appendix
+
+\section{GPC1 Database Schema Outline}
+\label{sec: database schema}
+
+Table \ref{tab: database schema} provides a list of a majority of the
+tables in the GPC1 database schema.  Tables that have been excluded
+are either no longer used in IPP processing, or are used for rare
+reductions that were not used for the PV3 data release.  The tables
+are grouped into stages, with the primary table and any secondary
+tables for that stage listed together, along with the primary key
+column that link the tables together.
+
+\begin{center}
+\begin{deluxetable}{lllll}
+  \tablecolumns{5}
+  \tablewidth{0pc}
+  \tablecaption{GPC1 Database Schema Outline}
+  \tablehead{\colhead{Stage} & \colhead{Primary Table} & \colhead{Secondary Table} & \colhead{Key} & \colhead{Notes}}
+  \startdata
+  \ippstage{addstar}      & \ippdbtable{addRun}       & \ippdbtable{addProcessedExp}     & \ippdbcolumn{add\_id} & \\
+  \ippstage{camera}       & \ippdbtable{camRun}       & \ippdbtable{camProcessedExp}     & \ippdbcolumn{cam\_id} & \\
+  \ippstage{chip}         & \ippdbtable{chipRun}      & \ippdbtable{chipProcessedImfile} & \ippdbcolumn{chip\_id} & \\
+  \ippstage{detrend}      & \ippdbtable{detRun}       & \ippdbtable{detRunSummary}       & \ippdbcolumn{det\_id} & \\
+                          &                           & \ippdbtable{detInputExp}         & & \\
+                          &                           & \ippdbtable{detRegisteredImfile} & & Information about detrends produced externally.\\
+                          &                           & \ippdbtable{detStackedImfile}    & & \\
+                          & \ippdbtable{detProcessedExp} & \ippdbtable{detProcessedImfile}  & & \\
+                          & \ippdbtable{detResidExp}  & \ippdbtable{detResidImfile}      & & \\
+                          & \ippdbtable{detNormalizedExp} & \ippdbtable{detNormalizedImfile} & & \\
+  \ippstage{diff}         & \ippdbtable{diffRun}      & \ippdbtable{diffSkyfile}         & \ippdbcolumn{diff\_id} & \\
+                          &                           & \ippdbtable{diffInputSkyfile}    & & \\
+  \ippstage{distribution} & \ippdbtable{distRun}      & \ippdbtable{distComponent}       & \ippdbcolumn{dist\_id} & \\
+                          &                           & \ippdbtable{distTarget}          & & \\
+  \ippstage{fake}         & \ippdbtable{fakeRun}      & \ippdbtable{fakeProcessedImfile} & \ippdbcolumn{fake\_id} & \\
+  \ippstage{fullforce}    & \ippdbtable{fullForceRun} & \ippdbtable{fullForceInput}      & \ippdbcolumn{ff\_id} & \\
+                          &                           & \ippdbtable{fullForceResult}     & & \\
+                          &                           & \ippdbtable{fullForceSummary}    & & Properties about average parameters from all results.\\
+  \ippstage{lap}          & \ippdbtable{lapSequence}  & \ippdbtable{lapRun}              & \ippdbcolumn{seq\_id} & Sequence of full reprocessing\\
+                          & \ippdbtable{lapRun}       & \ippdbtable{lapExp}              & \ippdbcolumn{lap\_id} & \\
+  \ippstage{publish}      & \ippdbtable{publishRun}   & \ippdbtable{publishDone}         & \ippdbcolumn{pub\_id} & \\
+                          &                           & \ippdbtable{publishClient}       & & \\
+  \ippstage{summitcopy}   & \ippdbtable{pzDataStore}  &                                  & & Lists locations to check for new exposures.\\
+                          & \ippdbtable{summitExp}    & \ippdbtable{summitImfile}        & \ippdbcolumn{summit\_id} & Exposures available at the telescope.\\
+                          & \ippdbtable{pzDownloadExp}& \ippdbtable{pzDownloadImfile}    & & Exposures that are being downloaded.\\
+                          & \ippdbtable{newExp}       & \ippdbtable{newImfile}           & \ippdbcolumn{exp\_id} & Exposures that have been saved to IPP cluster.\\
+
+  \ippstage{registration} & \ippdbtable{rawExp}       & \ippdbtable{rawImfile}           & \ippdbcolumn{exp\_id} & \\
+  \ippstage{remote}       & \ippdbtable{remoteRun}    & \ippdbtable{remoteComponent}     & \ippdbcolumn{remote\_id} & \\
+  \ippstage{skycal}       & \ippdbtable{skycalRun}    & \ippdbtable{skycalResult}        & \ippdbcolumn{skycal\_id} & \\
+  \ippstage{stack}        & \ippdbtable{stackRun}     & \ippdbtable{stackInputSkyfile}   & \ippdbcolumn{stack\_id} & \\
+                          &                           & \ippdbtable{stackSumSkyfile}     & & \\
+  \ippstage{staticsky}    & \ippdbtable{staticskyRun} & \ippdbtable{staticskyInput}      & \ippdbcolumn{sky\_id} & \\
+                          &                           & \ippdbtable{staticskyResult}     & & \\
+  \ippstage{warp}         & \ippdbtable{warpRun}      & \ippdbtable{warpImfile}          & \ippdbcolumn{warp\_id} & \\
+                          &                           & \ippdbtable{warpSkyCellMap}      & & Mapping of input chips to projection skycells.\\
+                          &                           & \ippdbtable{warpSkyfile}         & & \\
+  \enddata
+  \label{tab: database schema}
+\end{deluxetable}
+\end{center} 
+ 
 
 \end{document}