Index: trunk/doc/release.2015/ps1.datasystem/datasystem.tex
===================================================================
--- trunk/doc/release.2015/ps1.datasystem/datasystem.tex	(revision 40003)
+++ trunk/doc/release.2015/ps1.datasystem/datasystem.tex	(revision 40004)
@@ -88,5 +88,5 @@
 
 \section{Introduction}
-\label{sec: intro}
+\label{sec:intro}
 
 This is the second in a series of seven papers describing the
@@ -156,13 +156,13 @@
 
 This paper presents a description of the IPP data handling system.
-Section \ref{sec: subsystems} describes the major IPP subsystems that
+Section \ref{sec:subsystems} describes the major IPP subsystems that
 underlie the main pipeline, providing a set of common interfaces and
 tools used at multiple stages.  The main processing stages of the
-pipeline are described in Section \ref{sec: stages}, although all
+pipeline are described in Section \ref{sec:stages}, although all
 exposures may not necessarily pass through each of these stages.  The
 hardware systems that have done the processing for the PV3 data
-release are listed in Section \ref{sec: hardware}, with some details
+release are listed in Section \ref{sec:hardware}, with some details
 on the scale of computing needed to reduce this large number of
-exposures.  Finally, Section \ref{sec: discussion} presents a
+exposures.  Finally, Section \ref{sec:discussion} presents a
 discussion of some of the lessons learned in the creation of the IPP,
 and its utility in reducing data from other cameras and telescopes.
@@ -288,8 +288,8 @@
 
 \section{IPP Data Processing Stages}
-\label{sec: stages}
+\label{sec:stages}
 
 \subsection{Processing Database}
-\label{subsec: database}
+\label{sec:database}
 
 A critical element in the Pan-STARRS IPP infrastructure is the
@@ -315,24 +315,24 @@
 primary table which defines the conceptual list of processing items
 either to be done, in progress, or completed.  An associated secondary
-table lists the details of elements which have been processed.  Table
-\ref{tab: database schema} contains an outline of the database schema,
-showing the relations between tables organized by processing stage.
-As an example, one critical stage is the \ippstage{chip} processing
-stage, discussed below, in which the individual chips from an exposure
-are detrended and sources are detected.  Within the gpc1 database, the
-primary table called \ippdbtable{chipRun} in which each exposure has a
-single entry.  Associated with this table is the
-\ippdbtable{chipProcessedImfile} table, which contains one row for
-each of the (up to 60) chips associated with the exposure.  The
-primary tables, such as \ippdbtable{chipRun}, are populated once the
-system has decided that a specific item (e.g., an exposure) should be
-processed at that stage.  Initially, the entry is given a state of
-``run'', denoting that the exposure is ready to be processed.  The
-low-level table entries, such as the \ippdbtable{chipProcessedImfile}
-entries, are only populated once the element (e.g., the chip) has been
-processed by the analysis system.  Once all elements for a given
-stage, e.g., chips in this case, are completed, then the status of the
-top-level table entry (\ippdbtable{chipRun}) are switched from ``run''
-to ``full''.
+table (or set of tables) lists the details of elements which have been
+processed.  Table \ref{tab: database schema} contains an outline of
+the database schema, showing the relations between tables organized by
+processing stage.  As an example, one critical stage is the
+\ippstage{chip} processing stage (see \S\ref{sec:chip}) in which the
+individual chips from an exposure are detrended and sources are
+detected.  Within the gpc1 database, the primary table is called
+\ippdbtable{chipRun} in which each exposure has a single entry.
+Associated with this table is the \ippdbtable{chipProcessedImfile}
+table, which contains one row for each of the chips
+associated with the exposure (up to 60 for gpc1).  The primary tables, such as
+\ippdbtable{chipRun}, are populated once the system has decided that a
+specific item (e.g., an exposure) should be processed at that stage.
+Initially, the entry is given a state of ``run'', denoting that the
+exposure is ready to be processed.  The low-level table entries, such
+as the \ippdbtable{chipProcessedImfile} entries, are only populated
+once the element (e.g., the chip) has been processed by the analysis
+system.  Once all elements for a given stage, e.g., chips in this
+case, are completed, then the status of the top-level table entry
+(\ippdbtable{chipRun}) are switched from ``run'' to ``full''.
 
 If the analysis of an element (e.g., the individual OTA chip)
@@ -357,15 +357,15 @@
 data, dropping the failed chips from the rest of the analysis.  On the
 other hand, a \ippdbcolumn{fault} in one of the elements for a given
-stage will block any dependent stages from processing that item.  In
-this way, if such a temporary failure occurs, the system will not
-process an exposure through subsequent stages without the component
-that has failed temporarily.  Since many of the \ippdbcolumn{fault}s
-which occur are ephemeral, the processing stages are set up to
-occasional clear and re-try the faulted entries.  Thus, automatic
-processing is able to keep the data flowing even in the face of
-occasional network glitches or hardware crashes.
+stage will block any successive stages which depend on that result
+from processing that item.  In this way, if such a temporary failure
+occurs, the system will not process an exposure through subsequent
+stages without the component that has failed temporarily.  Since many
+of the \ippdbcolumn{fault}s which occur are ephemeral, the processing
+stages are set up to occasional clear and re-try the faulted entries.
+Thus, automatic processing is able to keep the data flowing even in
+the face of occasional network glitches or hardware crashes.
 
 \subsection{Summit copy}
-\label{subsec: summit copy}
+\label{sec:summitcopy}
 
 As exposures are taken by the PS1 telescope \& GPC1 camera system, the
@@ -403,5 +403,5 @@
 
 \subsection{Image Registration}
-\label{subsec: registration}
+\label{sec:registration}
 
 Once the chips for an exposure have all been downloaded, the exposure
@@ -453,5 +453,5 @@
 
 \subsection{Chip Processing}
-\label{subsec: chip}
+\label{sec:chip}
 
 The science analysis of an exposure begins with the \ippstage{chip}
@@ -556,5 +556,5 @@
 
 \subsection{Camera Calibration}
-\label{subsec: camera}
+\label{sec:camera}
 
 After sources have been detected and measured for each of the chips,
@@ -611,5 +611,6 @@
 
 \subsection{Fake Analysis}
-\label{subsec: fake}
+\label{sec:fake}
+\note{drop}
 
 The \ippstage{fake} stage was originally designed to do false source
@@ -628,5 +629,5 @@
 
 \subsection{Image Warping}
-\label{subsec: warp}
+\label{sec:warp}
 
 The \ippstage{warp} stage moves the data from a given exposure beyond
@@ -682,5 +683,5 @@
 
 \subsection{Stack Combination}
-\label{subsec: stack}
+\label{sec:stack}
 
 The skycell images generated by the \ippstage{warp} process are added
@@ -742,5 +743,5 @@
 
 \subsection{Stack Photometry}
-\label{subsec: staticsky}
+\label{sec:staticsky}
 
 Although images are generated in the \ippstage{stack} stage of the
@@ -801,5 +802,7 @@
 
 \subsection{Forced Warp Photometry}
-\label{subsec: fullforce}
+\label{sec:fullforce}
+
+\note{too much detail in this section; balance relative to psphot}
 
 Traditionally, projects which use multiple exposures to increase the
@@ -872,5 +875,5 @@
 are excessively masked on a particular image are not used to model the
 PSF).  \note{this doesn't seem correct, as each warp is run
-  independently.}  The PSF model is fitted to all of the known source
+  independently. EAM: not true!}  The PSF model is fitted to all of the known source
 positions in the warp images.  Aperture magnitudes, Kron magnitudes,
 and moments are also measured at this stage for each warp.  Note that
@@ -881,5 +884,5 @@
 warps.  When combined together, these low-significance measurements
 will result in a signficant measurement as the signal-to-noise
-increases by $\sqrt{N}$.
+increases by the square root of the number of measurements.
 
 Upon completion of the forced photometry (for point sources as well as
@@ -899,4 +902,5 @@
 \subsubsection{Forced Galaxy Models}
 \note{CZW: is this the appropriate place for this section?}
+\note{too much detail in this section; balance relative to psphot}
 
 The convolved galaxy models are also re-measured on the
@@ -953,5 +957,5 @@
 
 \subsection{Difference Images}
-\label{subsec: diff}
+\label{sec:diff}
 Two of the primary science drivers for the Pan-STARRS system are the
 search hazardous asteroids and the search for Type Ia supernovae to
@@ -1020,5 +1024,5 @@
 
 \subsection{DVO}
-\label{subsec: DVO}
+\label{sec:DVO}
 
 The Pan-STARRS IPP uses an internal database system, distinct from the
@@ -1027,5 +1031,5 @@
 part of the astrometric and photometric calibration process.  This
 database system, called the ``Desktop Virtual Observatory'' (DVO) was
-developed originally for the LONEOS project, and used as part of the
+developed originally for the LONEOS project \citep{}, and used as part of the
 CFHT Elixir system \citep{2004PASP..116..449M}.  The capabilities of
 this databasing system have been somewhat expanded for the Pan-STARRS
@@ -1068,41 +1072,46 @@
 type of database table is stored as a separate file, or a collection
 of files for table which are spatially partitioned.  The binary FITS
-tables may be optionally compressed using the (to date) experimental
-FITS binary table compression strategy outlined by \note{REF}.  In this
-compression scheme, using a strategy similar to that used for FITS
-image compression (\note{REF}), the data stored in the binary table is
-compressed and stored in the 'HEAP' section of the FITS table.  In
-brief, each column in the FITS table is compressed as one (or more)
-blocks.  The standard fields which describe the data column format
-(e.g., TFORM1) are replaced with columns which describe the location
-and size of the compressed data in the HEAP section; the information
-about the uncompressed data is moved to a field with 'Z' prepended
-(e.g., ZFORM1) and an additional field is added to define the
-compression algorithm (e.g., ZCTYP1).  The column names (e.g., TTYPE1)
-and units (e.g., TUNIT1) are retained in their original form.  The
-compression algorithm can treat the entire column as a single block of
-data, or it may be broken into a number of chunks, each compressed in
-turn (this must be the same for all columns).  Additional header
-information is added to describe the block sizes and infomation needed
-to describe the HEAP data section.  The compression algorithms
-currently defined consist of the GZIP, RICE, PLIO, and HCOMPRESS
-(REFS).  For GZIP, the compression algorithm may transpose the byte
-order before compression: for floating point data of a similiar
-dynamic range, this choice may allow for better compression as each
-byte in the 4 or 8 byte floating point value is more likely to be
-similar to the same byte in other rows than to the other bytes of the
-same row value.  This option is called \code{GZIP_2} in the FITS
+tables are compressed using the (to date) experimental FITS binary
+table compression strategy outlined by \note{REF}.  Table compression
+is in general an option in DVO; for the PV3 database, the large data
+volume (70TB compressed) drove the decision to compress the tables.
+
+The FITS binary table compression scheme uses a strategy similar to
+that used for FITS image compression (\note{REF}).  The binary tabular
+data is compressed and stored in the 'HEAP' section of the FITS table
+extension, with pointers to the compressed data stored in the regular
+data section.  Each column in the FITS table is compressed as one (or
+more) blocks.  The standard header keywords which describe the data
+column format (e.g., TFORM1) are replaced with keywords which describe
+the location and size of the compressed data in the HEAP section; the
+information about the uncompressed data is moved to a keyword with 'Z'
+prepended (e.g., ZFORM1) and an additional field is added to define
+the compression algorithm (e.g., ZCTYP1).  The column names (e.g.,
+TTYPE1) and units (e.g., TUNIT1) are retained in their original form.
+
+The compression algorithm can treat the entire column as a single
+block of data, or it may be broken into a number of chunks, each
+compressed in turn (this must be the same for all columns).
+Additional header information is added to describe the block sizes and
+infomation needed to describe the HEAP data section.  The compression
+algorithms currently defined consist of the GZIP, RICE, PLIO, and
+HCOMPRESS (REFS).  For GZIP, the compression algorithm may transpose
+the byte order before compression: for floating point data of a
+similiar dynamic range, this choice may allow for better compression
+as each byte in the 4 or 8 byte floating point value is more likely to
+be similar to the same byte in other rows than to the other bytes of
+the same row value.  This option is called \code{GZIP_2} in the FITS
 standard, as opposed to the standard order, \code{GZIP_1}.  The DVO
 system can be set to specify the compression options for each column
 in the tables.  In practice, we have chosen a default in which
-floating point numbers used \code{GZIP_2}, character strings use
-\code{GZIP_1}, integers use \code{RICE}.  
+floating point numbers use \code{GZIP_2}, character strings use
+\code{GZIP_1}, integers use \code{RICE}.
 
 \subsubsection{Sky Partition}
 
 DVO includes two major classes of database tables: those containing
-information directly about astronomical objects in the sky and those
-containing other supporting information.  The object-related tables
-are partitioned on the basis of position in the sky: objects within a
+information about astronomical objects in the sky and those containing
+other supporting information.  The object-related tables are
+partitioned on the basis of position in the sky: objects within a
 region bounded by lines of constant RA,DEC are contained in a specific
 file.  The boundaries and the associated partition names are stored in
@@ -1111,26 +1120,27 @@
 (\ippdbcolumn{R\_MIN}, \ippdbcolumn{R\_MAX}, \ippdbcolumn{D\_MIN},
 \ippdbcolumn{D\_MAX}), the name of the sky region, an ID
-(\ippdbcolumn{INDEX}, equal to the sequence number of the region in the
-table), and index entries to enable navigation within the table.  The
-regions are defined in a hierarchical sense, with a series of levels
-each containing a finer mesh of regions covering the sky.
+(\ippdbcolumn{INDEX}, equal to the sequence number of the region in
+the table), and index entries to enable navigation within the table.
+The regions are defined in a hierarchical sense, with a series of
+levels each containing a finer mesh of regions covering the sky.
 
 In the default used by the PV3 DVO, the partitioning scheme is based
 on the one used by the Hubble Space Telescope Guide Star Catalog
-files.  Level 0 is a single region covering the full sky.  Level 1
-divides the sky in Declination into bands 7.5\degree\ high.  Level 2
-subdivides these Declination bands in the RA direction, with spacing
-related to the stellar density.  Level 3 divides these RA chunks into
-4 - 8 smaller partitions.  This level exactly matches the HST GSC
-layout, and uses the same naming convention to identify the
-partitions: \code{n0000/0000}, etc. \note{more on the names?}.  Level
-4 further divides these regions by a factor of 16.  In the
-\ippdbtable{SkyTable}, a region at one level has a pointer to its
-parent region (the one which contains it) and a sequence pointing to
-its children (regions it contains).  The \ippdbtable{SkyTable} enables
-fast lookups of the on-disk partitions which map to a specific
-coordinate on the sky.  In general, a single DVO will have the full
-sky represented with tables at a single level, though it is possible
-for mixed levels to be used, this mode is not well tested.  For the
+files.  \note{add figure} Level 0 is a single region covering the full
+sky.  Level 1 divides the sky in Declination into bands
+7.5\degree\ high.  Level 2 subdivides these Declination bands in the
+RA direction, with spacing related to the stellar density.  Level 3
+divides these RA chunks into 4 - 8 smaller partitions.  This level
+exactly matches the HST GSC layout, and uses the same naming
+convention to identify the partitions: \code{n0000/0000},
+etc. \note{more on the names?}.  Level 4 further divides these regions
+by a factor of 16.  In the \ippdbtable{SkyTable}, a region at one
+level has a pointer to its parent region (the one which contains it)
+and a sequence pointing to its children (regions it contains).  The
+\ippdbtable{SkyTable} enables fast lookups of the on-disk partitions
+which map to a specific coordinate on the sky.  In general, a single
+DVO will have the full sky represented with tables at a single
+level. Although it is possible for mixed levels to be used, this mode
+is not well tested and is avoided in the PV3 DVO database.  For the
 PV3 master database, the partitioning at the \note{should this be
   4th?} 5th level results in \approx 150,000 regions to cover the full
@@ -1138,8 +1148,11 @@
 densest portions of the bulge contain at most \approx 300,000
 astronomical objects in the database files, with an associated maximum
-of 30 million measurements in these files.  With the compression
-scheme described above, this makes the largest database files \approx
+of \approx 30 million measurements in these files.  With the compression
+scheme described above, the largest database files are \approx
 3GB, which can be loaded into memory in 30 seconds on the processing
 machines that contain partition data.
+
+\note{is the use of the term 'partition host' consistent in this paper
+  and the calibration paper?}
 
 The DVO software system allows the tables which are partitioned across
@@ -1157,8 +1170,12 @@
 query operations will select the database information which matches
 the query request (i.e., applying restrictions as defined) and return
-to the master process the results.  The results from the various
+the results to the master process.  The results from the various
 partition hosts are then merged into a single result by the master
-process.  This parallelization is critical to querying and
-manipulating the enormous database on a reasonable timescale.
+process.  When the parallel partitioning for a DVO instance is
+defined, the tables are randomly assigned to the partition hosts.  As
+a result, queries which span more than a single parition are likely to
+spread the I/O load across a large number of machines.  This
+parallelization is critical to querying and manipulating the enormous
+database on a reasonable timescale.
 
 \subsubsection{Astronomical Objects}
@@ -1168,5 +1185,5 @@
 \ippdbtable{SecFilt}.  These two tables specify the main average
 properties of the astronomical object.  The \ippdbtable{Average} table includes the
-astrometric information ($\alpha, \delta, \mu \alpha, \mu \delta,
+astrometric information ($\alpha, \delta, \mu_\alpha, \mu_\delta,
 \pi$) and associated errors, data quality flags for each object, links
 to the other tables, and a number of IDs, with one row for each
@@ -1198,8 +1215,14 @@
 
 The individual measurements of the astronomical objects are carried in
-the table \ippdbtable{Measure}.  This table lists the values measured
-by \ippprog{psphot} for each \ippstage{chip}, \ippstage{warp}, or
-\ippstage{stack} stage image.  This includes the instrumental magnitudes for
-the PSF, aperture, and Kron photometry; raw position
+the table \ippdbtable{Measure}.  For measurements from PS1 in the PV3
+/ DR1 database, this would be values determined by \ippprog{psphot}
+for each \ippstage{chip}, \ippstage{warp}, or \ippstage{stack} stage
+image.  Measurements for other cameras processed by the IPP may also
+be included similarly in a DVO database.  Measurements from other
+sources, such as SDSS, 2MASS, or WISE, can also be included in this
+table (see \S\ref{sec:other.photometry}.
+
+The \ippdbtable{Measure} table includes the instrumental magnitudes
+for the PSF, aperture, and Kron photometry; raw position
 (\ippdbcolumn{Xccd}, \ippdbcolumn{Yccd}) and second moments
 (\ippdbcolumn{Mxx}, \ippdbcolumn{Myy}, \ippdbcolumn{Mxy}), along with
@@ -1207,8 +1230,8 @@
 (\ippdbcolumn{FWx}, \ippdbcolumn{FWy}, \ippdbcolumn{theta}).  Metadata
 about the exposure that the measurement was derived from is also
-include, such as the exposure time, the date \& time of the
+included, such as the exposure time, the date \& time of the
 observation, airmass, azimuth, and \ippdbcolumn{photcode} information
 specifying the filter.  The \ippdbtable{Measure} table also carries
-the calibration magnitude offsts ($M_{\rm cal}$ and $M_{\rm flat}$,
+the calibration magnitude offsets ($M_{\rm cal}$ and $M_{\rm flat}$,
 discussed below) and the astrometrically calibrated position.
 Astrometric offsets for several systematic corrections discussed below
@@ -1216,5 +1239,6 @@
 photometry may have non-significant values, the table is somewhat
 de-normalized in that it also carries instrumental flux values for the
-PSF, aperture, and Kron photometry.
+PSF, aperture, and Kron photometry.  In this case, we have chosen to
+trade storage space for computing time.
 
 In the \ippdbtable{Measure} table, there are three fields which
@@ -1225,10 +1249,10 @@
 the \ippdbtable{Average} table the measurement belongs.  These two 32
 bit fields can thus be combined into a single 64 bit ID unique for all
-objects in the database.  In addition, the field
+objects in the database.  \note{PSPS IDs} In addition, the field
 \ippdbtable{Measure}.\ippdbcolumn{averef} specifies the row number in
 the \ippdbtable{Average} table of the associated object.  The
 \ippdbtable{Measure} table may be unsorted, in which case it is slow
 to find the measurements associated with a specific object (a full
-table scan is required).  After the table is sorted, the
+table scan is required).  After the table is sorted and indexed, the
 \ippdbcolumn{Measure} rows for a given object are grouped together.
 In this case, the fields
@@ -1290,23 +1314,29 @@
 \subsubsection{Other Tables} 
 
-Data from GPC1 (and other cameras processed by the IPP) are loaded
-into DVO in units \code{smf} files generated by the \ippstage{camera}
-calibration stage (see section \ref{subsec: camera} below).  As
-described above, these files contain all measurements from a complete
-exposure, with measurements from each chip grouped into separate FITS
-tables.  When these measurements are loaded into the
-\ippdbtable{Measure} and similar tables, a subset of the information
-from the chip header is used to populated a row in the DVO
-\ippdbtable{Image} table.  This table contains one row for each chip
-known to DVO, with information such as the filter
-(\ippdbcolumn{photcode}), the exposure time, the airmass, the
-astrometric calibration terms, the photometric zero point, etc.  For
-GPC1 and other mosaic cameras, an additional row is defined to carry
-the projection and camera distortion elements of the astrometry model.
-As chips are loaded into this table, they are assigned an internal ID
-(a running sequence in the table).  Images may also be assigned an
-external ID: in the case of the GPC1 images, this ID is defined by the
-processing mysql database and is guaranteed to be unique within the
-processing system.
+Measurements which are loaded into DVO may be associated with a
+specific image (such as the measurements for a single chip from the
+GPC1 camera) or they may not have such an association (such as
+measurements from 2MASS, WISE, or externally supplied reference
+measurements).  For data which is associated with an image, a subset
+of the information about that image (e.g., from the header of the FITS
+file) is used to populate a row in the DVO \ippdbtable{Image} table.
+This table contains one row for each chip image known to DVO, with
+information such as the filter (\ippdbcolumn{photcode}), the exposure
+time, the airmass, the astrometric calibration terms, the photometric
+zero point, etc.  For GPC1 and other mosaic cameras, an additional row
+is defined to carry the projection and camera distortion elements of
+the astrometry model.  As images are loaded into this table, they
+are assigned an internal ID (a running sequence in the table).  Images
+may also be assigned an external ID: in the case of the GPC1 images,
+this ID is defined by the processing mysql database and is guaranteed
+to be unique within the processing system.
+
+%% Data from GPC1 (and other cameras processed by the IPP) are loaded
+%% into DVO in units \code{smf} files generated by the \ippstage{camera}
+%% calibration stage (see section \ref{sec:camera} below).  As
+%% described above, these files contain all measurements from a complete
+%% exposure, with measurements from each chip grouped into separate FITS
+%% tables.  When these measurements are loaded into the
+%% \ippdbtable{Measure} and similar tables, 
 
 Other tables are used to track information used by the calibration
@@ -1317,5 +1347,5 @@
 
 \subsection{Addstar : DVO Ingest}
-\label{subsec: addstar}
+\label{sec:addstar}
 \note{CZW: This should be reviewed.}
 
@@ -1351,12 +1381,12 @@
 
 \subsection{Calibration Operations}
-\label{subsec: calibration}
+\label{sec:calibration}
 
 \subsection{IPP to PSPS}
-\label{subsec: ipp2psps}
+\label{sec:ipp2psps}
 \note{Default to pointing to Flewelling et al 2017?}
 
 \subsection{PSPS Load \& Merge}
-\label{subsec: psps}
+\label{sec:psps}
 \note{Default as well to pointing to Flewelling et al 2017?}
 
@@ -1364,5 +1394,7 @@
 
 \subsection{Pantasks \& Parallel Processing}
-\label{subsec: pantasks}
+\label{sec:pantasks}
+
+\note{this section needs to be re-written : pclient vs pcontrol vs pantasks}
 
 The actual processing of data is managed by the \ippprog{pantasks}
@@ -1377,6 +1409,5 @@
 
 The \ippmisc{load} task for a particular stage queries the processing
-database via an appropriate \ippmisc{ippTool} (see section \ref{subsec:
-  ipptools} below) for a list of jobs that are waiting to be run.
+database via an appropriate \ippmisc{ippTool} (see section \ref{sec:ipptools} below) for a list of jobs that are waiting to be run.
 This task is executed on the host running the \ippprog{pantasks}
 server, and only one of each type of \ippmisc{load} task is permitted to
@@ -1441,7 +1472,8 @@
 
 \subsubsection{Stage automation}
-\label{subsec: automation}
+\label{sec:automation}
 \note{I'm not convinced this is the right place for it, but it felt like a natural extension of the ``advance''}.
 
+\note{wording..} 
 Beyond the warp stage, there is no longer a single ``next'' stage into
 which data can be queued.  Because of this, more robust methods are
@@ -1491,5 +1523,5 @@
 entries using whatever exposures are available if one has not yet been
 constructed by the time the morning dark exposures are registered into
-the database.
+the database. \note{wording}
 
 Automating the nightly processing is important, as it ensures that
@@ -1497,5 +1529,5 @@
 reducing the lag between an observation and the reduced data.  The
 other processing task that requires automation is the reprocessing of
-the entire $3\Pi$ survey, as the size of the dataset make it
+the entire $3\pi$ survey, as the size of the dataset make it
 challenging to do manually.  To manage this, the ``large area
 processing'' (LAP) task and script are used.  The first stage of this
@@ -1506,33 +1538,33 @@
 considered.  A \ippdbcolumn{projection\_cell} is a unit of sky defined
 to be a square four degrees on each side which has a single tangent
-plane projection \citep[][see]{waters2017}.  Once this entry is
-defined, is is populated with exposures (stored in the
-\ippdbtable{lapExp} table in the database), with any exposure located
-within 5 degrees of the center of the projection cell included.  This
-radius ensures that any exposure that overlaps the projection cell
-will be included.  Once the exposures have been added, the other
-exposures within the same sequence are checked to see if a
-\ippstage{chip} stage entry has been generated, and if so, the
-\ippdbcolumn{chip\_id} for that entry is saved into the
-\ippdbtable{lapExp} as well.  This linkage ensures that each exposure
-is only processed once.  If no entry is found, a new \ippstage{chip}
-entry is queued for processing.  The task periodically checks the
-status of the exposures in each \ippdbtable{lapRun} entry, and if they
-have all completed the \ippstage{warp} stage, then a \ippstage{stack}
-is queued for each skycell contained within the
+plane projection \citep[][see]{waters2017}.  \note{does waters2017
+  discuss RINGS.V3? if not, where?}  Once this entry is defined, is is
+populated with exposures (stored in the \ippdbtable{lapExp} table in
+the database), with any exposure located within 5 degrees of the
+center of the projection cell included.  This radius ensures that any
+exposure that overlaps the projection cell will be included.  Once the
+exposures have been added, the other exposures within the same
+sequence are checked to see if a \ippstage{chip} stage entry has been
+generated, and if so, the \ippdbcolumn{chip\_id} for that entry is
+saved into the \ippdbtable{lapExp} as well.  This linkage ensures that
+each exposure is only processed once.  If no entry is found, a new
+\ippstage{chip} entry is queued for processing.  The task periodically
+checks the status of the exposures in each \ippdbtable{lapRun} entry,
+and if they have all completed the \ippstage{warp} stage, then a
+\ippstage{stack} is queued for each skycell contained within the
 \ippdbcolumn{projection\_cell}.
 
 \subsection{Nebulous}
-\label{subsec: nebulous}
+\label{sec:nebulous}
 Storing the large volume of data that is generated by the GPC1 camera
 was recognized early in the Pan-STARRS project as a major concern.
 The \ippprog{Nebulous} system was designed to organize this data.  The
-main components of this system is a database storing the locations of
-the files, with a Simple Object Access Protocol interface between the
-database and the other IPP programs.  The actual files are stored on
-NFS mounted partitions on a series of storage nodes in the IPP cluster
-that can be accessed throughout the cluster.  This distribution of
-files is useful to balance the disk I/O, as this parallelizes the
-load.
+main components of this system are a database storing the locations of
+the files, with a Simple Object Access Protocol (SOAP) interface
+between the database and the other IPP programs \note{define / mention
+  http}.  The actual files are stored on a collection of computers
+with substantial disk partitions in the IPP cluster, shared within the
+cluster via NFS.  This distribution of files is useful to balance the
+disk I/O, as this parallelizes the load.
 
 The original design of \ippprog{Nebulous} was intended to aid in the
@@ -1546,14 +1578,17 @@
 reduced the degree to which the IPP processing is targeted.
 
+\note{this is a critical paragraph and needs to be re-written to be
+  more accessible}
 When a program creates a new file in \ippprog{Nebulous}, it supplies
 an URI of the form \code{neb://HOST.VOLUME/PATH/FILENAME}.  The host
 and volume specifiers are optional, and allow a file to be created on
 a specific node.  The path and filename appear similar to a standard
-full file location, and are used internally as the ``external id''.  A
-storage object entry is then created in the database for this id, and
-an instance of the file created on the specified node (or at random
-from available nodes if left empty).  This instance is created in a
-deterministic filename location.  The external id is hashed using the
-SHA-1 function, and the first four hexadecimal digits of this hash are
+full file location, and are used internally as the ``external id''.
+\note{mention the nebulous schema before this?}  A storage object
+entry is then created in the database for this id, and an instance of
+the file created on the specified node (or at random from available
+nodes if left empty).  This instance is created in a deterministic
+filename location.  The external id is hashed using the SHA-1
+function, and the first four hexadecimal digits of this hash are
 separated into two two-digit strings and used as the top and second
 level directory location for the disk file.  The disk file is created
@@ -1564,5 +1599,5 @@
 \code{/data/HOST.VOLUME/nebulous/d5/d8/9244993440.PATH:FILENAME}.
 This file naming structure is useful, as it duplicates database
-contents on disk.
+contents on disk.  \note{rephrase}
 
 The storage volumes that contain the data on disk are defined in the
@@ -1578,5 +1613,6 @@
 volume should only be used as a backup volume (which accepts only
 replicated copies), and the value of $5$ is used to indicate that the
-volume is permanently unavailable, and should be ignored.
+volume is permanently unavailable, and should be ignored. \note{more
+  detail, more specific}
 
 In addition to this permanent table describing the volumes, a
@@ -1587,38 +1623,42 @@
 \ippprog{Nebulous} load balancing routines to prioritize those volumes
 with large unused disk space before filling the volumes with only
-small amounts remaining.  This table is regenerated every ten to
-twenty minutes, after a scan of each of the volumes listed in the
+small amounts remaining.  This table is updated every ten to twenty
+minutes, after a scan of each of the volumes listed in the
 \ippdbtable{volume} table.
 
 The final table controlling the operations of the \ippprog{Nebulous}
 volumes is the \ippdbtable{cabinet} table, which organizes the
-individual volumes into ``cabinets,'' a concept loosly based on the
+individual volumes into ``cabinets,'' a concept loosely based on the
 physical arrangement of the storage servers in the data center.  These
 cabinets are used to prevent the replication of a storage object
 within a group of volumes where all instances of the object could be
-taken off line by a single failure.  As the data center cabinets share
-power supplies among the servers they contain, ensuring physical
-distance between replicated copies is important to guarantee that a
-temporary failure of one of these devices does not significantly
-impact processing.
+taken off line by a single failure.  Since servers within a given
+cabinet in the data center share a common set of PDUs \note{define},
+it is important to ensure physical distance between replicated copies
+to guarantee that a temporary failure of one of the cabinet PDUs does
+not significantly impact processing.
+
+\note{need a paragraph or two on stats: how many objects, how many
+  instances?}
 
 \subsection{Datastore repositories}
-\label{subsec: datastore}
+\label{sec:datastore}
 
 Transferring data between the IPP and other parts of the Pan-STARRS
 system is generally accomplished via a ``datastore'', an http service
-that exposes data in a common form.  One of the main datastores used
-by the IPP is the one located at the summit.  This datastore exposes,
-a list of the exposures obtained since the start of the PS1
-operations.  Requests to this server may restrict to the latest by
-time.  Each row in the listing includes basic information about the
-exposure: an exposure identifier (e.g., o5432g0123o;
-see~\ref{GPC1.names} for details), the date and time of the exposure,
-the telescope commanded pointing, the filter and exposure time, and
-the observation comment for that exposure.  The row also provides a
-link to a listing of the chips associated with that exposure.  This
-listing includes a link to the individual chip FITS files as well as
-an md5 checksum.  Systems which are allowed access may download chip
-FITS files via http requests to the provided links.
+that exposes data in a common form.  \note{add Isani / Hoblitt
+  reference?}  One of the main datastores used by the IPP is the one
+located at the summit.  This datastore exposes, a list of the
+exposures obtained since the start of the PS1 operations.  Requests to
+this server may restrict to the latest by time.  Each row in the
+listing includes basic information about the exposure: an exposure
+identifier (e.g., o5432g0123o; see~\ref{GPC1.names} for details), the
+date and time of the exposure, the telescope commanded pointing, the
+filter and exposure time, and the observation comment for that
+exposure.  The row also provides a link to a listing of the chips
+associated with that exposure.  This listing includes a link to the
+individual chip FITS files as well as an md5 checksum.  Systems which
+are allowed access may download chip FITS files via http requests to
+the provided links.
 
 The IPP also uses datastores to provide access to its own data
@@ -1633,8 +1673,8 @@
 
 \subsection{ippTools and ippScripts}
-\label{subsec: ipptools}
+\label{sec:ipptools}
 
 % \section{IPP Software Subsystems}
-% \label{sec: subsystems}
+% \label{sec:subsystems}
 
 The IPP relies on a number of common libraries and programs to handle
@@ -1712,5 +1752,5 @@
 
 \subsection{psLib and psModules}
-\label{subsec: pslib}
+\label{sec:pslib}
 
 Underlying all of the analysis programs are the \ippmisc{psLib} and
@@ -1744,10 +1784,10 @@
 
 \section{IPP Hardware Systems}
-\label{sec: hardware}
+\label{sec:hardware}
 
 \note{what about psps hardware? mops hardware?}
 
 \subsection{Kihei Processing Cluster} 
-\label{subsec: kihei}
+\label{sec:kihei}
 
 The majority of all Pan-STARRS processing has been performed on the
@@ -1790,5 +1830,5 @@
 
 \subsection{Los Alamos National Labs} 
-\label{subsec: LANL}
+\label{sec:LANL}
 
 In order to increase the processing rate for the $3\Pi$ PV3 data, we
@@ -1892,5 +1932,5 @@
 
 \subsection{UH Cray Cluster} 
-\label{subsec: UH Cray}
+\label{sec:UHCray}
 
 In December 2014, the University of Hawaii installed a 178-compute
@@ -1918,5 +1958,5 @@
 
 \section{Discussion}
-\label{sec: discussion}
+\label{sec:discussion}
 
 \acknowledgments
@@ -1945,5 +1985,5 @@
 
 \section{GPC1 Database Schema Outline}
-\label{sec: database schema}
+\label{sec:database.schema}
 
 Table \ref{tab: database schema} provides a list of a majority of the
@@ -1954,4 +1994,6 @@
 tables for that stage listed together, along with the primary key
 column that link the tables together.
+
+\note{logical or alphabetical sequence?  alignment is broken}
 
 \begin{center}