Changeset 40578 for trunk

Timestamp:

Dec 7, 2018, 12:33:01 PM (8 years ago)

Author:

eugene

Message:

cleanup text, add skypartition figure, repair system overview paper

Location:

trunk/doc/release.2015/ps1.datasystem

Files:

• PS1_Data_Analysis_System_Overview.pdf (modified) ( previous)
• datasystem.tex (modified) (40 diffs)
• dvo.sh (added)
• skypartition.png (added)

trunk/doc/release.2015/ps1.datasystem/datasystem.tex

@@ -241 +241 @@
   responsible for linking individual detections of solar-system
   objects together and determining the orbits \citep[][]{2013PASP..125..357D}.
-\item PSPS : this system ingests the calibrated measurements from the
-  IPP, MOPS, and others and generates a high-availability database
-  with web-based interactions for public consumption \citet[][]{flewelling2017}.
+\item Published Science Products Subsystem (PSPS) : this system
+  ingests the calibrated measurements from the IPP, MOPS, and others
+  and generates a high-availability database with web-based
+  interactions for public consumption \citet[][]{flewelling2017}.
 
 \end{itemize}
@@ -255 +256 @@
 at MAST, to those which perform offline analysis for eventual ingest
 back into the Pan-STARRS databases and archive.  The latter category
-includes the ubercal photometric analysis \citep{ubercal}, the photo-z
-analysis \citep{photoz}, and the QSO / RR Lyra search efforts
-\citep{hernitschek2016}.  In addition, collaborations within the wider
+includes the ubercal photometric analysis \citep{2012ApJ...756..158S}, the photo-z
+analysis \citep{2012ApJ...746..128S}, and the QSO / RR Lyra search efforts
+\citep{2016ApJ...817...73H}.  In addition, collaborations within the wider
 Pan-STARRS community have implemented a variety of science-level
 analyses of their own to support their science goals \citep[e.g., M31
-  variable search][]{M31.REF}.
+variable search][]{2014ApJ...797...22L,2012AJ....143...89L}.
 
 Figure~\ref{fig:analysis.elements} illustrates the many elements of
@@ -266 +267 @@
 analysis steps which occur within the Pan-STARRS Observatory, with an
 emphasis on the analysis, calibration, and database ingest stages.
-The MOPS is described in detail by \cite{2013PASP..125..357D}, while
-the summit systems are described by \note{REF?}.
+The MOPS is described in detail by \cite{2013PASP..125..357D}.
+
+% the summit systems are described by \note{REF?}.
 
 \begin{figure*}[htbp]
@@ -277 +279 @@
     external groups (``customers'').  The arrows show a simplified representation
   of the major flow of data between the analysis stages and data
-  processing elements. \note{arrow types are unclear for on-demand vs DVO}}
+  processing elements.}
   \end{center}
 \end{figure*}
@@ -298 +300 @@
 (\IPPstage{stack}) or used in an image subtraction (\IPPstage{diff}).
 As part of nightly science processing, images for certain fields such
-as the Medium Deep survey fields \citep[see][]{MDref}, are stacked
+as the Medium Deep survey fields \citep[see][]{huber2017}, are stacked
 together in nightly chunks, providing deeper detection capability on
 1-day timescales.  Depending on the survey mode, difference images are
@@ -1104 +1106 @@
 epoch.  The quality of such a difference image can be enhanced by
 convolving one or both of the images so that the PSFs in the two
-images are matched \citep[e.g.,][]{AlardLupton}.
+images are matched \citep[e.g.,][]{1998ApJ...503..325A}.
 
 In the \ippstage{diff} stage, the IPP generates difference images for
@@ -1159 +1161 @@
 entry as such.
 
-\section{Post-Processing : Database Ingest and Calibration}
-\label{sec:postprocessing}
-
 \begin{table}[hb]
 \begin{center}
-\caption{DVO Database Tables\label{tab:DVO_schema} \note{fix names, include missing}}
+\caption{DVO Database Tables\label{tab:DVO_schema}}
 \begin{tabular}{ll}
 \hline
@@ -1171 +1170 @@
 \hline
 Images               & The images that have objects in the DB. \\
-% Image Overlaps       & Image regions which are touched by specific images. \\
-Objects              & The objects --- average properties of multiple detections of the same object. \\
-Average              & Average photometry in multiple filters \\
-% Solar System Objects & Identification of solar system objects \\
-Measure              & Detections of sources in an image identified with an Object. \\
-% Orphaned Detections  & Detections of sources in an image not identified with an Object. \\
-% Non-detections       & Non-detections of objects in an image. \\
+Average              & Astronomical objects including their astrometric properties. \\
+SecFilt              & Average photometry of the objects in multiple filters (one filter per row) \\
+Measure              & Detections of sources identified with an Object, potentially linked to an image. \\
+StarPar              & Stellar parameters determined by the Harvard group \citep{2015ApJ...810...25G} \\
+Lensing              & Lensing (KSB) parameters and fixed circular aperture photometry from the warps \\
+LensObj              & Average lensing and fixed circular aperture photometry \\
+Galphot              & Result of galaxy model fits (forced galaxy models) \\
 SkyRegions           & spatial distribution of tables \\
-% Filters              & Filters understood by the system. \\
 Photcodes            & Transformations between different photometric systems \\
-% Zero Points          & History of Zero-point \& Airmass terms \\
-% Distortion Models    & History of Optical Distortion terms \\
 Hosts                & computers used to store the tables \\
 \hline
@@ -1188 +1184 @@
 \end{center}
 \end{table}
+
+\section{Post-Processing : Database Ingest and Calibration}
+\label{sec:postprocessing}
 
 \subsection{DVO}
@@ -1200 +1199 @@
 part of the astrometric and photometric calibration process.  This
 database system, called the ``Desktop Virtual Observatory'' (DVO) was
-developed originally for the LONEOS project \citep{}, and used as part of the
-CFHT Elixir system \citep{2004PASP..116..449M}.  The capabilities of
-this databasing system have been somewhat expanded for the Pan-STARRS
-context.  
+developed originally for the LONEOS project
+\citep{1995DPS....27.0110B}, and used as part of the CFHT Elixir
+system \citep{2004PASP..116..449M}.  The capabilities of this
+databasing system have been somewhat expanded for the Pan-STARRS
+context.
 
 % overview
 DVO tracks three main classes of information: 1) average properties of
 astronomical objects; 2) measurements of those objects (from which the
-average properties are derived); 3) properties of the images which provided
-some or all of the measuements.  Figure~\ref{fig:DVO_schema}
-illustrates the schematic relationship between these types of
-measurements.
+average properties are derived); 3) properties of the images which
+provided some or all of the measuements.  In addition, certain
+metadata tables define general features of the database.
+Table~\ref{tab:DVO_schema} lists the full collection of database
+tables used by DVO.
+
+%Figure~\ref{fig:DVO_schema}
+%illustrates the schematic relationship between these types of
+%measurements.
+
+\begin{figure*}[htbp]
+  \begin{center}
+ \includegraphics[width=\hsize,clip]{skypartition.png}
+  \caption{\label{fig:sky.partition} Level 3 sky paritioning.  The
+    blue grid shows the outlines of the different regions assigned to
+    separate tables in the sky partitioning scheme.  The Galactic
+    plane is shown as a solid red line while the ecliptic is shown in
+    green.  This organization of the sky duplicates that used by the
+    HST Guide Star Catalog \citep{1988IAUS..133..239J}.  
+ }
+\end{center}
+\end{figure*}
 
 In the most basic implementation, a collection of measurements for
@@ -1227 +1245 @@
 and the derived astronomical objects.
 
-\subsubsection{DVO Schema}
-
-Table~\ref{tab:DVO_schema} lists the full collection of database
-tables used by DVO.  These tables fall into one of several classes:
-those which store information about the average properties of
-astronomical objects; those which store information about individual
-measurements; those which store information about the images; those
-which store supporting information (metadata).
+% 
+%% These tables fall into one of several classes:
+%% those which store information about the average properties of
+%% astronomical objects; those which store information about individual
+%% measurements; those which store information about the images; those
+%% which store supporting information (metadata).
 
 %% DVO includes two major classes of database tables: those containing
@@ -1251 +1267 @@
 %% levels each containing a finer mesh of regions covering the sky.
 
+\subsubsection{DVO Schema}
+
 \subsubsubsection{Photcodes}
 
@@ -1269 +1287 @@
 from external data sources for which DVO does not have any information
 to determine a calibration (e.g., instrumental magnitudes and detector
-coordinates).  These are measurements are reference values and are
+coordinates).  These measurements are reference values and are
 assigned \ippmisc{REF} photcodes.
 
@@ -1283 +1301 @@
 transform a measurement in the specific photcode to a common system.
 For example, a \ippmisc{DEP} photcode GPC1.g.X01 would have the
-nominal zero point (24.563) and airmass term (0.147).  The structures
-allow for individual chips to have different color terms to bring them
-to a common filter system.
-
-Beyond the basic use, DVO has the ability to accept data from other
-kinds of data sources in which measurements are not clearly associated
-with specific images.  DVO ingest methods are defined for several
-large-scale surveys for which the published data represent average
-properties derived from multiple measurements, and for which the
-measurement-to-image relationship is not provided.  Ingests methods
-have been defined, for example, for 2MASS, WISE, Gaia, USNO-B.  In each
-of these cases, the astrometric and photometric measurements are
-stored in the \ippdbtable{Measure} table, with the data source
-identified by the photcode of the measurement.
+nominal zero point (24.563) and airmass term (0.147).  The database
+elements allow for individual chips to have different color terms to
+bring them to a common filter system.
+
+DVO ingest methods are defined for several large-scale surveys for
+which the published data represent average properties derived from
+multiple measurements, and for which the measurement-to-image
+relationship is not provided.  Ingests methods have been defined, for
+example, for 2MASS, WISE, Gaia, USNO-B.  In each of these cases, the
+astrometric and photometric measurements are stored in the
+\ippdbtable{Measure} table, with the data source identified by the
+photcode of the measurement.
 
 \subsubsubsection{Measurement Tables}
@@ -1302 +1318 @@
 In most cases, the individual measurements of the astronomical objects
 are carried in the table \ippdbtable{Measure}.  For measurements from
-PS1 in the PV3 / DR1 database, this would consist of values determined
-by \ippprog{psphot} for each \ippstage{chip}, \ippstage{warp}, or
-\ippstage{stack} stage image.  Measurements for other cameras
-processed by the IPP may also be included similarly in a DVO database.
-Measurements from other sources, such as SDSS, 2MASS, or WISE, can
-also be included in this table, distinguished by their different
-photcodes.
+PS1 in the PV3 / DR1 or DR2 databases, this would consist of values
+determined by \ippprog{psphot} for each \ippstage{chip},
+\ippstage{warp}, or \ippstage{stack} stage image.  Measurements for
+other cameras processed by the IPP may also be included similarly in a
+DVO database.  Measurements from other sources, such as SDSS, 2MASS,
+or WISE, can also be included in this table, distinguished by their
+different photcodes.
 
 The \ippdbtable{Measure} table includes the instrumental magnitudes
@@ -1338 +1354 @@
 (respectively 3.0, 4.63, and 7.43 arcsec).  This table contains one
 row for every warp image on which the object was measured. 
+
+The \ippdbtable{Galphot} table stores the results of the forced galaxy
+fitting analysis for each object that has been measured.  This table
+contains one row per filter and model type (Sersic, Exponential, or
+DeVaucouleur) if forced galaxy models have been calculate for the
+object.
+
+The \ippdbtable{Starpar} table carries measurements provided by the
+Harvard team (Green, Schlafly, Finkbeiner) from the analysis of the
+SED of objects in the PS1 $3\pi$ data, using the PV2 analysis version
+\citep{2015ApJ...810...25G,2014ApJ...783..114G}.  In this work, the
+goal was a 3D model of the dust in the Galaxy based on Pan-STARRS and
+2MASS photometry.  As part of this analysis, the authors fit the SEDs
+of all stellar sources (as determined by a cut based on the PSF -
+aperture magnitudes) with stellar models including free parameters of
+extinction, distance modulus, metallicity, and absolute r-band
+magnitude.  While these photometric distance modulus measurements are
+not extremely precise, they provide a constraint on the distance which
+is used in our analysis of the astrometry
+\citep[see][]{magnier2017.calibration}.
 
 %% Similarly to the \ippdbtable{Measure} table, the fields
@@ -1413 +1449 @@
 calculated.
 
-The \ippdbtable{Galphot} table stores the results of the forced galaxy
-fitting analysis for each object that has been measured.  This table
-contains one row per filter and model type (Sersic, Exponential, or
-DeVaucouleur) if forced galaxy models have been calculate for the
-object.
-
-The \ippdbtable{Starpar} table carries measurements provide by Greg
-Green \& Eddie Schlafly from their analysis of the SED of objects in
-the PS1 $3\pi$ data, using the PV1 version of the analysis
-\citep{2015ApJ...810...25G}.  In this work, the goal was a 3D model of
-the dust in the Galaxy based on Pan-STARRS and 2MASS photometry.  As
-part of this analysis, the authors fit the SEDs of all \note{stellar?}
-sources with stellar models including free parameters of extinction,
-distance modulus, metallicity, and absolute r-band magnitude.  While
-these photometric distance modulus measurements are not extremely
-precise (see below), they provide a constraint on the distance is used
-in our analysis of the astrometry
-\citep[see][]{magnier2017.calibration}.
-
-In the \ippdbtable{Measure} table, there are three fields which
-provide two independent links from the specific measurement to the
-associated object: \ippdbtable{Measure}.\ippdbcolumn{catID} specifies
-the spatial partition to which the measurement belongs (see
-Section~\ref{sec:SkyPartition} below);
-\ippdbtable{Measure}.\ippdbcolumn{objID} specifies to which entry in
-the \ippdbtable{Average} table the measurement belongs.  These two 32
-bit fields can thus be combined into a single 64 bit ID unique for all
-objects in the database.  In addition, the field
-\ippdbtable{Measure}.\ippdbcolumn{averef} specifies the row number in
-the \ippdbtable{Average} table of the associated object.  The
-\ippdbtable{Measure} table may be unsorted, in which case it is slow
-to find the measurements associated with a specific object (a full
-table scan is required).  After the table is sorted and indexed, the
-\ippdbcolumn{Measure} rows for a given object are grouped together.
-In this case, the fields
-\ippdbtable{Average}.\ippdbcolumn{measureOffset} and
-\ippdbcolumn{Average}.\ippdbcolumn{Nmeasure} define an index for the
-code to jump to the list of measurements for a single object.  The
-field \ippdbtable{Measure}.\ippdbcolumn{imageID} defines the link from
-the measurement to the image which supplied the measurement.
-
-\note{Discuss PSPS IDs} 
-
 \subsubsubsection{Image Tables} 
 
@@ -1470 +1463 @@
 zero point, etc.  For GPC1 and other mosaic cameras, an additional row
 is defined to carry the projection and camera distortion elements of
-the astrometry model.  As images are loaded into this table, they
-are assigned an internal ID (a running sequence in the table).  Images
-may also be assigned an external ID: in the case of the GPC1 images,
-this ID is defined by the processing mysql database and is guaranteed
-to be unique within the processing system.
+the astrometry model.  As images are loaded into this table, they are
+assigned an internal ID (a running sequence in the table), stored in
+the field \ippdbcolumn{imageID}.  Images may also be assigned an ID
+derived from the external source of the image (field
+\ippdbcolumn{externID}): in the case of the GPC1 images, this ID is
+defined by the processing mysql database and is guaranteed to be
+unique within the processing system.  In the case of GPC1 exposures,
+the external image ID is set to the database field
+\ippdbtable{chipImfile}.\ippdbcolumn{chip_imfile_id}. A second field
+(\ippdbcolumn{sourceID}) identifies which of the possible image-like
+tables supplied this image, guaranteeing uniqueness of image IDs
+across the different IPP stages.
 
 %% Data from GPC1 (and other cameras processed by the IPP) are loaded
@@ -1491 +1491 @@
 flat-field corrections determined by the astrometry calibration
 analysis \citep[see][]{magnier2017.calibration}.
-\note{use names and match DVO schema table}
 
 \subsubsection{Sky Partition}
@@ -1513 +1512 @@
 files.  Level 0 is a single region covering the full sky.  Level 1
 divides the sky in declination into bands 7.5\degree\ high, as defined
-by the HST GSC.  Level 2 subdivides these declination bands in the RA
-direction, with spacing related to the stellar density.  Level 3
-divides these RA chunks into 4 - 8 smaller partitions.  This level
-exactly matches the HST GSC layout, and uses the same naming
-convention to identify the partitions: \code{n0000/0000}, etc. Level 4
-further divides these regions by a factor of 16.  In the
-\ippdbtable{SkyTable}, a region at one level has a pointer to its
-parent region (the one which contains it) and a sequence pointing to
-its children (regions it contains).  The \ippdbtable{SkyTable} enables
-fast lookups of the on-disk partitions which map to a specific
-coordinate on the sky.  In general, a single DVO will have the full
-sky represented with tables at a single level, although it is possible
-for mixed levels to be used.  For the PV3 master database, the
-partitioning is at Level 4, resulting in \approx 150,000 regions to
-cover the full sky, of which \approx 110,000 are used for the PV3
-$3\pi$ data.  The densest portions of the bulge contain at most
-\approx 300,000 astronomical objects in the database files, with an
-associated maximum of \approx 30 million measurements in these files.
-With the compression scheme described below, the largest database
-files are \approx 3GB, which can be loaded into memory in 30 seconds
-on the processing machines that contain partition data.
+by the HST Guide Star Catalog
+\citep[GSC][]{1988IAUS..133..239J,1990AJ.....99.2019L}.  Level 2
+subdivides these declination bands in the RA direction, with spacing
+related to the stellar density.  Level 3 divides these RA chunks into
+4 - 8 smaller partitions.  This level exactly matches the HST GSC
+layout, and uses the same naming convention to identify the
+partitions: \code{n0000/0000}, etc. Level 4 further divides these
+regions by a factor of 16.  In the \ippdbtable{SkyTable}, a region at
+one level has a pointer to its parent region (the one which contains
+it) and a sequence pointing to its children (regions it contains).
+The \ippdbtable{SkyTable} enables fast lookups of the on-disk
+partitions which map to a specific coordinate on the sky.  In general,
+a single DVO will have the full sky represented with tables at a
+single level, although it is possible for mixed levels to be used.
+For the PV3 master database, the partitioning is at Level 4, resulting
+in \approx 150,000 regions to cover the full sky, of which \approx
+110,000 are used for the PV3 $3\pi$ data.  The densest portions of the
+bulge contain at most \approx 300,000 astronomical objects in the
+database files, with an associated maximum of \approx 30 million
+measurements in these files.  With the compression scheme described
+below, the largest database files are \approx 3GB, which can be loaded
+into memory in 30 seconds on the processing machines that contain
+partition data.
 
 % parallel partitions
@@ -1559 +1560 @@
 reasonable timescale.
 
+\subsubsection{Object and Measurement IDs}
+
+Within the DVO system, certain integer fields are used to provide unique
+identifiers for measurements and objects.  The original implementation
+of DVO was limited to 32-bit integer fields, but since the maximum
+number of objects and measurements was expected to be larger than
+$2^{32}$, two 32-bit integer fields are joined together to make
+sufficiently large IDs.
+
+In the table of objects (\ippdbtable{Average}), the fields
+\ippdbcolumn{objID} and \ippdbcolumn{catID} together form a unique
+64-bit integer value to identify the objects.  The \ippdbcolumn{catID}
+field is a sequence number for the sky partition table (the
+`catalog') in which the object is contained, while \ippdbcolumn{objID}
+is an incrementing sequence number within that sky partition
+table.  As long as no sky partition tables contain more that
+$2^{32}$ objects, these fields will not overflow.  These two fields
+are included in the \ippdbtable{Measure}, \ippdbtable{GalPhot},
+\ippdbtable{StarPar}, \ippdbtable{Lensing}, and \ippdbtable{LensObj}
+tables to link the entries in those tables back their corresponding
+object.  Note that \ippdbtable{SecFilt} does {\em not} contain these
+ID fields; the rows in this table are maintained in the correct
+sequence to match the \ippdbtable{Average} table entries.
+
+The \ippdbtable{Measure} table, containing the detections of objects
+from individual exposures or stack, or the (potentially
+non-signficant) measurements from a warp, uses the 32-bit integer
+fields \ippdbcolumn{detID} and \ippdbcolumn{imageID} to uniquely
+identify each entry.  The \ippdbcolumn{imageID} is the running
+sequence number of the ``image'' (GPC1 OTA, stack, warp, or other
+other source of the measurement) in which the object was measured.
+The \ippdbcolumn{imageID} is a value internal to DVO, and is unique
+across all types of images.  The \ippdbcolumn{detID} field is a 32-bit
+integer giving the sequence number of the detection within the image.
+For images processed by the IPP (e.g., using \ippprog{psphot}), the
+\ippdbcolumn{detID} corresponds to the output field labeled as
+\ippmisc{IPP_IDET} in those data products.  Since measurements from
+the same image may be spread across multiple sky partition tables,
+both \ippdbcolumn{detID} and \ippdbcolumn{imageID} much be used to
+uniquely identify a detection within the database.  
+
+In the \ippdbtable{Measure} table, the field \ippdbcolumn{averef}
+specifies the row number in the \ippdbtable{Average} table of the
+associated object.  The \ippdbtable{Measure} table may be unsorted, in
+which case it is slow to find the measurements associated with a
+specific object (a full table scan is required, referencing
+\ippdbcolumn{objID}).  After the table is sorted and indexed, the
+\ippdbcolumn{Measure} rows for a given object are grouped together.
+In this case, the fields \ippdbtable{Average}.\ippdbcolumn{measureOffset} and
+\ippdbcolumn{Average}.\ippdbcolumn{Nmeasure} define an index for the
+code to jump to the list of measurements for a single object.  The
+field \ippdbtable{Measure}.\ippdbcolumn{imageID} defines the link from
+the measurement to the image which supplied the measurement.
+
+DVO is also used to construct the unique object and detection IDs used
+by the Published Science Products Subsystem (PSPS).  Within the PSPS,
+the field named \ippdbcolumn{objID} in that database is used to
+allows valid joins between tables to select the different kinds of
+attributes of the same astronomical objects.  This 64-bit integer ID
+is constructed based on the coordinates of the object, as described by
+\cite[][]{flewelling2017}.  In short, the digits of the right
+ascension and declination coordinates are used to define a single
+64-bit integer with spatial resolution of roughly 3 milliarcseconds.
+This values used by this field are generated by the DVO system and
+stored in the \ippdbtable{Average} table in the field
+\ippdbcolumn{extID}.  
+
+Within the PSPS, the \ippdbtable{Detection} table carries an ID which
+is unique for each measurement, equivalent to the DVO
+\ippdbcolumn{det_id}, \ippdbcolumn{image_id} pair.  In this case, the
+PSPS \ippdbcolumn{detectID} is constructed from the MJD of the
+exposure, the number of the OTA (e.g., OTA64), and the detection
+sequence within the image to form a single unique 64-bit integer value.
+For detections from the stack images, the MJD is not unique, so a
+different rubrick is used to define IDs for those detections.  The
+field \ippdbcolumn{XstackDetectID} (where '\ippdbcolumn{X}' is one of
+g,r,i,z,y) is constructed from the GPC1 stack ID
+(\ippdbtable{stackRun.stack_id}), the detection sequence within the
+stack image, and the same value used to define \ippdbcolumn{sourceID}
+above.  These two types of detection IDs are generated by the program
+\ippprog{addstar} when the images and stacks are ingested into DVO.
+
 \subsubsection{DVO Data Storage}
 
@@ -1567 +1650 @@
 of files for tables which are spatially partitioned.  The binary FITS
 tables are compressed using the (to date) experimental FITS binary
-table compression strategy outlined by \citet{RickWhite}.  Table compression
+table compression strategy outlined by \citet{2012arXiv1201.1340P}.  Table compression
 is an option in DVO; for the PV3 database, the large data
 volume (70TB compressed) drove the decision to compress the tables.
@@ -1573 +1656 @@
 % FITS table compression details
 The FITS binary table compression scheme uses a strategy similar to
-that used for FITS image compression (\note{REF}).  The binary tabular
-data is compressed and stored in the ``HEAP'' section of the FITS table
-extension, with pointers to the compressed data stored in the regular
-data section.  Each column in the FITS table is compressed as one (or
-more) blocks.  The standard header keywords which describe the data
-column format (e.g., TFORM1) are replaced with keywords which describe
-the location and size of the compressed data in the HEAP section; the
-information about the uncompressed data is moved to a keyword with ``Z''
-prepended (e.g., ZFORM1) and an additional field is added to define
-the compression algorithm (e.g., ZCTYP1).  The column names (e.g.,
-TTYPE1) and units (e.g., TUNIT1) are retained in their original form.
+that used for FITS image compression
+\citep[][]{1999ASPC..172..125W,2000ASPC..216..551P}.  The binary
+tabular data is compressed and stored in the ``HEAP'' section of the
+FITS table extension, with pointers to the compressed data stored in
+the regular data section.  Each column in the FITS table is compressed
+as one (or more) blocks.  The standard header keywords which describe
+the data column format (e.g., TFORM1) are replaced with keywords which
+describe the location and size of the compressed data in the HEAP
+section; the information about the uncompressed data is moved to a
+keyword with ``Z'' prepended (e.g., ZFORM1) and an additional field is
+added to define the compression algorithm (e.g., ZCTYP1).  The column
+names (e.g., TTYPE1) and units (e.g., TUNIT1) are retained in their
+original form.
 
 % FITS table compression details
@@ -1698 +1783 @@
 catalog files (``smf files'') and determined the zero points of those
 exposures which were believed to be obtained in photometric
-conditions.  This process, called ``\"ubercal'', is described in detail
-by \cite{2012ApJ...756..158S} for the first (PV1) version.  In brief, photometric
-periods, with time-scales of at least \note{half of a night}, are
-identified by a combination of automatic analysis and manual
-inspection.  A single solution for all images in a given filter is
-determined to minimize scatter for individual stars.  The free
+conditions.  This process, called ``\"ubercal'', is described in
+detail by \cite{2012ApJ...756..158S} for the first (PV1) version.  In
+brief, photometric periods, with time-scales of a large fraction of a
+night, are identified by a combination of automatic analysis and
+manual inspection.  A single solution for all images in a given filter
+is determined to minimize scatter for individual stars.  The free
 parameters in this solution consist of a single zero point and airmass
 slope for each photometric period along with a collection of
@@ -1709 +1794 @@
 seasons'').  For the PV3 \"ubercal analysis, the flat-field offsets
 were determined on a $2\times2$ grid for each chip and 5 flat-field
-seasons were chosen (listed in Table~\ref{tab:flat-field-seasons}).
-The boundaries of the flat-field seasons were determined by
-independent inspection of the residuals observed in the Medium Deep
-fields.
+seasons were identified.  The boundaries of the flat-field seasons
+were determined by independent inspection of the residuals observed in
+the Medium Deep fields.
+
+%%  (listed in Table~\ref{tab:flat-field-seasons}) XXX add this table
 
 After the \"ubercal analysis of the photometric periods is completed,
@@ -1741 +1827 @@
 Telescope Sciences Institute through their Mikulski Archive for Space
 Telescopes (MAST).  The underying database at MAST is a copy of a
-database generated at the IfA by the subsystem
-called PSPS : the \note{define PSPS}.  The construction of the PSPS
-version of the PS1 database starts once the PS1 photometry and
-astrometry measurements have been calibrated within the DVO system.
-The construction takes place in several stages, described in detail by
-\cite{flewelling2017}.  We summarize those steps here.
+database generated at the IfA by the Published Science Products
+Subsystem (PSPS).  The construction of the PSPS version of the PS1
+database starts once the PS1 photometry and astrometry measurements
+have been calibrated within the DVO system.  The construction takes
+place in several stages, described in detail by \cite{flewelling2017}.
+We summarize those steps here.
 
 The first stage of constructing the PSPS database consists of the
@@ -1824 +1910 @@
 collection of ``tasks'' which describe the concept of a command which
 might be run and to regularly generate new commands based on that
-concept.  The ``tasks'' are defined using the opihi scripting language
+concept.  The ``tasks'' are defined using the \ippprog{opihi} scripting language
 (also shared by DVO and other user-interactive programs within the
 IPP).
 
-\ippprog{Pantasks} repeatedly checks each task in an attempt to generate a new
-command: we say \ippprog{pantasks} attempts to ``execute'' the task in each of
-these attempts.  Tasks may specify the time between execution
+\ippprog{Pantasks} repeatedly checks each task in an attempt to
+generate a new command: we say \ippprog{pantasks} attempts to
+``execute'' the task.  Tasks may specify the time between execution
 attempts, with a 1 second default.
 
@@ -1839 +1925 @@
 executed.  A dynamic command is defined within a special block of the
 task, called \code{task.exec}.  This block is a snipet of code (in the
-opihi language) which is run each time the task is executed.  The
+\ippprog{opihi} language) which is run each time the task is executed.  The
 \code{task.exec} code may refer to variables or other data structures
-defined by the opihi language within the \ippprog{pantasks} environment.  Within
-a single \ippprog{pantasks} instance, all opihi variables and data
-structures have global context (\ie, all are visible to all tasks).
-Variables are by default global, but within the context of an opihi
-macro (equivalent of a function call), variables may be
-locally-scoped.  Other data structures (see below) are global and must
-be protected with name space choices.
-
-Within the \ippprog{task.exec} macro, the command to be run must be
-defined with the function ``command''.  Once the \ippprog{task.exec}
-macro exits successfully, the defined command is then added to the list of jobs
-to be run within the UNIX environment.  Jobs may be run in one of two
-ways: locally or via the parallel processing system.  The task, or the
-\ippprog{task.exec} macro, uses the ``host'; command to define how to
-run the job.  If the host is set to ``local'', then the job is run in
-the background by \ippprog{pantasks} itself (using the C \code{execvp}
-function).  Otherwise, the job is sent to the parallel processing
-system to be run on another machine within the cluster.  If the host
-is set to the special value ``anyhost'', then the parallel processing
-system is allowed to choose the processing computer arbitrarily.  Any
-other value is taken to be the DNS name of the computer on which this
-job should run.  If the option \code{-required} is supplied to the
-\code{host} command, then the parallel processing system must ensure
-that the job only runs on the specifically named computer.  Otherwise,
-the parallel processing system may choose to redirect the command to
-another computer using its own rules, e.g. to balance processing load
-across the cluster.
-
-When the \ippprog{task.exec} macro is run, the code may choose (e.g.,
+defined by the \ippprog{opihi} language within the \ippprog{pantasks}
+environment.  Within a single \ippprog{pantasks} instance, all \ippprog{opihi}
+variables and data structures have global context by default (\ie, all
+are visible to all tasks).  Within the context of an \ippprog{opihi} macro
+(equivalent of a function call), variables may be locally-scoped.
+Other data structures (see below) are global and must be protected
+with name space choices.
+
+Within the \code{task.exec} macro, the command to be run is defined by
+the script.  Once the \code{task.exec} macro exits successfully, the
+defined command is then added to the list of jobs to be run within the
+UNIX environment.  Jobs may be run in one of two ways: locally or via
+the parallel processing system.  The task, or the \code{task.exec}
+macro, uses the \code{host} command to define how to run the job.  If
+the host is set to ``local'', then the job is run in the background by
+\ippprog{pantasks} itself (using the C \code{execvp} function).
+Otherwise, the job is sent to the parallel processing system to be run
+on another machine within the cluster.  If the host is set to the
+special value ``anyhost'', then the parallel processing system is
+allowed to choose the processing computer arbitrarily.  Any other
+value is taken to be the DNS name of the computer on which this job
+should run.  The host may (optionally) be required for the command, in
+which case the parallel processing system must ensure that the job
+only runs on the specifically-named computer.  Otherwise, the parallel
+processing system may choose to redirect the command to another
+computer using its own rules, e.g. to balance processing load across
+the cluster.
+
+When the \code{task.exec} macro is run, the code may choose (e.g.,
 based on tests of some global variables) to exit the macro with an
-error condition, e.g., with the ``break'' command.  In this
-circumstance, no job is produced by the task.  The task will be tried
-again the next time it is executed.  This feature allows for the user
-to set processing blocks which depend on some external tests.  For
-example, some task may check external network connectivity and set a
-variable based on the network status; other tasks may then choose to
-wait until the network is available before attempting to run.
-
-Other task options discussed below exist to control the system
-behavior in detail.  Note that the options below may be dynamically
-reset by the \ippprog{task.exec} macro.   
-
-\note{this section probably has too much detail; move this into an
-  online user guide?}
-
-The option ``npending'' may be used to limit the number of jobs which
-are simultaneously executed for a specific task.  For example, some
-classes of jobs should only be run one-at-a-time because they are not
-protected against collisions or they may overload a resource.  The use
-of ``npending'' allows these situations to be handled cleanly within
-\ippprog{pantasks} (avoiding cumbersome coding within with program or supporting
-script).
-
-The option ``nmax'' limits the total number of jobs which a task
-generates.  This option may be useful in cases where
-\ippprog{pantasks} is used to perform a limited set of operations.
-\note{do we actually use this in IPP?}
-
-The option ``trange'' allows the user to restrict the time period during
-which the specific tasks is executed.  This option is given with a
-start and an end time for the limiting time range.  These times may be
-of one of several forms: ``HH:MM:SS'' specifies a time within a day
-(in UT or local time?).  ``Day[@HH:MM:SS]'' specifies a time on a
-specific day, e.g., \code{trange Mon@13:00 Tue@09:00} says the task
-should be run from 1pm on Mondays to 9am on Tuesdays.  ``YYYY/MM/DD,HH:MM:SS''
-specifies a time on a specific date within the year.  The start and
-end times must be of the same class.  The \code{trange} command has
-some optional arguments as well.  The option \code{-nmax NNN} defines
-the maximum number of jobs which may be run in that time range.  The
-option \code{-exclude} specifies that the time range is a period when
-the task should {\em not} be executed.  An arbirary number of time
-ranges may be specified \note{how are they evaluated?}
-
-The option \code{nice} specifies the ``nice'' level at which the job is
-run when it is executed.  The parallel processing system must respect
-this concept.
-
-The option \code{active} can be used to turn on and off a task for
-periods.  Since a user command or a macro run by \ippprog{pantasks} can
-re-define task options, the \code{active} state may be changed
-independently of the task execute.  This is useful for keeping tasks
-defined by a \ippprog{pantasks} instance, but allowing the user to
-prevent them from running for some reason.
+error condition.  In this circumstance, no job is produced by the
+task.  The task will be tried again the next time it is executed.
+This feature allows for the user to set processing blocks which depend
+on some external tests.  For example, some task may check external
+network connectivity and set a variable based on the network status;
+other tasks may then choose to wait until the network is available
+before attempting to run.
+
+Other task options exist to control the system behavior in detail.
+These options may be dynamically reset by the \code{task.exec} macro.
+Some options control the number of jobs, such as limiting the number
+of currently-outsanding jobs for a given task, or limiting the total
+number generated.  Other options can be used to control the time when
+jobs of a certain task are allowed to run.  It is also possible to
+specify the UNIX ``nice'' level at which the job is
+run when it is executed.  Finally, individual tasks may be disabled
+while the system is still running.
 
 \subsubsection{pcontrol}
@@ -1931 +1981 @@
 across many machines in the computing cluster.  The parallel
 processing system used by \ippprog{pantasks} is an independent
-software system.  The default parallel processing system is a program
-called \ippprog{pcontrol}\footnote{Alternatives are possible: e.g.,
-  {\tt condor} has been experimentally integrated with
+software system called \ippprog{pcontrol}\footnote{Alternatives are
+  possible: e.g., {\tt condor} has been experimentally integrated with
   \ippprog{pantasks} for tests}.
 
@@ -1946 +1995 @@
 one of several states: pending (ready to run), running (jobs which are
 running), exit (job has completed), busy (job is being checked by
-\ippprog{pcontrol}), crash (job has exited with a signal(?), normally
-segv).
+\ippprog{pcontrol}), crash (job has exited with a signal, normally
+\code{segv}).
 
 Similarly, the hosts may also have one of several states: off, down,
@@ -2001 +2050 @@
 
 The \ippprog{pantasks} program can be run as a stand-alone program
-which presents an opihi shell interface to the user when it is
+which presents an \ippprog{opihi} shell interface to the user when it is
 started.  This mode is useful for testing as all errors are reported
-back to the opihi shell.  However, when the user exits the shell, the
+back to the \ippprog{opihi} shell.  However, when the user exits the shell, the
 \ippprog{pantasks} instance exits, shutting down \ippprog{pcontrol} and all remote client
 connections.  In standard operations, \ippprog{pantasks} is run in a client
@@ -2025 +2074 @@
 \end{verbatim}
 \caption{\label{fig:task_example} Example of a simple static
-  task in the opihi-based scripting language used by ippprog{pantasks}.  In
-  this example, ippprog{pantasks} would run a single instance of the command
+  task in the opihi-based scripting language used by pantasks.  In
+  this example, pantasks would run a single instance of the command
   ({\tt ls /tmp}) every 5 seconds, sending the stdout and stderr to
   the listed files. }
@@ -2036 +2085 @@
 \subsubsection{Pantasks scripts: ippTasks}
 
-\ippprog{Pantasks} provides an environment in which commands can be generated
-and extensive parallel processing managed.  The details of how to
-implement the different stages of IPP processing are captured in a
-collection of scripts written for \ippprog{pantasks} in the \code{opihi}
-language.  In general, each stage is defined by an associated script
-collected together under the \ippmisc{ippTasks} collection.  While
-each script has its own details, there are a number of common
-elements.
+\ippprog{Pantasks} provides an environment in which commands can be
+generated and extensive parallel processing managed.  The details of
+how to implement the different stages of IPP processing are captured
+in a collection of scripts written for \ippprog{pantasks} in the
+\ippprog{opihi} language.  In general, each stage is defined by an
+associated script collected together under the \ippmisc{ippTasks}
+collection.  While each script has its own details, there are a number
+of common elements.
 
 Most stages consist of two related tasks: a \ippmisc{load} task, which
@@ -2059 +2108 @@
 job is permitted to run simultaneously, preventing race conditions.
 
-The results from the database query job are stored in an opihi data
+The results from the database query job are stored in an \ippprog{opihi} data
 structure called a \ippmisc{book} within the \ippprog{pantasks}
 environment.  Each row in the result set is saved to a separate entry
@@ -2077 +2126 @@
 \ippmisc{book} for any pages with \ippdbcolumn{pantasksState} set to
 \ippmisc{DONE}, and removes them from the book, as these represent
-jobs that have finished. \note{the manipulation above takes place in
-  the task.exit subscript}
+jobs that have finished.
+
+% \note{the manipulation above takes place in the task.exit subscript}
 
 The associated \ippmisc{run} task generates jobs constructed from the
@@ -2099 +2149 @@
 program to do the data analysis work and a supporting Perl script
 which performs the database update upon completion.  Upon completion,
-the \ippprog{pantasks} \ippmisc{RUN} tasks is responsible for updating the
+the \ippprog{pantasks} \ippmisc{RUN} task is responsible for updating the
 status within the book, but not within the processing database.  This
 split keeps the interactions at the \ippprog{pantasks} level relatively light,
@@ -2147 +2197 @@
 \label{sec:automation}
 
-Outside of the basic sequence of \ippstage{chip} to \ippstage{warp}, there is no single
-natural next step.  For example: a stack can be generated with any
-number of input warps; a difference image can be generated between a
-warp and any one of many other warps or stacks.  Without a single
-sequence, more complex and sophisticated decisions much be made.
+Beyond of the basic sequence of \ippstage{chip} to \ippstage{warp},
+there is no single natural ``next step''.  For example: a stack can be
+generated with any number of input warps; a difference image can be
+generated between a warp and any one of many other warps or stacks.
+Without a single sequence, more complex and sophisticated decisions
+much be made.
 
 For nightly processing of data obtained at the summit, this is handled
@@ -2252 +2303 @@
 generated by the GPC1 camera.  The \ippprog{Nebulous} system was
 designed to aid in thie process.  \ippprog{Nebulous} is not a file
-system per-se, but only method of tracking the locations of files
+system per-se, but only a method of tracking the locations of files
 within the file system, and of tracking duplicate copies of the same
 file.  The core of \ippprog{Nebulous} is a mysql database which tracks
@@ -2276 +2327 @@
 All of the analysis stages which interact with that chip could then be
 preferentially targeted to be run on that computer.  The localization
-in \ippprog{Nebulous} and the host targeted processing in \ippprog{pantasks}
-can therefore work together to encourage processing to require only
-local disk access, reducing the I/O local on the network
-infrastructure.  In the early stages of the Pan-STARRS project, this
-was important because network bandwidth was an expensive resource.  In
-practice, the as-built IPP has had sufficient network bandwidth that
-this targetting was not required.  In practice, due to the timing of
-hardware acquisition, occasional hardware failures, and other
-organizational details, targeted processing has only been used to a
-moderate degree within the Pan-STARRS cluster. 
+in \ippprog{Nebulous} and the host targeted processing in
+\ippprog{pantasks} can therefore work together to encourage processing
+to require only local disk access, reducing the I/O local on the
+network infrastructure.  In the early stages of the Pan-STARRS
+project, this was important because network bandwidth was an expensive
+resource.  In practice, the as-built IPP has had sufficient network
+bandwidth that this targetting was not completely required.  In
+practice, due to the timing of hardware acquisition, occasional
+hardware failures, and other organizational details, targeted
+processing has only been used to a moderate degree within the
+Pan-STARRS cluster.
 
 \subsubsection{Implementation Details}
 
 The user interfaces to Nebulous consist of command-line programs as
-well as APIs in both C and Perl.  
-
-The basic user commands to interact with Nebulous are to 1) query the
-database for an existing storage object, and find a valid file
-instance associated with that object; 2) create a new storage object,
-which instantiates an empty file that can be opened for writing; 3)
-replicate an existing storage object to create more file instances; 4)
-cull a single file instance of storage object from the cluster; and 5)
-remove a storage object, and ensure that all file instances are
-removed.  The filehandles returned for newly created instances can
-then be opened for reading and writing data to that instance.
+well as APIs in both C and Perl.  The basic user commands to interact
+with Nebulous are to 1) query the database for an existing storage
+object, and find a valid file instance associated with that object; 2)
+create a new storage object, which instantiates an empty file that can
+be opened for writing; 3) replicate an existing storage object to
+create more file instances; 4) cull a single file instance of storage
+object from the cluster; and 5) remove a storage object, and ensure
+that all file instances are removed.  The filehandles returned for
+newly created instances can then be opened for reading and writing
+data to that instance.
 
 % The basic user commands to interact
@@ -2429 +2480 @@
 Requests to this server may restrict to the latest by time.  Each row
 in the listing includes basic information about the exposure: an
-exposure identifier (e.g., o5432g0123o; see~\ref{GPC1.names} for
-details), the date and time of the exposure, the telescope commanded
-pointing, the filter and exposure time, and the observation comment
-for that exposure.  The row also provides a link to a listing of the
-chips associated with that exposure.  This listing includes a link to
-the individual chip FITS files as well as an md5 checksum.  Systems
-which are allowed access may download the raw chip FITS files via http
-requests to the provided links.
+exposure identifier \citep[e.g., o5432g0123o; see][for
+  details]{chambers2017}, the date and time of the exposure, the
+telescope commanded pointing, the filter and exposure time, and the
+observation comment for that exposure.  The row also provides a link
+to a listing of the chips associated with that exposure.  This listing
+includes a link to the individual chip FITS files as well as an md5
+checksum.  Systems which are allowed access may download the raw chip
+FITS files via http requests to the provided links.
 
 % \note{add a discussion of gpc1 filenames?}