Changeset 40002 for trunk/doc/release.2015/ps1.datasystem/datasystem.tex

Timestamp:

Mar 16, 2017, 9:41:23 AM (9 years ago)

Author:

eugene

Message:

moving around the sections : group the operations and automation elements together

File:

• trunk/doc/release.2015/ps1.datasystem/datasystem.tex (modified) (4 diffs)

trunk/doc/release.2015/ps1.datasystem/datasystem.tex

@@ -1026 +1026 @@
 the rest of the pipeline.
 
-\subsection{Nebulous}
-\label{subsec: nebulous}
-Storing the large volume of data that is generated by the GPC1 camera
-was recognized early in the Pan-STARRS project as a major concern.
-The \ippprog{Nebulous} system was designed to organize this data.  The
-main components of this system is a database storing the locations of
-the files, with a Simple Object Access Protocol interface between the
-database and the other IPP programs.  The actual files are stored on
-NFS mounted partitions on a series of storage nodes in the IPP cluster
-that can be accessed throughout the cluster.  This distribution of
-files is useful to balance the disk I/O, as this parallelizes the
-load.
-
-The original design of \ippprog{Nebulous} was intended to aid in the
-targetted processing of data, by having specific image data (such as
-all the images from one OTA device) located on a single storage node.
-This would allow any jobs involving that data to be assigned to the
-storage node, eliminating network I/O.  Important data could be
-duplicated to additional data nodes, with the alternate locations
-stored in the database.  In practice, however, hardware failures and
-increases in hard drive storage volumes and network bandwidth have
-reduced the degree to which the IPP processing is targeted.
-
-When a program creates a new file in \ippprog{Nebulous}, it supplies
-an URI of the form \code{neb://HOST.VOLUME/PATH/FILENAME}.  The host
-and volume specifiers are optional, and allow a file to be created on
-a specific node.  The path and filename appear similar to a standard
-full file location, and are used internally as the ``external id''.  A
-storage object entry is then created in the database for this id, and
-an instance of the file created on the specified node (or at random
-from available nodes if left empty).  This instance is created in a
-deterministic filename location.  The external id is hashed using the
-SHA-1 function, and the first four hexadecimal digits of this hash are
-separated into two two-digit strings and used as the top and second
-level directory location for the disk file.  The disk file is created
-using the database instance id, and a transformed version of the
-external id, which has colons replacing any forward slash characters,
-to convert the external id path into a filename.  For the example URI
-above, this results in a file located on disk in a location like
-\code{/data/HOST.VOLUME/nebulous/d5/d8/9244993440.PATH:FILENAME}.
-This file naming structure is useful, as it duplicates database
-contents on disk.
-
-The storage volumes that contain the data on disk are defined in the
-\ippprog{Nebulous} database in a number of ways.  First, the locations
-and mount points for the actual NFS storage are listed in the
-\ippdbtable{volume} table.  This table contains columns indicating if
-the volume should be used for reading (\ippdbcolumn{available}) and
-writing (\ippdbcolumn{allocate}), allowing these properties to be
-manually set, which is useful in scheduling downtime for hardware
-issues.  Another column, \ippdbcolumn{xattr}, is used to control the
-behavior of this volume, with specific values used to denote desired
-behavior.  For instance, the value of $1$ is used to indicate that a
-volume should only be used as a backup volume (which accepts only
-replicated copies), and the value of $5$ is used to indicate that the
-volume is permanently unavailable, and should be ignored.
-
-In addition to this permanent table describing the volumes, a
-\ippdbtable{mountedvol} table also exists that only lists those
-volumes that are currently visible and accessible from the
-\ippprog{Nebulous} database server.  This table also lists the total
-and currently available disk space on each volume, allowing the
-\ippprog{Nebulous} load balancing routines to prioritize those volumes
-with large unused disk space before filling the volumes with only
-small amounts remaining.  This table is regenerated every ten to
-twenty minutes, after a scan of each of the volumes listed in the
-\ippdbtable{volume} table.
-
-The final table controlling the operations of the \ippprog{Nebulous}
-volumes is the \ippdbtable{cabinet} table, which organizes the
-individual volumes into ``cabinets,'' a concept loosly based on the
-physical arrangement of the storage servers in the data center.  These
-cabinets are used to prevent the replication of a storage object
-within a group of volumes where all instances of the object could be
-taken off line by a single failure.  As the data center cabinets share
-power supplies among the servers they contain, ensuring physical
-distance between replicated copies is important to guarantee that a
-temporary failure of one of these devices does not significantly
-impact processing.
+\section{Post-Processing : Database Ingest \& Calibration}
 
 \subsection{DVO}
@@ -1403 +1325 @@
 analysis \citep[][see]{magnier2017a}.
 
-\subsection{Datastore repositories}
-\label{subsec: datastore}
-
-Transferring data between the IPP and other parts of the Pan-STARRS
-system is generally accomplished via a ``datastore'', an http service
-that exposes data in a common form.  One of the main datastores used
-by the IPP is the one located at the summit.  This datastore exposes,
-a list of the exposures obtained since the start of the PS1
-operations.  Requests to this server may restrict to the latest by
-time.  Each row in the listing includes basic information about the
-exposure: an exposure identifier (e.g., o5432g0123o;
-see~\ref{GPC1.names} for details), the date and time of the exposure,
-the telescope commanded pointing, the filter and exposure time, and
-the observation comment for that exposure.  The row also provides a
-link to a listing of the chips associated with that exposure.  This
-listing includes a link to the individual chip FITS files as well as
-an md5 checksum.  Systems which are allowed access may download chip
-FITS files via http requests to the provided links.
-
-The IPP also uses datastores to provide access to its own data
-products.  The detections identified in the \ippstage{diff} stage
-images are organized by the \ippstage{publish} stage, which writes
-output files containing those detections to a datastore that is
-monitored by the Moving Object Processing System
-\citep[][MOPS]{2013PASP..125..357D}, which analyses the detections to
-identify asteroids.  Separate datastores are also used by the
-\ippstage{distribution} stage to provide access to data products to
-the Pan-STARRS Science Consortium members.  
+\subsection{Addstar : DVO Ingest}
+\label{subsec: addstar}
+\note{CZW: This should be reviewed.}
+
+Upon completion of the processing of each stage, the results of the
+photometry analysis are isolated in a large number of individual
+catalogs, with little connection between the separate measurements of
+astronomical sources.  Unifying these measurements in a DVO database
+is the purpose of the \ippstage{addstar} processing.  The catalogs for
+the \ippstage{camera}, \ippstage{staticsky}, \ippstage{skycal},
+\ippstage{fullforce}, and \ippstage{diff} are processed in this
+fashion, although not every measurement in each catalog are included
+in the final DVO that is constructed.
+
+The construction of this final DVO is performed in a hierarchical
+process.  The individual catalogs are added to a \ippmisc{minidvo},
+which is simply a DVO database defined over some subset of possible
+inputs.  These \ippmisc{minidvos} are then merged into larger
+databases to construct the final completely catalog.  \note{describe
+  database tables}
+
+Each catalog that is to be added to DVO has an entry created in the
+\ippdbtable{addRun} database table.  This entry notes which
+\ippdbcolumn{stage} is the source of the catalog, and links to the
+appropriate database table with the \ippdbcolumn{stage\_id} field.  As
+some stages, such as the \ippstage{diff} stage, create more than a
+single catalog, multiple entries with the \ippdbcolumn{stage\_id} are
+created, with the \ippdbcolumn{stage\_extra1} field containing an
+index to the individual components.  The catalog specified by the
+entry is added to the target \ippmisc{minidvo} by the
+\ippprog{addstar} program, \note{describe what's done?}.  When this
+completes, an entry containing the statistics of the job is added to
+the \ippdbtable{addProcessedExp} table.
+
+\subsection{Calibration Operations}
+\label{subsec: calibration}
+
+\subsection{IPP to PSPS}
+\label{subsec: ipp2psps}
+\note{Default to pointing to Flewelling et al 2017?}
+
+\subsection{PSPS Load \& Merge}
+\label{subsec: psps}
+\note{Default as well to pointing to Flewelling et al 2017?}
+
+\section{Operations \& Automation}
 
 \subsection{Pantasks \& Parallel Processing}
@@ -1592 +1532 @@
 \ippdbcolumn{projection\_cell}.
 
+\subsection{Nebulous}
+\label{subsec: nebulous}
+Storing the large volume of data that is generated by the GPC1 camera
+was recognized early in the Pan-STARRS project as a major concern.
+The \ippprog{Nebulous} system was designed to organize this data.  The
+main components of this system is a database storing the locations of
+the files, with a Simple Object Access Protocol interface between the
+database and the other IPP programs.  The actual files are stored on
+NFS mounted partitions on a series of storage nodes in the IPP cluster
+that can be accessed throughout the cluster.  This distribution of
+files is useful to balance the disk I/O, as this parallelizes the
+load.
+
+The original design of \ippprog{Nebulous} was intended to aid in the
+targetted processing of data, by having specific image data (such as
+all the images from one OTA device) located on a single storage node.
+This would allow any jobs involving that data to be assigned to the
+storage node, eliminating network I/O.  Important data could be
+duplicated to additional data nodes, with the alternate locations
+stored in the database.  In practice, however, hardware failures and
+increases in hard drive storage volumes and network bandwidth have
+reduced the degree to which the IPP processing is targeted.
+
+When a program creates a new file in \ippprog{Nebulous}, it supplies
+an URI of the form \code{neb://HOST.VOLUME/PATH/FILENAME}.  The host
+and volume specifiers are optional, and allow a file to be created on
+a specific node.  The path and filename appear similar to a standard
+full file location, and are used internally as the ``external id''.  A
+storage object entry is then created in the database for this id, and
+an instance of the file created on the specified node (or at random
+from available nodes if left empty).  This instance is created in a
+deterministic filename location.  The external id is hashed using the
+SHA-1 function, and the first four hexadecimal digits of this hash are
+separated into two two-digit strings and used as the top and second
+level directory location for the disk file.  The disk file is created
+using the database instance id, and a transformed version of the
+external id, which has colons replacing any forward slash characters,
+to convert the external id path into a filename.  For the example URI
+above, this results in a file located on disk in a location like
+\code{/data/HOST.VOLUME/nebulous/d5/d8/9244993440.PATH:FILENAME}.
+This file naming structure is useful, as it duplicates database
+contents on disk.
+
+The storage volumes that contain the data on disk are defined in the
+\ippprog{Nebulous} database in a number of ways.  First, the locations
+and mount points for the actual NFS storage are listed in the
+\ippdbtable{volume} table.  This table contains columns indicating if
+the volume should be used for reading (\ippdbcolumn{available}) and
+writing (\ippdbcolumn{allocate}), allowing these properties to be
+manually set, which is useful in scheduling downtime for hardware
+issues.  Another column, \ippdbcolumn{xattr}, is used to control the
+behavior of this volume, with specific values used to denote desired
+behavior.  For instance, the value of $1$ is used to indicate that a
+volume should only be used as a backup volume (which accepts only
+replicated copies), and the value of $5$ is used to indicate that the
+volume is permanently unavailable, and should be ignored.
+
+In addition to this permanent table describing the volumes, a
+\ippdbtable{mountedvol} table also exists that only lists those
+volumes that are currently visible and accessible from the
+\ippprog{Nebulous} database server.  This table also lists the total
+and currently available disk space on each volume, allowing the
+\ippprog{Nebulous} load balancing routines to prioritize those volumes
+with large unused disk space before filling the volumes with only
+small amounts remaining.  This table is regenerated every ten to
+twenty minutes, after a scan of each of the volumes listed in the
+\ippdbtable{volume} table.
+
+The final table controlling the operations of the \ippprog{Nebulous}
+volumes is the \ippdbtable{cabinet} table, which organizes the
+individual volumes into ``cabinets,'' a concept loosly based on the
+physical arrangement of the storage servers in the data center.  These
+cabinets are used to prevent the replication of a storage object
+within a group of volumes where all instances of the object could be
+taken off line by a single failure.  As the data center cabinets share
+power supplies among the servers they contain, ensuring physical
+distance between replicated copies is important to guarantee that a
+temporary failure of one of these devices does not significantly
+impact processing.
+
+\subsection{Datastore repositories}
+\label{subsec: datastore}
+
+Transferring data between the IPP and other parts of the Pan-STARRS
+system is generally accomplished via a ``datastore'', an http service
+that exposes data in a common form.  One of the main datastores used
+by the IPP is the one located at the summit.  This datastore exposes,
+a list of the exposures obtained since the start of the PS1
+operations.  Requests to this server may restrict to the latest by
+time.  Each row in the listing includes basic information about the
+exposure: an exposure identifier (e.g., o5432g0123o;
+see~\ref{GPC1.names} for details), the date and time of the exposure,
+the telescope commanded pointing, the filter and exposure time, and
+the observation comment for that exposure.  The row also provides a
+link to a listing of the chips associated with that exposure.  This
+listing includes a link to the individual chip FITS files as well as
+an md5 checksum.  Systems which are allowed access may download chip
+FITS files via http requests to the provided links.
+
+The IPP also uses datastores to provide access to its own data
+products.  The detections identified in the \ippstage{diff} stage
+images are organized by the \ippstage{publish} stage, which writes
+output files containing those detections to a datastore that is
+monitored by the Moving Object Processing System
+\citep[][MOPS]{2013PASP..125..357D}, which analyses the detections to
+identify asteroids.  Separate datastores are also used by the
+\ippstage{distribution} stage to provide access to data products to
+the Pan-STARRS Science Consortium members.  
+
 \subsection{ippTools and ippScripts}
 \label{subsec: ipptools}
@@ -1693 +1742 @@
 
 \note{This likely needs cleaning up and more information.}
-
-\subsection{Addstar : DVO Ingest}
-\label{subsec: addstar}
-\note{CZW: This should be reviewed.}
-
-Upon completion of the processing of each stage, the results of the
-photometry analysis are isolated in a large number of individual
-catalogs, with little connection between the separate measurements of
-astronomical sources.  Unifying these measurements in a DVO database
-is the purpose of the \ippstage{addstar} processing.  The catalogs for
-the \ippstage{camera}, \ippstage{staticsky}, \ippstage{skycal},
-\ippstage{fullforce}, and \ippstage{diff} are processed in this
-fashion, although not every measurement in each catalog are included
-in the final DVO that is constructed.
-
-The construction of this final DVO is performed in a hierarchical
-process.  The individual catalogs are added to a \ippmisc{minidvo},
-which is simply a DVO database defined over some subset of possible
-inputs.  These \ippmisc{minidvos} are then merged into larger
-databases to construct the final completely catalog.  \note{describe
-  database tables}
-
-Each catalog that is to be added to DVO has an entry created in the
-\ippdbtable{addRun} database table.  This entry notes which
-\ippdbcolumn{stage} is the source of the catalog, and links to the
-appropriate database table with the \ippdbcolumn{stage\_id} field.  As
-some stages, such as the \ippstage{diff} stage, create more than a
-single catalog, multiple entries with the \ippdbcolumn{stage\_id} are
-created, with the \ippdbcolumn{stage\_extra1} field containing an
-index to the individual components.  The catalog specified by the
-entry is added to the target \ippmisc{minidvo} by the
-\ippprog{addstar} program, \note{describe what's done?}.  When this
-completes, an entry containing the statistics of the job is added to
-the \ippdbtable{addProcessedExp} table.
-
-\subsection{Calibration Operations}
-\label{subsec: calibration}
-
-\subsection{IPP to PSPS}
-\label{subsec: ipp2psps}
-\note{Default to pointing to Flewelling et al 2017?}
-
-\subsection{PSPS Load \& Merge}
-\label{subsec: psps}
-\note{Default as well to pointing to Flewelling et al 2017?}
 
 \section{IPP Hardware Systems}