Index: trunk/doc/release.2015/ps1.datasystem/datasystem.tex =================================================================== --- trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 40001) +++ trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 40002) @@ -1026,83 +1026,5 @@ the rest of the pipeline. -\subsection{Nebulous} -\label{subsec: nebulous} -Storing the large volume of data that is generated by the GPC1 camera -was recognized early in the Pan-STARRS project as a major concern. -The \ippprog{Nebulous} system was designed to organize this data. The -main components of this system is a database storing the locations of -the files, with a Simple Object Access Protocol interface between the -database and the other IPP programs. The actual files are stored on -NFS mounted partitions on a series of storage nodes in the IPP cluster -that can be accessed throughout the cluster. This distribution of -files is useful to balance the disk I/O, as this parallelizes the -load. - -The original design of \ippprog{Nebulous} was intended to aid in the -targetted processing of data, by having specific image data (such as -all the images from one OTA device) located on a single storage node. -This would allow any jobs involving that data to be assigned to the -storage node, eliminating network I/O. Important data could be -duplicated to additional data nodes, with the alternate locations -stored in the database. In practice, however, hardware failures and -increases in hard drive storage volumes and network bandwidth have -reduced the degree to which the IPP processing is targeted. - -When a program creates a new file in \ippprog{Nebulous}, it supplies -an URI of the form \code{neb://HOST.VOLUME/PATH/FILENAME}. The host -and volume specifiers are optional, and allow a file to be created on -a specific node. The path and filename appear similar to a standard -full file location, and are used internally as the ``external id''. A -storage object entry is then created in the database for this id, and -an instance of the file created on the specified node (or at random -from available nodes if left empty). This instance is created in a -deterministic filename location. The external id is hashed using the -SHA-1 function, and the first four hexadecimal digits of this hash are -separated into two two-digit strings and used as the top and second -level directory location for the disk file. The disk file is created -using the database instance id, and a transformed version of the -external id, which has colons replacing any forward slash characters, -to convert the external id path into a filename. For the example URI -above, this results in a file located on disk in a location like -\code{/data/HOST.VOLUME/nebulous/d5/d8/9244993440.PATH:FILENAME}. -This file naming structure is useful, as it duplicates database -contents on disk. - -The storage volumes that contain the data on disk are defined in the -\ippprog{Nebulous} database in a number of ways. First, the locations -and mount points for the actual NFS storage are listed in the -\ippdbtable{volume} table. This table contains columns indicating if -the volume should be used for reading (\ippdbcolumn{available}) and -writing (\ippdbcolumn{allocate}), allowing these properties to be -manually set, which is useful in scheduling downtime for hardware -issues. Another column, \ippdbcolumn{xattr}, is used to control the -behavior of this volume, with specific values used to denote desired -behavior. For instance, the value of $1$ is used to indicate that a -volume should only be used as a backup volume (which accepts only -replicated copies), and the value of $5$ is used to indicate that the -volume is permanently unavailable, and should be ignored. - -In addition to this permanent table describing the volumes, a -\ippdbtable{mountedvol} table also exists that only lists those -volumes that are currently visible and accessible from the -\ippprog{Nebulous} database server. This table also lists the total -and currently available disk space on each volume, allowing the -\ippprog{Nebulous} load balancing routines to prioritize those volumes -with large unused disk space before filling the volumes with only -small amounts remaining. This table is regenerated every ten to -twenty minutes, after a scan of each of the volumes listed in the -\ippdbtable{volume} table. - -The final table controlling the operations of the \ippprog{Nebulous} -volumes is the \ippdbtable{cabinet} table, which organizes the -individual volumes into ``cabinets,'' a concept loosly based on the -physical arrangement of the storage servers in the data center. These -cabinets are used to prevent the replication of a storage object -within a group of volumes where all instances of the object could be -taken off line by a single failure. As the data center cabinets share -power supplies among the servers they contain, ensuring physical -distance between replicated copies is important to guarantee that a -temporary failure of one of these devices does not significantly -impact processing. +\section{Post-Processing : Database Ingest \& Calibration} \subsection{DVO} @@ -1403,32 +1325,50 @@ analysis \citep[][see]{magnier2017a}. -\subsection{Datastore repositories} -\label{subsec: datastore} - -Transferring data between the IPP and other parts of the Pan-STARRS -system is generally accomplished via a ``datastore'', an http service -that exposes data in a common form. One of the main datastores used -by the IPP is the one located at the summit. This datastore exposes, -a list of the exposures obtained since the start of the PS1 -operations. Requests to this server may restrict to the latest by -time. Each row in the listing includes basic information about the -exposure: an exposure identifier (e.g., o5432g0123o; -see~\ref{GPC1.names} for details), the date and time of the exposure, -the telescope commanded pointing, the filter and exposure time, and -the observation comment for that exposure. The row also provides a -link to a listing of the chips associated with that exposure. This -listing includes a link to the individual chip FITS files as well as -an md5 checksum. Systems which are allowed access may download chip -FITS files via http requests to the provided links. - -The IPP also uses datastores to provide access to its own data -products. The detections identified in the \ippstage{diff} stage -images are organized by the \ippstage{publish} stage, which writes -output files containing those detections to a datastore that is -monitored by the Moving Object Processing System -\citep[][MOPS]{2013PASP..125..357D}, which analyses the detections to -identify asteroids. Separate datastores are also used by the -\ippstage{distribution} stage to provide access to data products to -the Pan-STARRS Science Consortium members. +\subsection{Addstar : DVO Ingest} +\label{subsec: addstar} +\note{CZW: This should be reviewed.} + +Upon completion of the processing of each stage, the results of the +photometry analysis are isolated in a large number of individual +catalogs, with little connection between the separate measurements of +astronomical sources. Unifying these measurements in a DVO database +is the purpose of the \ippstage{addstar} processing. The catalogs for +the \ippstage{camera}, \ippstage{staticsky}, \ippstage{skycal}, +\ippstage{fullforce}, and \ippstage{diff} are processed in this +fashion, although not every measurement in each catalog are included +in the final DVO that is constructed. + +The construction of this final DVO is performed in a hierarchical +process. The individual catalogs are added to a \ippmisc{minidvo}, +which is simply a DVO database defined over some subset of possible +inputs. These \ippmisc{minidvos} are then merged into larger +databases to construct the final completely catalog. \note{describe + database tables} + +Each catalog that is to be added to DVO has an entry created in the +\ippdbtable{addRun} database table. This entry notes which +\ippdbcolumn{stage} is the source of the catalog, and links to the +appropriate database table with the \ippdbcolumn{stage\_id} field. As +some stages, such as the \ippstage{diff} stage, create more than a +single catalog, multiple entries with the \ippdbcolumn{stage\_id} are +created, with the \ippdbcolumn{stage\_extra1} field containing an +index to the individual components. The catalog specified by the +entry is added to the target \ippmisc{minidvo} by the +\ippprog{addstar} program, \note{describe what's done?}. When this +completes, an entry containing the statistics of the job is added to +the \ippdbtable{addProcessedExp} table. + +\subsection{Calibration Operations} +\label{subsec: calibration} + +\subsection{IPP to PSPS} +\label{subsec: ipp2psps} +\note{Default to pointing to Flewelling et al 2017?} + +\subsection{PSPS Load \& Merge} +\label{subsec: psps} +\note{Default as well to pointing to Flewelling et al 2017?} + +\section{Operations \& Automation} \subsection{Pantasks \& Parallel Processing} @@ -1592,4 +1532,113 @@ \ippdbcolumn{projection\_cell}. +\subsection{Nebulous} +\label{subsec: nebulous} +Storing the large volume of data that is generated by the GPC1 camera +was recognized early in the Pan-STARRS project as a major concern. +The \ippprog{Nebulous} system was designed to organize this data. The +main components of this system is a database storing the locations of +the files, with a Simple Object Access Protocol interface between the +database and the other IPP programs. The actual files are stored on +NFS mounted partitions on a series of storage nodes in the IPP cluster +that can be accessed throughout the cluster. This distribution of +files is useful to balance the disk I/O, as this parallelizes the +load. + +The original design of \ippprog{Nebulous} was intended to aid in the +targetted processing of data, by having specific image data (such as +all the images from one OTA device) located on a single storage node. +This would allow any jobs involving that data to be assigned to the +storage node, eliminating network I/O. Important data could be +duplicated to additional data nodes, with the alternate locations +stored in the database. In practice, however, hardware failures and +increases in hard drive storage volumes and network bandwidth have +reduced the degree to which the IPP processing is targeted. + +When a program creates a new file in \ippprog{Nebulous}, it supplies +an URI of the form \code{neb://HOST.VOLUME/PATH/FILENAME}. The host +and volume specifiers are optional, and allow a file to be created on +a specific node. The path and filename appear similar to a standard +full file location, and are used internally as the ``external id''. A +storage object entry is then created in the database for this id, and +an instance of the file created on the specified node (or at random +from available nodes if left empty). This instance is created in a +deterministic filename location. The external id is hashed using the +SHA-1 function, and the first four hexadecimal digits of this hash are +separated into two two-digit strings and used as the top and second +level directory location for the disk file. The disk file is created +using the database instance id, and a transformed version of the +external id, which has colons replacing any forward slash characters, +to convert the external id path into a filename. For the example URI +above, this results in a file located on disk in a location like +\code{/data/HOST.VOLUME/nebulous/d5/d8/9244993440.PATH:FILENAME}. +This file naming structure is useful, as it duplicates database +contents on disk. + +The storage volumes that contain the data on disk are defined in the +\ippprog{Nebulous} database in a number of ways. First, the locations +and mount points for the actual NFS storage are listed in the +\ippdbtable{volume} table. This table contains columns indicating if +the volume should be used for reading (\ippdbcolumn{available}) and +writing (\ippdbcolumn{allocate}), allowing these properties to be +manually set, which is useful in scheduling downtime for hardware +issues. Another column, \ippdbcolumn{xattr}, is used to control the +behavior of this volume, with specific values used to denote desired +behavior. For instance, the value of $1$ is used to indicate that a +volume should only be used as a backup volume (which accepts only +replicated copies), and the value of $5$ is used to indicate that the +volume is permanently unavailable, and should be ignored. + +In addition to this permanent table describing the volumes, a +\ippdbtable{mountedvol} table also exists that only lists those +volumes that are currently visible and accessible from the +\ippprog{Nebulous} database server. This table also lists the total +and currently available disk space on each volume, allowing the +\ippprog{Nebulous} load balancing routines to prioritize those volumes +with large unused disk space before filling the volumes with only +small amounts remaining. This table is regenerated every ten to +twenty minutes, after a scan of each of the volumes listed in the +\ippdbtable{volume} table. + +The final table controlling the operations of the \ippprog{Nebulous} +volumes is the \ippdbtable{cabinet} table, which organizes the +individual volumes into ``cabinets,'' a concept loosly based on the +physical arrangement of the storage servers in the data center. These +cabinets are used to prevent the replication of a storage object +within a group of volumes where all instances of the object could be +taken off line by a single failure. As the data center cabinets share +power supplies among the servers they contain, ensuring physical +distance between replicated copies is important to guarantee that a +temporary failure of one of these devices does not significantly +impact processing. + +\subsection{Datastore repositories} +\label{subsec: datastore} + +Transferring data between the IPP and other parts of the Pan-STARRS +system is generally accomplished via a ``datastore'', an http service +that exposes data in a common form. One of the main datastores used +by the IPP is the one located at the summit. This datastore exposes, +a list of the exposures obtained since the start of the PS1 +operations. Requests to this server may restrict to the latest by +time. Each row in the listing includes basic information about the +exposure: an exposure identifier (e.g., o5432g0123o; +see~\ref{GPC1.names} for details), the date and time of the exposure, +the telescope commanded pointing, the filter and exposure time, and +the observation comment for that exposure. The row also provides a +link to a listing of the chips associated with that exposure. This +listing includes a link to the individual chip FITS files as well as +an md5 checksum. Systems which are allowed access may download chip +FITS files via http requests to the provided links. + +The IPP also uses datastores to provide access to its own data +products. The detections identified in the \ippstage{diff} stage +images are organized by the \ippstage{publish} stage, which writes +output files containing those detections to a datastore that is +monitored by the Moving Object Processing System +\citep[][MOPS]{2013PASP..125..357D}, which analyses the detections to +identify asteroids. Separate datastores are also used by the +\ippstage{distribution} stage to provide access to data products to +the Pan-STARRS Science Consortium members. + \subsection{ippTools and ippScripts} \label{subsec: ipptools} @@ -1693,49 +1742,4 @@ \note{This likely needs cleaning up and more information.} - -\subsection{Addstar : DVO Ingest} -\label{subsec: addstar} -\note{CZW: This should be reviewed.} - -Upon completion of the processing of each stage, the results of the -photometry analysis are isolated in a large number of individual -catalogs, with little connection between the separate measurements of -astronomical sources. Unifying these measurements in a DVO database -is the purpose of the \ippstage{addstar} processing. The catalogs for -the \ippstage{camera}, \ippstage{staticsky}, \ippstage{skycal}, -\ippstage{fullforce}, and \ippstage{diff} are processed in this -fashion, although not every measurement in each catalog are included -in the final DVO that is constructed. - -The construction of this final DVO is performed in a hierarchical -process. The individual catalogs are added to a \ippmisc{minidvo}, -which is simply a DVO database defined over some subset of possible -inputs. These \ippmisc{minidvos} are then merged into larger -databases to construct the final completely catalog. \note{describe - database tables} - -Each catalog that is to be added to DVO has an entry created in the -\ippdbtable{addRun} database table. This entry notes which -\ippdbcolumn{stage} is the source of the catalog, and links to the -appropriate database table with the \ippdbcolumn{stage\_id} field. As -some stages, such as the \ippstage{diff} stage, create more than a -single catalog, multiple entries with the \ippdbcolumn{stage\_id} are -created, with the \ippdbcolumn{stage\_extra1} field containing an -index to the individual components. The catalog specified by the -entry is added to the target \ippmisc{minidvo} by the -\ippprog{addstar} program, \note{describe what's done?}. When this -completes, an entry containing the statistics of the job is added to -the \ippdbtable{addProcessedExp} table. - -\subsection{Calibration Operations} -\label{subsec: calibration} - -\subsection{IPP to PSPS} -\label{subsec: ipp2psps} -\note{Default to pointing to Flewelling et al 2017?} - -\subsection{PSPS Load \& Merge} -\label{subsec: psps} -\note{Default as well to pointing to Flewelling et al 2017?} \section{IPP Hardware Systems}