Context Navigation

← Previous Change
Next Change →

datasystem.tex

Timestamp:

Dec 7, 2018, 12:33:01 PM (8 years ago)

Author:

eugene

Message:

cleanup text, add skypartition figure, repair system overview paper

File:

: 1 edited

trunk/doc/release.2015/ps1.datasystem/datasystem.tex (modified) (40 diffs)

Legend:

: Unmodified
: Added
: Removed

trunk/doc/release.2015/ps1.datasystem/datasystem.tex

-              r40566
+              r40578
   responsible for linking individual detections of solar-system
   objects together and determining the orbits \citep[][]{2013PASP..125..357D}.
+\item PSPS : this system ingests the calibrated measurements from the
+  IPP, MOPS, and others and generates a high-availability database
+  with web-based interactions for public consumption \citet[][]{flewelling2017}.
+\item Published Science Products Subsystem (PSPS) : this system
+  ingests the calibrated measurements from the IPP, MOPS, and others
+  and generates a high-availability database with web-based
+  interactions for public consumption \citet[][]{flewelling2017}.
 \end{itemize}
 …
 at MAST, to those which perform offline analysis for eventual ingest
 back into the Pan-STARRS databases and archive.  The latter category
 includes the ubercal photometric analysis \citep{ubercal}, the photo-z
 analysis \citep{photoz}, and the QSO / RR Lyra search efforts
 \citep{hernitschek2016}.  In addition, collaborations within the wider
+includes the ubercal photometric analysis \citep{2012ApJ...756..158S}, the photo-z
+analysis \citep{2012ApJ...746..128S}, and the QSO / RR Lyra search efforts
+\citep{2016ApJ...817...73H}.  In addition, collaborations within the wider
 Pan-STARRS community have implemented a variety of science-level
 analyses of their own to support their science goals \citep[e.g., M31
   variable search][]{M31.REF}.
+variable search][]{2014ApJ...797...22L,2012AJ....143...89L}.
 Figure~\ref{fig:analysis.elements} illustrates the many elements of
 …
 analysis steps which occur within the Pan-STARRS Observatory, with an
 emphasis on the analysis, calibration, and database ingest stages.
+The MOPS is described in detail by \cite{2013PASP..125..357D}, while
+the summit systems are described by \note{REF?}.
+The MOPS is described in detail by \cite{2013PASP..125..357D}.
+% the summit systems are described by \note{REF?}.
 \begin{figure*}[htbp]
 …
     external groups (``customers'').  The arrows show a simplified representation
   of the major flow of data between the analysis stages and data
   processing elements. \note{arrow types are unclear for on-demand vs DVO}}
+  processing elements.}
   \end{center}
 \end{figure*}
 …
 (\IPPstage{stack}) or used in an image subtraction (\IPPstage{diff}).
 As part of nightly science processing, images for certain fields such
 as the Medium Deep survey fields \citep[see][]{MDref}, are stacked
+as the Medium Deep survey fields \citep[see][]{huber2017}, are stacked
 together in nightly chunks, providing deeper detection capability on
 -day timescales.  Depending on the survey mode, difference images are
 …
 epoch.  The quality of such a difference image can be enhanced by
 convolving one or both of the images so that the PSFs in the two
 images are matched \citep[e.g.,][]{AlardLupton}.
+images are matched \citep[e.g.,][]{1998ApJ...503..325A}.
 In the \ippstage{diff} stage, the IPP generates difference images for
 …
 entry as such.
-\section{Post-Processing : Database Ingest and Calibration}
-\label{sec:postprocessing}
 \begin{table}[hb]
 \begin{center}
 \caption{DVO Database Tables\label{tab:DVO_schema} \note{fix names, include missing}}
+\caption{DVO Database Tables\label{tab:DVO_schema}}
 \begin{tabular}{ll}
 \hline
 …
 \hline
 Images               & The images that have objects in the DB. \\
 % Image Overlaps       & Image regions which are touched by specific images. \\
 Objects              & The objects --- average properties of multiple detections of the same object. \\
 Average              & Average photometry in multiple filters \\
 % Solar System Objects & Identification of solar system objects \\
 Measure              & Detections of sources in an image identified with an Object. \\
 % Orphaned Detections  & Detections of sources in an image not identified with an Object. \\
 % Non-detections       & Non-detections of objects in an image. \\
+Average              & Astronomical objects including their astrometric properties. \\
+SecFilt              & Average photometry of the objects in multiple filters (one filter per row) \\
+Measure              & Detections of sources identified with an Object, potentially linked to an image. \\
+StarPar              & Stellar parameters determined by the Harvard group \citep{2015ApJ...810...25G} \\
+Lensing              & Lensing (KSB) parameters and fixed circular aperture photometry from the warps \\
+LensObj              & Average lensing and fixed circular aperture photometry \\
+Galphot              & Result of galaxy model fits (forced galaxy models) \\
 SkyRegions           & spatial distribution of tables \\
-% Filters              & Filters understood by the system. \\
 Photcodes            & Transformations between different photometric systems \\
-% Zero Points          & History of Zero-point \& Airmass terms \\
-% Distortion Models    & History of Optical Distortion terms \\
 Hosts                & computers used to store the tables \\
 \hline
 …
 \end{center}
 \end{table}
+\section{Post-Processing : Database Ingest and Calibration}
+\label{sec:postprocessing}
 \subsection{DVO}
 …
 part of the astrometric and photometric calibration process.  This
 database system, called the ``Desktop Virtual Observatory'' (DVO) was
+developed originally for the LONEOS project \citep{}, and used as part of the
+CFHT Elixir system \citep{2004PASP..116..449M}.  The capabilities of
+this databasing system have been somewhat expanded for the Pan-STARRS
+context.
+developed originally for the LONEOS project
+\citep{1995DPS....27.0110B}, and used as part of the CFHT Elixir
+system \citep{2004PASP..116..449M}.  The capabilities of this
+databasing system have been somewhat expanded for the Pan-STARRS
+context.
 % overview
 DVO tracks three main classes of information: 1) average properties of
 astronomical objects; 2) measurements of those objects (from which the
+average properties are derived); 3) properties of the images which provided
+some or all of the measuements.  Figure~\ref{fig:DVO_schema}
+illustrates the schematic relationship between these types of
+measurements.
+average properties are derived); 3) properties of the images which
+provided some or all of the measuements.  In addition, certain
+metadata tables define general features of the database.
+Table~\ref{tab:DVO_schema} lists the full collection of database
+tables used by DVO.
+%Figure~\ref{fig:DVO_schema}
+%illustrates the schematic relationship between these types of
+%measurements.
+\begin{figure*}[htbp]
+  \begin{center}
+ \includegraphics[width=\hsize,clip]{skypartition.png}
+  \caption{\label{fig:sky.partition} Level 3 sky paritioning.  The
+    blue grid shows the outlines of the different regions assigned to
+    separate tables in the sky partitioning scheme.  The Galactic
+    plane is shown as a solid red line while the ecliptic is shown in
+    green.  This organization of the sky duplicates that used by the
+    HST Guide Star Catalog \citep{1988IAUS..133..239J}.
+ }
+\end{center}
+\end{figure*}
 In the most basic implementation, a collection of measurements for
 …
 and the derived astronomical objects.
+\subsubsection{DVO Schema}
+Table~\ref{tab:DVO_schema} lists the full collection of database
+tables used by DVO.  These tables fall into one of several classes:
+those which store information about the average properties of
+astronomical objects; those which store information about individual
+measurements; those which store information about the images; those
+which store supporting information (metadata).
+%
+%% These tables fall into one of several classes:
+%% those which store information about the average properties of
+%% astronomical objects; those which store information about individual
+%% measurements; those which store information about the images; those
+%% which store supporting information (metadata).
 %% DVO includes two major classes of database tables: those containing
 …
 %% levels each containing a finer mesh of regions covering the sky.
+\subsubsection{DVO Schema}
 \subsubsubsection{Photcodes}
 …
 from external data sources for which DVO does not have any information
 to determine a calibration (e.g., instrumental magnitudes and detector
 coordinates).  These are measurements are reference values and are
+coordinates).  These measurements are reference values and are
 assigned \ippmisc{REF} photcodes.
 …
 transform a measurement in the specific photcode to a common system.
 For example, a \ippmisc{DEP} photcode GPC1.g.X01 would have the
+nominal zero point (24.563) and airmass term (0.147).  The structures
+allow for individual chips to have different color terms to bring them
+to a common filter system.
+Beyond the basic use, DVO has the ability to accept data from other
+kinds of data sources in which measurements are not clearly associated
+with specific images.  DVO ingest methods are defined for several
+large-scale surveys for which the published data represent average
+properties derived from multiple measurements, and for which the
+measurement-to-image relationship is not provided.  Ingests methods
+have been defined, for example, for 2MASS, WISE, Gaia, USNO-B.  In each
+of these cases, the astrometric and photometric measurements are
+stored in the \ippdbtable{Measure} table, with the data source
+identified by the photcode of the measurement.
+nominal zero point (24.563) and airmass term (0.147).  The database
+elements allow for individual chips to have different color terms to
+bring them to a common filter system.
+DVO ingest methods are defined for several large-scale surveys for
+which the published data represent average properties derived from
+multiple measurements, and for which the measurement-to-image
+relationship is not provided.  Ingests methods have been defined, for
+example, for 2MASS, WISE, Gaia, USNO-B.  In each of these cases, the
+astrometric and photometric measurements are stored in the
+\ippdbtable{Measure} table, with the data source identified by the
+photcode of the measurement.
 \subsubsubsection{Measurement Tables}
 …
 In most cases, the individual measurements of the astronomical objects
 are carried in the table \ippdbtable{Measure}.  For measurements from
 PS1 in the PV3 / DR1 database, this would consist of values determined
+by \ippprog{psphot} for each \ippstage{chip}, \ippstage{warp}, or
 \ippstage{stack} stage image.  Measurements for other cameras
+processed by the IPP may also be included similarly in a DVO database.
+Measurements from other sources, such as SDSS, 2MASS, or WISE, can
+also be included in this table, distinguished by their different
 photcodes.
+PS1 in the PV3 / DR1 or DR2 databases, this would consist of values
+determined by \ippprog{psphot} for each \ippstage{chip},
+\ippstage{warp}, or \ippstage{stack} stage image.  Measurements for
+other cameras processed by the IPP may also be included similarly in a
+DVO database.  Measurements from other sources, such as SDSS, 2MASS,
+or WISE, can also be included in this table, distinguished by their
+different photcodes.
 The \ippdbtable{Measure} table includes the instrumental magnitudes
 …
 (respectively 3.0, 4.63, and 7.43 arcsec).  This table contains one
 row for every warp image on which the object was measured.
+The \ippdbtable{Galphot} table stores the results of the forced galaxy
+fitting analysis for each object that has been measured.  This table
+contains one row per filter and model type (Sersic, Exponential, or
+DeVaucouleur) if forced galaxy models have been calculate for the
+object.
+The \ippdbtable{Starpar} table carries measurements provided by the
+Harvard team (Green, Schlafly, Finkbeiner) from the analysis of the
+SED of objects in the PS1 $3\pi$ data, using the PV2 analysis version
+\citep{2015ApJ...810...25G,2014ApJ...783..114G}.  In this work, the
+goal was a 3D model of the dust in the Galaxy based on Pan-STARRS and
+MASS photometry.  As part of this analysis, the authors fit the SEDs
+of all stellar sources (as determined by a cut based on the PSF -
+aperture magnitudes) with stellar models including free parameters of
+extinction, distance modulus, metallicity, and absolute r-band
+magnitude.  While these photometric distance modulus measurements are
+not extremely precise, they provide a constraint on the distance which
+is used in our analysis of the astrometry
+\citep[see][]{magnier2017.calibration}.
 %% Similarly to the \ippdbtable{Measure} table, the fields
 …
 calculated.
-The \ippdbtable{Galphot} table stores the results of the forced galaxy
-fitting analysis for each object that has been measured.  This table
-contains one row per filter and model type (Sersic, Exponential, or
-DeVaucouleur) if forced galaxy models have been calculate for the
-object.
-The \ippdbtable{Starpar} table carries measurements provide by Greg
-Green \& Eddie Schlafly from their analysis of the SED of objects in
-the PS1 $3\pi$ data, using the PV1 version of the analysis
-\citep{2015ApJ...810...25G}.  In this work, the goal was a 3D model of
-the dust in the Galaxy based on Pan-STARRS and 2MASS photometry.  As
-part of this analysis, the authors fit the SEDs of all \note{stellar?}
-sources with stellar models including free parameters of extinction,
-distance modulus, metallicity, and absolute r-band magnitude.  While
-these photometric distance modulus measurements are not extremely
-precise (see below), they provide a constraint on the distance is used
-in our analysis of the astrometry
-\citep[see][]{magnier2017.calibration}.
-In the \ippdbtable{Measure} table, there are three fields which
-provide two independent links from the specific measurement to the
-associated object: \ippdbtable{Measure}.\ippdbcolumn{catID} specifies
-the spatial partition to which the measurement belongs (see
-Section~\ref{sec:SkyPartition} below);
-\ippdbtable{Measure}.\ippdbcolumn{objID} specifies to which entry in
-the \ippdbtable{Average} table the measurement belongs.  These two 32
-bit fields can thus be combined into a single 64 bit ID unique for all
-objects in the database.  In addition, the field
-\ippdbtable{Measure}.\ippdbcolumn{averef} specifies the row number in
-the \ippdbtable{Average} table of the associated object.  The
-\ippdbtable{Measure} table may be unsorted, in which case it is slow
-to find the measurements associated with a specific object (a full
-table scan is required).  After the table is sorted and indexed, the
-\ippdbcolumn{Measure} rows for a given object are grouped together.
-In this case, the fields
-\ippdbtable{Average}.\ippdbcolumn{measureOffset} and
-\ippdbcolumn{Average}.\ippdbcolumn{Nmeasure} define an index for the
-code to jump to the list of measurements for a single object.  The
-field \ippdbtable{Measure}.\ippdbcolumn{imageID} defines the link from
-the measurement to the image which supplied the measurement.
-\note{Discuss PSPS IDs}
 \subsubsubsection{Image Tables}
 …
 zero point, etc.  For GPC1 and other mosaic cameras, an additional row
 is defined to carry the projection and camera distortion elements of
+the astrometry model.  As images are loaded into this table, they
+are assigned an internal ID (a running sequence in the table).  Images
+may also be assigned an external ID: in the case of the GPC1 images,
+this ID is defined by the processing mysql database and is guaranteed
+to be unique within the processing system.
+the astrometry model.  As images are loaded into this table, they are
+assigned an internal ID (a running sequence in the table), stored in
+the field \ippdbcolumn{imageID}.  Images may also be assigned an ID
+derived from the external source of the image (field
+\ippdbcolumn{externID}): in the case of the GPC1 images, this ID is
+defined by the processing mysql database and is guaranteed to be
+unique within the processing system.  In the case of GPC1 exposures,
+the external image ID is set to the database field
+\ippdbtable{chipImfile}.\ippdbcolumn{chip_imfile_id}. A second field
+(\ippdbcolumn{sourceID}) identifies which of the possible image-like
+tables supplied this image, guaranteeing uniqueness of image IDs
+across the different IPP stages.
 %% Data from GPC1 (and other cameras processed by the IPP) are loaded
 …
 flat-field corrections determined by the astrometry calibration
 analysis \citep[see][]{magnier2017.calibration}.
-\note{use names and match DVO schema table}
 \subsubsection{Sky Partition}
 …
 files.  Level 0 is a single region covering the full sky.  Level 1
 divides the sky in declination into bands 7.5\degree\ high, as defined
+by the HST GSC.  Level 2 subdivides these declination bands in the RA
+direction, with spacing related to the stellar density.  Level 3
+divides these RA chunks into 4 - 8 smaller partitions.  This level
+exactly matches the HST GSC layout, and uses the same naming
+convention to identify the partitions: \code{n0000/0000}, etc. Level 4
+further divides these regions by a factor of 16.  In the
+\ippdbtable{SkyTable}, a region at one level has a pointer to its
+parent region (the one which contains it) and a sequence pointing to
+its children (regions it contains).  The \ippdbtable{SkyTable} enables
+fast lookups of the on-disk partitions which map to a specific
+coordinate on the sky.  In general, a single DVO will have the full
+sky represented with tables at a single level, although it is possible
+for mixed levels to be used.  For the PV3 master database, the
+partitioning is at Level 4, resulting in \approx 150,000 regions to
+cover the full sky, of which \approx 110,000 are used for the PV3
+$3\pi$ data.  The densest portions of the bulge contain at most
+\approx 300,000 astronomical objects in the database files, with an
+associated maximum of \approx 30 million measurements in these files.
+With the compression scheme described below, the largest database
+files are \approx 3GB, which can be loaded into memory in 30 seconds
+on the processing machines that contain partition data.
+by the HST Guide Star Catalog
+\citep[GSC][]{1988IAUS..133..239J,1990AJ.....99.2019L}.  Level 2
+subdivides these declination bands in the RA direction, with spacing
+related to the stellar density.  Level 3 divides these RA chunks into
+- 8 smaller partitions.  This level exactly matches the HST GSC
+layout, and uses the same naming convention to identify the
+partitions: \code{n0000/0000}, etc. Level 4 further divides these
+regions by a factor of 16.  In the \ippdbtable{SkyTable}, a region at
+one level has a pointer to its parent region (the one which contains
+it) and a sequence pointing to its children (regions it contains).
+The \ippdbtable{SkyTable} enables fast lookups of the on-disk
+partitions which map to a specific coordinate on the sky.  In general,
+a single DVO will have the full sky represented with tables at a
+single level, although it is possible for mixed levels to be used.
+For the PV3 master database, the partitioning is at Level 4, resulting
+in \approx 150,000 regions to cover the full sky, of which \approx
+,000 are used for the PV3 $3\pi$ data.  The densest portions of the
+bulge contain at most \approx 300,000 astronomical objects in the
+database files, with an associated maximum of \approx 30 million
+measurements in these files.  With the compression scheme described
+below, the largest database files are \approx 3GB, which can be loaded
+into memory in 30 seconds on the processing machines that contain
+partition data.
 % parallel partitions
 …
 reasonable timescale.
+\subsubsection{Object and Measurement IDs}
+Within the DVO system, certain integer fields are used to provide unique
+identifiers for measurements and objects.  The original implementation
+of DVO was limited to 32-bit integer fields, but since the maximum
+number of objects and measurements was expected to be larger than
+$2^{32}$, two 32-bit integer fields are joined together to make
+sufficiently large IDs.
+In the table of objects (\ippdbtable{Average}), the fields
+\ippdbcolumn{objID} and \ippdbcolumn{catID} together form a unique
+-bit integer value to identify the objects.  The \ippdbcolumn{catID}
+field is a sequence number for the sky partition table (the
+`catalog') in which the object is contained, while \ippdbcolumn{objID}
+is an incrementing sequence number within that sky partition
+table.  As long as no sky partition tables contain more that
+$2^{32}$ objects, these fields will not overflow.  These two fields
+are included in the \ippdbtable{Measure}, \ippdbtable{GalPhot},
+\ippdbtable{StarPar}, \ippdbtable{Lensing}, and \ippdbtable{LensObj}
+tables to link the entries in those tables back their corresponding
+object.  Note that \ippdbtable{SecFilt} does {\em not} contain these
+ID fields; the rows in this table are maintained in the correct
+sequence to match the \ippdbtable{Average} table entries.
+The \ippdbtable{Measure} table, containing the detections of objects
+from individual exposures or stack, or the (potentially
+non-signficant) measurements from a warp, uses the 32-bit integer
+fields \ippdbcolumn{detID} and \ippdbcolumn{imageID} to uniquely
+identify each entry.  The \ippdbcolumn{imageID} is the running
+sequence number of the ``image'' (GPC1 OTA, stack, warp, or other
+other source of the measurement) in which the object was measured.
+The \ippdbcolumn{imageID} is a value internal to DVO, and is unique
+across all types of images.  The \ippdbcolumn{detID} field is a 32-bit
+integer giving the sequence number of the detection within the image.
+For images processed by the IPP (e.g., using \ippprog{psphot}), the
+\ippdbcolumn{detID} corresponds to the output field labeled as
+\ippmisc{IPP_IDET} in those data products.  Since measurements from
+the same image may be spread across multiple sky partition tables,
+both \ippdbcolumn{detID} and \ippdbcolumn{imageID} much be used to
+uniquely identify a detection within the database.
+In the \ippdbtable{Measure} table, the field \ippdbcolumn{averef}
+specifies the row number in the \ippdbtable{Average} table of the
+associated object.  The \ippdbtable{Measure} table may be unsorted, in
+which case it is slow to find the measurements associated with a
+specific object (a full table scan is required, referencing
+\ippdbcolumn{objID}).  After the table is sorted and indexed, the
+\ippdbcolumn{Measure} rows for a given object are grouped together.
+In this case, the fields \ippdbtable{Average}.\ippdbcolumn{measureOffset} and
+\ippdbcolumn{Average}.\ippdbcolumn{Nmeasure} define an index for the
+code to jump to the list of measurements for a single object.  The
+field \ippdbtable{Measure}.\ippdbcolumn{imageID} defines the link from
+the measurement to the image which supplied the measurement.
+DVO is also used to construct the unique object and detection IDs used
+by the Published Science Products Subsystem (PSPS).  Within the PSPS,
+the field named \ippdbcolumn{objID} in that database is used to
+allows valid joins between tables to select the different kinds of
+attributes of the same astronomical objects.  This 64-bit integer ID
+is constructed based on the coordinates of the object, as described by
+\cite[][]{flewelling2017}.  In short, the digits of the right
+ascension and declination coordinates are used to define a single
+-bit integer with spatial resolution of roughly 3 milliarcseconds.
+This values used by this field are generated by the DVO system and
+stored in the \ippdbtable{Average} table in the field
+\ippdbcolumn{extID}.
+Within the PSPS, the \ippdbtable{Detection} table carries an ID which
+is unique for each measurement, equivalent to the DVO
+\ippdbcolumn{det_id}, \ippdbcolumn{image_id} pair.  In this case, the
+PSPS \ippdbcolumn{detectID} is constructed from the MJD of the
+exposure, the number of the OTA (e.g., OTA64), and the detection
+sequence within the image to form a single unique 64-bit integer value.
+For detections from the stack images, the MJD is not unique, so a
+different rubrick is used to define IDs for those detections.  The
+field \ippdbcolumn{XstackDetectID} (where '\ippdbcolumn{X}' is one of
+g,r,i,z,y) is constructed from the GPC1 stack ID
+(\ippdbtable{stackRun.stack_id}), the detection sequence within the
+stack image, and the same value used to define \ippdbcolumn{sourceID}
+above.  These two types of detection IDs are generated by the program
+\ippprog{addstar} when the images and stacks are ingested into DVO.
 \subsubsection{DVO Data Storage}
 …
 of files for tables which are spatially partitioned.  The binary FITS
 tables are compressed using the (to date) experimental FITS binary
 table compression strategy outlined by \citet{RickWhite}.  Table compression
+table compression strategy outlined by \citet{2012arXiv1201.1340P}.  Table compression
 is an option in DVO; for the PV3 database, the large data
 volume (70TB compressed) drove the decision to compress the tables.
 …
 % FITS table compression details
 The FITS binary table compression scheme uses a strategy similar to
+that used for FITS image compression (\note{REF}).  The binary tabular
+data is compressed and stored in the ``HEAP'' section of the FITS table
+extension, with pointers to the compressed data stored in the regular
+data section.  Each column in the FITS table is compressed as one (or
+more) blocks.  The standard header keywords which describe the data
+column format (e.g., TFORM1) are replaced with keywords which describe
+the location and size of the compressed data in the HEAP section; the
+information about the uncompressed data is moved to a keyword with ``Z''
+prepended (e.g., ZFORM1) and an additional field is added to define
+the compression algorithm (e.g., ZCTYP1).  The column names (e.g.,
+TTYPE1) and units (e.g., TUNIT1) are retained in their original form.
+that used for FITS image compression
+\citep[][]{1999ASPC..172..125W,2000ASPC..216..551P}.  The binary
+tabular data is compressed and stored in the ``HEAP'' section of the
+FITS table extension, with pointers to the compressed data stored in
+the regular data section.  Each column in the FITS table is compressed
+as one (or more) blocks.  The standard header keywords which describe
+the data column format (e.g., TFORM1) are replaced with keywords which
+describe the location and size of the compressed data in the HEAP
+section; the information about the uncompressed data is moved to a
+keyword with ``Z'' prepended (e.g., ZFORM1) and an additional field is
+added to define the compression algorithm (e.g., ZCTYP1).  The column
+names (e.g., TTYPE1) and units (e.g., TUNIT1) are retained in their
+original form.
 % FITS table compression details
 …
 catalog files (``smf files'') and determined the zero points of those
 exposures which were believed to be obtained in photometric
 conditions.  This process, called ``\"ubercal'', is described in detail
+by \cite{2012ApJ...756..158S} for the first (PV1) version.  In brief, photometric
+periods, with time-scales of at least \note{half of a night}, are
+identified by a combination of automatic analysis and manual
+inspection.  A single solution for all images in a given filter is
 determined to minimize scatter for individual stars.  The free
+conditions.  This process, called ``\"ubercal'', is described in
+detail by \cite{2012ApJ...756..158S} for the first (PV1) version.  In
+brief, photometric periods, with time-scales of a large fraction of a
+night, are identified by a combination of automatic analysis and
+manual inspection.  A single solution for all images in a given filter
+is determined to minimize scatter for individual stars.  The free
 parameters in this solution consist of a single zero point and airmass
 slope for each photometric period along with a collection of
 …
 seasons'').  For the PV3 \"ubercal analysis, the flat-field offsets
 were determined on a $2\times2$ grid for each chip and 5 flat-field
+seasons were chosen (listed in Table~\ref{tab:flat-field-seasons}).
+The boundaries of the flat-field seasons were determined by
+independent inspection of the residuals observed in the Medium Deep
+fields.
+seasons were identified.  The boundaries of the flat-field seasons
+were determined by independent inspection of the residuals observed in
+the Medium Deep fields.
+%%  (listed in Table~\ref{tab:flat-field-seasons}) XXX add this table
 After the \"ubercal analysis of the photometric periods is completed,
 …
 Telescope Sciences Institute through their Mikulski Archive for Space
 Telescopes (MAST).  The underying database at MAST is a copy of a
 database generated at the IfA by the subsystem
+called PSPS : the \note{define PSPS}.  The construction of the PSPS
+version of the PS1 database starts once the PS1 photometry and
+astrometry measurements have been calibrated within the DVO system.
+The construction takes place in several stages, described in detail by
 \cite{flewelling2017}.  We summarize those steps here.
+database generated at the IfA by the Published Science Products
+Subsystem (PSPS).  The construction of the PSPS version of the PS1
+database starts once the PS1 photometry and astrometry measurements
+have been calibrated within the DVO system.  The construction takes
+place in several stages, described in detail by \cite{flewelling2017}.
+We summarize those steps here.
 The first stage of constructing the PSPS database consists of the
 …
 collection of ``tasks'' which describe the concept of a command which
 might be run and to regularly generate new commands based on that
 concept.  The ``tasks'' are defined using the opihi scripting language
+concept.  The ``tasks'' are defined using the \ippprog{opihi} scripting language
 (also shared by DVO and other user-interactive programs within the
 IPP).
 \ippprog{Pantasks} repeatedly checks each task in an attempt to generate a new
+command: we say \ippprog{pantasks} attempts to ``execute'' the task in each of
 these attempts.  Tasks may specify the time between execution
+\ippprog{Pantasks} repeatedly checks each task in an attempt to
+generate a new command: we say \ippprog{pantasks} attempts to
+``execute'' the task.  Tasks may specify the time between execution
 attempts, with a 1 second default.
 …
 executed.  A dynamic command is defined within a special block of the
 task, called \code{task.exec}.  This block is a snipet of code (in the
 opihi language) which is run each time the task is executed.  The
+\ippprog{opihi} language) which is run each time the task is executed.  The
 \code{task.exec} code may refer to variables or other data structures
 defined by the opihi language within the \ippprog{pantasks} environment.  Within
+a single \ippprog{pantasks} instance, all opihi variables and data
+structures have global context (\ie, all are visible to all tasks).
+Variables are by default global, but within the context of an opihi
+macro (equivalent of a function call), variables may be
+locally-scoped.  Other data structures (see below) are global and must
 be protected with name space choices.
 Within the \ippprog{task.exec} macro, the command to be run must be
+defined with the function ``command''.  Once the \ippprog{task.exec}
+macro exits successfully, the defined command is then added to the list of jobs
+to be run within the UNIX environment.  Jobs may be run in one of two
+ways: locally or via the parallel processing system.  The task, or the
+\ippprog{task.exec} macro, uses the ``host'; command to define how to
+run the job.  If the host is set to ``local'', then the job is run in
+the background by \ippprog{pantasks} itself (using the C \code{execvp}
+function).  Otherwise, the job is sent to the parallel processing
+system to be run on another machine within the cluster.  If the host
+is set to the special value ``anyhost'', then the parallel processing
+system is allowed to choose the processing computer arbitrarily.  Any
+other value is taken to be the DNS name of the computer on which this
+job should run.  If the option \code{-required} is supplied to the
+\code{host} command, then the parallel processing system must ensure
+that the job only runs on the specifically named computer.  Otherwise,
+the parallel processing system may choose to redirect the command to
+another computer using its own rules, e.g. to balance processing load
 across the cluster.
 When the \ippprog{task.exec} macro is run, the code may choose (e.g.,
+defined by the \ippprog{opihi} language within the \ippprog{pantasks}
+environment.  Within a single \ippprog{pantasks} instance, all \ippprog{opihi}
+variables and data structures have global context by default (\ie, all
+are visible to all tasks).  Within the context of an \ippprog{opihi} macro
+(equivalent of a function call), variables may be locally-scoped.
+Other data structures (see below) are global and must be protected
+with name space choices.
+Within the \code{task.exec} macro, the command to be run is defined by
+the script.  Once the \code{task.exec} macro exits successfully, the
+defined command is then added to the list of jobs to be run within the
+UNIX environment.  Jobs may be run in one of two ways: locally or via
+the parallel processing system.  The task, or the \code{task.exec}
+macro, uses the \code{host} command to define how to run the job.  If
+the host is set to ``local'', then the job is run in the background by
+\ippprog{pantasks} itself (using the C \code{execvp} function).
+Otherwise, the job is sent to the parallel processing system to be run
+on another machine within the cluster.  If the host is set to the
+special value ``anyhost'', then the parallel processing system is
+allowed to choose the processing computer arbitrarily.  Any other
+value is taken to be the DNS name of the computer on which this job
+should run.  The host may (optionally) be required for the command, in
+which case the parallel processing system must ensure that the job
+only runs on the specifically-named computer.  Otherwise, the parallel
+processing system may choose to redirect the command to another
+computer using its own rules, e.g. to balance processing load across
+the cluster.
+When the \code{task.exec} macro is run, the code may choose (e.g.,
 based on tests of some global variables) to exit the macro with an
+error condition, e.g., with the ``break'' command.  In this
+circumstance, no job is produced by the task.  The task will be tried
+again the next time it is executed.  This feature allows for the user
+to set processing blocks which depend on some external tests.  For
+example, some task may check external network connectivity and set a
+variable based on the network status; other tasks may then choose to
+wait until the network is available before attempting to run.
+Other task options discussed below exist to control the system
+behavior in detail.  Note that the options below may be dynamically
+reset by the \ippprog{task.exec} macro.
+\note{this section probably has too much detail; move this into an
+  online user guide?}
+The option ``npending'' may be used to limit the number of jobs which
+are simultaneously executed for a specific task.  For example, some
+classes of jobs should only be run one-at-a-time because they are not
+protected against collisions or they may overload a resource.  The use
+of ``npending'' allows these situations to be handled cleanly within
+\ippprog{pantasks} (avoiding cumbersome coding within with program or supporting
+script).
+The option ``nmax'' limits the total number of jobs which a task
+generates.  This option may be useful in cases where
+\ippprog{pantasks} is used to perform a limited set of operations.
+\note{do we actually use this in IPP?}
+The option ``trange'' allows the user to restrict the time period during
+which the specific tasks is executed.  This option is given with a
+start and an end time for the limiting time range.  These times may be
+of one of several forms: ``HH:MM:SS'' specifies a time within a day
+(in UT or local time?).  ``Day[@HH:MM:SS]'' specifies a time on a
+specific day, e.g., \code{trange Mon@13:00 Tue@09:00} says the task
+should be run from 1pm on Mondays to 9am on Tuesdays.  ``YYYY/MM/DD,HH:MM:SS''
+specifies a time on a specific date within the year.  The start and
+end times must be of the same class.  The \code{trange} command has
+some optional arguments as well.  The option \code{-nmax NNN} defines
+the maximum number of jobs which may be run in that time range.  The
+option \code{-exclude} specifies that the time range is a period when
+the task should {\em not} be executed.  An arbirary number of time
+ranges may be specified \note{how are they evaluated?}
+The option \code{nice} specifies the ``nice'' level at which the job is
+run when it is executed.  The parallel processing system must respect
+this concept.
+The option \code{active} can be used to turn on and off a task for
+periods.  Since a user command or a macro run by \ippprog{pantasks} can
+re-define task options, the \code{active} state may be changed
+independently of the task execute.  This is useful for keeping tasks
+defined by a \ippprog{pantasks} instance, but allowing the user to
+prevent them from running for some reason.
+error condition.  In this circumstance, no job is produced by the
+task.  The task will be tried again the next time it is executed.
+This feature allows for the user to set processing blocks which depend
+on some external tests.  For example, some task may check external
+network connectivity and set a variable based on the network status;
+other tasks may then choose to wait until the network is available
+before attempting to run.
+Other task options exist to control the system behavior in detail.
+These options may be dynamically reset by the \code{task.exec} macro.
+Some options control the number of jobs, such as limiting the number
+of currently-outsanding jobs for a given task, or limiting the total
+number generated.  Other options can be used to control the time when
+jobs of a certain task are allowed to run.  It is also possible to
+specify the UNIX ``nice'' level at which the job is
+run when it is executed.  Finally, individual tasks may be disabled
+while the system is still running.
 \subsubsection{pcontrol}
 …
 across many machines in the computing cluster.  The parallel
 processing system used by \ippprog{pantasks} is an independent
+software system.  The default parallel processing system is a program
+called \ippprog{pcontrol}\footnote{Alternatives are possible: e.g.,
+  {\tt condor} has been experimentally integrated with
+software system called \ippprog{pcontrol}\footnote{Alternatives are
+  possible: e.g., {\tt condor} has been experimentally integrated with
   \ippprog{pantasks} for tests}.
 …
 one of several states: pending (ready to run), running (jobs which are
 running), exit (job has completed), busy (job is being checked by
 \ippprog{pcontrol}), crash (job has exited with a signal(?), normally
 segv).
+\ippprog{pcontrol}), crash (job has exited with a signal, normally
+\code{segv}).
 Similarly, the hosts may also have one of several states: off, down,
 …
 The \ippprog{pantasks} program can be run as a stand-alone program
 which presents an opihi shell interface to the user when it is
+which presents an \ippprog{opihi} shell interface to the user when it is
 started.  This mode is useful for testing as all errors are reported
 back to the opihi shell.  However, when the user exits the shell, the
+back to the \ippprog{opihi} shell.  However, when the user exits the shell, the
 \ippprog{pantasks} instance exits, shutting down \ippprog{pcontrol} and all remote client
 connections.  In standard operations, \ippprog{pantasks} is run in a client
 …
 \end{verbatim}
 \caption{\label{fig:task_example} Example of a simple static
   task in the opihi-based scripting language used by ippprog{pantasks}.  In
   this example, ippprog{pantasks} would run a single instance of the command
+  task in the opihi-based scripting language used by pantasks.  In
+  this example, pantasks would run a single instance of the command
   ({\tt ls /tmp}) every 5 seconds, sending the stdout and stderr to
   the listed files. }
 …
 \subsubsection{Pantasks scripts: ippTasks}
 \ippprog{Pantasks} provides an environment in which commands can be generated
+and extensive parallel processing managed.  The details of how to
+implement the different stages of IPP processing are captured in a
+collection of scripts written for \ippprog{pantasks} in the \code{opihi}
+language.  In general, each stage is defined by an associated script
+collected together under the \ippmisc{ippTasks} collection.  While
+each script has its own details, there are a number of common
 elements.
+\ippprog{Pantasks} provides an environment in which commands can be
+generated and extensive parallel processing managed.  The details of
+how to implement the different stages of IPP processing are captured
+in a collection of scripts written for \ippprog{pantasks} in the
+\ippprog{opihi} language.  In general, each stage is defined by an
+associated script collected together under the \ippmisc{ippTasks}
+collection.  While each script has its own details, there are a number
+of common elements.
 Most stages consist of two related tasks: a \ippmisc{load} task, which
 …
 job is permitted to run simultaneously, preventing race conditions.
 The results from the database query job are stored in an opihi data
+The results from the database query job are stored in an \ippprog{opihi} data
 structure called a \ippmisc{book} within the \ippprog{pantasks}
 environment.  Each row in the result set is saved to a separate entry
 …
 \ippmisc{book} for any pages with \ippdbcolumn{pantasksState} set to
 \ippmisc{DONE}, and removes them from the book, as these represent
+jobs that have finished. \note{the manipulation above takes place in
+  the task.exit subscript}
+jobs that have finished.
+% \note{the manipulation above takes place in the task.exit subscript}
 The associated \ippmisc{run} task generates jobs constructed from the
 …
 program to do the data analysis work and a supporting Perl script
 which performs the database update upon completion.  Upon completion,
 the \ippprog{pantasks} \ippmisc{RUN} tasks is responsible for updating the
+the \ippprog{pantasks} \ippmisc{RUN} task is responsible for updating the
 status within the book, but not within the processing database.  This
 split keeps the interactions at the \ippprog{pantasks} level relatively light,
 …
 \label{sec:automation}
+Outside of the basic sequence of \ippstage{chip} to \ippstage{warp}, there is no single
+natural next step.  For example: a stack can be generated with any
+number of input warps; a difference image can be generated between a
+warp and any one of many other warps or stacks.  Without a single
+sequence, more complex and sophisticated decisions much be made.
+Beyond of the basic sequence of \ippstage{chip} to \ippstage{warp},
+there is no single natural ``next step''.  For example: a stack can be
+generated with any number of input warps; a difference image can be
+generated between a warp and any one of many other warps or stacks.
+Without a single sequence, more complex and sophisticated decisions
+much be made.
 For nightly processing of data obtained at the summit, this is handled
 …
 generated by the GPC1 camera.  The \ippprog{Nebulous} system was
 designed to aid in thie process.  \ippprog{Nebulous} is not a file
 system per-se, but only method of tracking the locations of files
+system per-se, but only a method of tracking the locations of files
 within the file system, and of tracking duplicate copies of the same
 file.  The core of \ippprog{Nebulous} is a mysql database which tracks
 …
 All of the analysis stages which interact with that chip could then be
 preferentially targeted to be run on that computer.  The localization
+in \ippprog{Nebulous} and the host targeted processing in \ippprog{pantasks}
+can therefore work together to encourage processing to require only
+local disk access, reducing the I/O local on the network
+infrastructure.  In the early stages of the Pan-STARRS project, this
+was important because network bandwidth was an expensive resource.  In
+practice, the as-built IPP has had sufficient network bandwidth that
+this targetting was not required.  In practice, due to the timing of
+hardware acquisition, occasional hardware failures, and other
+organizational details, targeted processing has only been used to a
+moderate degree within the Pan-STARRS cluster.
+in \ippprog{Nebulous} and the host targeted processing in
+\ippprog{pantasks} can therefore work together to encourage processing
+to require only local disk access, reducing the I/O local on the
+network infrastructure.  In the early stages of the Pan-STARRS
+project, this was important because network bandwidth was an expensive
+resource.  In practice, the as-built IPP has had sufficient network
+bandwidth that this targetting was not completely required.  In
+practice, due to the timing of hardware acquisition, occasional
+hardware failures, and other organizational details, targeted
+processing has only been used to a moderate degree within the
+Pan-STARRS cluster.
 \subsubsection{Implementation Details}
 The user interfaces to Nebulous consist of command-line programs as
+well as APIs in both C and Perl.
+The basic user commands to interact with Nebulous are to 1) query the
+database for an existing storage object, and find a valid file
+instance associated with that object; 2) create a new storage object,
+which instantiates an empty file that can be opened for writing; 3)
+replicate an existing storage object to create more file instances; 4)
+cull a single file instance of storage object from the cluster; and 5)
+remove a storage object, and ensure that all file instances are
+removed.  The filehandles returned for newly created instances can
+then be opened for reading and writing data to that instance.
+well as APIs in both C and Perl.  The basic user commands to interact
+with Nebulous are to 1) query the database for an existing storage
+object, and find a valid file instance associated with that object; 2)
+create a new storage object, which instantiates an empty file that can
+be opened for writing; 3) replicate an existing storage object to
+create more file instances; 4) cull a single file instance of storage
+object from the cluster; and 5) remove a storage object, and ensure
+that all file instances are removed.  The filehandles returned for
+newly created instances can then be opened for reading and writing
+data to that instance.
 % The basic user commands to interact
 …
 Requests to this server may restrict to the latest by time.  Each row
 in the listing includes basic information about the exposure: an
 exposure identifier (e.g., o5432g0123o; see~\ref{GPC1.names} for
+details), the date and time of the exposure, the telescope commanded
+pointing, the filter and exposure time, and the observation comment
+for that exposure.  The row also provides a link to a listing of the
+chips associated with that exposure.  This listing includes a link to
+the individual chip FITS files as well as an md5 checksum.  Systems
 which are allowed access may download the raw chip FITS files via http
 requests to the provided links.
+exposure identifier \citep[e.g., o5432g0123o; see][for
+  details]{chambers2017}, the date and time of the exposure, the
+telescope commanded pointing, the filter and exposure time, and the
+observation comment for that exposure.  The row also provides a link
+to a listing of the chips associated with that exposure.  This listing
+includes a link to the individual chip FITS files as well as an md5
+checksum.  Systems which are allowed access may download the raw chip
+FITS files via http requests to the provided links.
 % \note{add a discussion of gpc1 filenames?}

Note: See TracChangeset for help on using the changeset viewer.

Context Navigation

Changeset 40578 for trunk/doc/release.2015/ps1.datasystem/datasystem.tex

Legend:

trunk/doc/release.2015/ps1.datasystem/datasystem.tex

Download in other formats: