Index: trunk/doc/release.2015/ps1.datasystem/datasystem.tex =================================================================== --- trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 40577) +++ trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 40578) @@ -241,7 +241,8 @@ responsible for linking individual detections of solar-system objects together and determining the orbits \citep[][]{2013PASP..125..357D}. -\item PSPS : this system ingests the calibrated measurements from the - IPP, MOPS, and others and generates a high-availability database - with web-based interactions for public consumption \citet[][]{flewelling2017}. +\item Published Science Products Subsystem (PSPS) : this system + ingests the calibrated measurements from the IPP, MOPS, and others + and generates a high-availability database with web-based + interactions for public consumption \citet[][]{flewelling2017}. \end{itemize} @@ -255,10 +256,10 @@ at MAST, to those which perform offline analysis for eventual ingest back into the Pan-STARRS databases and archive. The latter category -includes the ubercal photometric analysis \citep{ubercal}, the photo-z -analysis \citep{photoz}, and the QSO / RR Lyra search efforts -\citep{hernitschek2016}. In addition, collaborations within the wider +includes the ubercal photometric analysis \citep{2012ApJ...756..158S}, the photo-z +analysis \citep{2012ApJ...746..128S}, and the QSO / RR Lyra search efforts +\citep{2016ApJ...817...73H}. In addition, collaborations within the wider Pan-STARRS community have implemented a variety of science-level analyses of their own to support their science goals \citep[e.g., M31 - variable search][]{M31.REF}. +variable search][]{2014ApJ...797...22L,2012AJ....143...89L}. Figure~\ref{fig:analysis.elements} illustrates the many elements of @@ -266,6 +267,7 @@ analysis steps which occur within the Pan-STARRS Observatory, with an emphasis on the analysis, calibration, and database ingest stages. -The MOPS is described in detail by \cite{2013PASP..125..357D}, while -the summit systems are described by \note{REF?}. +The MOPS is described in detail by \cite{2013PASP..125..357D}. + +% the summit systems are described by \note{REF?}. \begin{figure*}[htbp] @@ -277,5 +279,5 @@ external groups (``customers''). The arrows show a simplified representation of the major flow of data between the analysis stages and data - processing elements. \note{arrow types are unclear for on-demand vs DVO}} + processing elements.} \end{center} \end{figure*} @@ -298,5 +300,5 @@ (\IPPstage{stack}) or used in an image subtraction (\IPPstage{diff}). As part of nightly science processing, images for certain fields such -as the Medium Deep survey fields \citep[see][]{MDref}, are stacked +as the Medium Deep survey fields \citep[see][]{huber2017}, are stacked together in nightly chunks, providing deeper detection capability on 1-day timescales. Depending on the survey mode, difference images are @@ -1104,5 +1106,5 @@ epoch. The quality of such a difference image can be enhanced by convolving one or both of the images so that the PSFs in the two -images are matched \citep[e.g.,][]{AlardLupton}. +images are matched \citep[e.g.,][]{1998ApJ...503..325A}. In the \ippstage{diff} stage, the IPP generates difference images for @@ -1159,10 +1161,7 @@ entry as such. -\section{Post-Processing : Database Ingest and Calibration} -\label{sec:postprocessing} - \begin{table}[hb] \begin{center} -\caption{DVO Database Tables\label{tab:DVO_schema} \note{fix names, include missing}} +\caption{DVO Database Tables\label{tab:DVO_schema}} \begin{tabular}{ll} \hline @@ -1171,16 +1170,13 @@ \hline Images & The images that have objects in the DB. \\ -% Image Overlaps & Image regions which are touched by specific images. \\ -Objects & The objects --- average properties of multiple detections of the same object. \\ -Average & Average photometry in multiple filters \\ -% Solar System Objects & Identification of solar system objects \\ -Measure & Detections of sources in an image identified with an Object. \\ -% Orphaned Detections & Detections of sources in an image not identified with an Object. \\ -% Non-detections & Non-detections of objects in an image. \\ +Average & Astronomical objects including their astrometric properties. \\ +SecFilt & Average photometry of the objects in multiple filters (one filter per row) \\ +Measure & Detections of sources identified with an Object, potentially linked to an image. \\ +StarPar & Stellar parameters determined by the Harvard group \citep{2015ApJ...810...25G} \\ +Lensing & Lensing (KSB) parameters and fixed circular aperture photometry from the warps \\ +LensObj & Average lensing and fixed circular aperture photometry \\ +Galphot & Result of galaxy model fits (forced galaxy models) \\ SkyRegions & spatial distribution of tables \\ -% Filters & Filters understood by the system. \\ Photcodes & Transformations between different photometric systems \\ -% Zero Points & History of Zero-point \& Airmass terms \\ -% Distortion Models & History of Optical Distortion terms \\ Hosts & computers used to store the tables \\ \hline @@ -1188,4 +1184,7 @@ \end{center} \end{table} + +\section{Post-Processing : Database Ingest and Calibration} +\label{sec:postprocessing} \subsection{DVO} @@ -1200,16 +1199,35 @@ part of the astrometric and photometric calibration process. This database system, called the ``Desktop Virtual Observatory'' (DVO) was -developed originally for the LONEOS project \citep{}, and used as part of the -CFHT Elixir system \citep{2004PASP..116..449M}. The capabilities of -this databasing system have been somewhat expanded for the Pan-STARRS -context. +developed originally for the LONEOS project +\citep{1995DPS....27.0110B}, and used as part of the CFHT Elixir +system \citep{2004PASP..116..449M}. The capabilities of this +databasing system have been somewhat expanded for the Pan-STARRS +context. % overview DVO tracks three main classes of information: 1) average properties of astronomical objects; 2) measurements of those objects (from which the -average properties are derived); 3) properties of the images which provided -some or all of the measuements. Figure~\ref{fig:DVO_schema} -illustrates the schematic relationship between these types of -measurements. +average properties are derived); 3) properties of the images which +provided some or all of the measuements. In addition, certain +metadata tables define general features of the database. +Table~\ref{tab:DVO_schema} lists the full collection of database +tables used by DVO. + +%Figure~\ref{fig:DVO_schema} +%illustrates the schematic relationship between these types of +%measurements. + +\begin{figure*}[htbp] + \begin{center} + \includegraphics[width=\hsize,clip]{skypartition.png} + \caption{\label{fig:sky.partition} Level 3 sky paritioning. The + blue grid shows the outlines of the different regions assigned to + separate tables in the sky partitioning scheme. The Galactic + plane is shown as a solid red line while the ecliptic is shown in + green. This organization of the sky duplicates that used by the + HST Guide Star Catalog \citep{1988IAUS..133..239J}. + } +\end{center} +\end{figure*} In the most basic implementation, a collection of measurements for @@ -1227,12 +1245,10 @@ and the derived astronomical objects. -\subsubsection{DVO Schema} - -Table~\ref{tab:DVO_schema} lists the full collection of database -tables used by DVO. These tables fall into one of several classes: -those which store information about the average properties of -astronomical objects; those which store information about individual -measurements; those which store information about the images; those -which store supporting information (metadata). +% +%% These tables fall into one of several classes: +%% those which store information about the average properties of +%% astronomical objects; those which store information about individual +%% measurements; those which store information about the images; those +%% which store supporting information (metadata). %% DVO includes two major classes of database tables: those containing @@ -1251,4 +1267,6 @@ %% levels each containing a finer mesh of regions covering the sky. +\subsubsection{DVO Schema} + \subsubsubsection{Photcodes} @@ -1269,5 +1287,5 @@ from external data sources for which DVO does not have any information to determine a calibration (e.g., instrumental magnitudes and detector -coordinates). These are measurements are reference values and are +coordinates). These measurements are reference values and are assigned \ippmisc{REF} photcodes. @@ -1283,18 +1301,16 @@ transform a measurement in the specific photcode to a common system. For example, a \ippmisc{DEP} photcode GPC1.g.X01 would have the -nominal zero point (24.563) and airmass term (0.147). The structures -allow for individual chips to have different color terms to bring them -to a common filter system. - -Beyond the basic use, DVO has the ability to accept data from other -kinds of data sources in which measurements are not clearly associated -with specific images. DVO ingest methods are defined for several -large-scale surveys for which the published data represent average -properties derived from multiple measurements, and for which the -measurement-to-image relationship is not provided. Ingests methods -have been defined, for example, for 2MASS, WISE, Gaia, USNO-B. In each -of these cases, the astrometric and photometric measurements are -stored in the \ippdbtable{Measure} table, with the data source -identified by the photcode of the measurement. +nominal zero point (24.563) and airmass term (0.147). The database +elements allow for individual chips to have different color terms to +bring them to a common filter system. + +DVO ingest methods are defined for several large-scale surveys for +which the published data represent average properties derived from +multiple measurements, and for which the measurement-to-image +relationship is not provided. Ingests methods have been defined, for +example, for 2MASS, WISE, Gaia, USNO-B. In each of these cases, the +astrometric and photometric measurements are stored in the +\ippdbtable{Measure} table, with the data source identified by the +photcode of the measurement. \subsubsubsection{Measurement Tables} @@ -1302,11 +1318,11 @@ In most cases, the individual measurements of the astronomical objects are carried in the table \ippdbtable{Measure}. For measurements from -PS1 in the PV3 / DR1 database, this would consist of values determined -by \ippprog{psphot} for each \ippstage{chip}, \ippstage{warp}, or -\ippstage{stack} stage image. Measurements for other cameras -processed by the IPP may also be included similarly in a DVO database. -Measurements from other sources, such as SDSS, 2MASS, or WISE, can -also be included in this table, distinguished by their different -photcodes. +PS1 in the PV3 / DR1 or DR2 databases, this would consist of values +determined by \ippprog{psphot} for each \ippstage{chip}, +\ippstage{warp}, or \ippstage{stack} stage image. Measurements for +other cameras processed by the IPP may also be included similarly in a +DVO database. Measurements from other sources, such as SDSS, 2MASS, +or WISE, can also be included in this table, distinguished by their +different photcodes. The \ippdbtable{Measure} table includes the instrumental magnitudes @@ -1338,4 +1354,24 @@ (respectively 3.0, 4.63, and 7.43 arcsec). This table contains one row for every warp image on which the object was measured. + +The \ippdbtable{Galphot} table stores the results of the forced galaxy +fitting analysis for each object that has been measured. This table +contains one row per filter and model type (Sersic, Exponential, or +DeVaucouleur) if forced galaxy models have been calculate for the +object. + +The \ippdbtable{Starpar} table carries measurements provided by the +Harvard team (Green, Schlafly, Finkbeiner) from the analysis of the +SED of objects in the PS1 $3\pi$ data, using the PV2 analysis version +\citep{2015ApJ...810...25G,2014ApJ...783..114G}. In this work, the +goal was a 3D model of the dust in the Galaxy based on Pan-STARRS and +2MASS photometry. As part of this analysis, the authors fit the SEDs +of all stellar sources (as determined by a cut based on the PSF - +aperture magnitudes) with stellar models including free parameters of +extinction, distance modulus, metallicity, and absolute r-band +magnitude. While these photometric distance modulus measurements are +not extremely precise, they provide a constraint on the distance which +is used in our analysis of the astrometry +\citep[see][]{magnier2017.calibration}. %% Similarly to the \ippdbtable{Measure} table, the fields @@ -1413,47 +1449,4 @@ calculated. -The \ippdbtable{Galphot} table stores the results of the forced galaxy -fitting analysis for each object that has been measured. This table -contains one row per filter and model type (Sersic, Exponential, or -DeVaucouleur) if forced galaxy models have been calculate for the -object. - -The \ippdbtable{Starpar} table carries measurements provide by Greg -Green \& Eddie Schlafly from their analysis of the SED of objects in -the PS1 $3\pi$ data, using the PV1 version of the analysis -\citep{2015ApJ...810...25G}. In this work, the goal was a 3D model of -the dust in the Galaxy based on Pan-STARRS and 2MASS photometry. As -part of this analysis, the authors fit the SEDs of all \note{stellar?} -sources with stellar models including free parameters of extinction, -distance modulus, metallicity, and absolute r-band magnitude. While -these photometric distance modulus measurements are not extremely -precise (see below), they provide a constraint on the distance is used -in our analysis of the astrometry -\citep[see][]{magnier2017.calibration}. - -In the \ippdbtable{Measure} table, there are three fields which -provide two independent links from the specific measurement to the -associated object: \ippdbtable{Measure}.\ippdbcolumn{catID} specifies -the spatial partition to which the measurement belongs (see -Section~\ref{sec:SkyPartition} below); -\ippdbtable{Measure}.\ippdbcolumn{objID} specifies to which entry in -the \ippdbtable{Average} table the measurement belongs. These two 32 -bit fields can thus be combined into a single 64 bit ID unique for all -objects in the database. In addition, the field -\ippdbtable{Measure}.\ippdbcolumn{averef} specifies the row number in -the \ippdbtable{Average} table of the associated object. The -\ippdbtable{Measure} table may be unsorted, in which case it is slow -to find the measurements associated with a specific object (a full -table scan is required). After the table is sorted and indexed, the -\ippdbcolumn{Measure} rows for a given object are grouped together. -In this case, the fields -\ippdbtable{Average}.\ippdbcolumn{measureOffset} and -\ippdbcolumn{Average}.\ippdbcolumn{Nmeasure} define an index for the -code to jump to the list of measurements for a single object. The -field \ippdbtable{Measure}.\ippdbcolumn{imageID} defines the link from -the measurement to the image which supplied the measurement. - -\note{Discuss PSPS IDs} - \subsubsubsection{Image Tables} @@ -1470,9 +1463,16 @@ zero point, etc. For GPC1 and other mosaic cameras, an additional row is defined to carry the projection and camera distortion elements of -the astrometry model. As images are loaded into this table, they -are assigned an internal ID (a running sequence in the table). Images -may also be assigned an external ID: in the case of the GPC1 images, -this ID is defined by the processing mysql database and is guaranteed -to be unique within the processing system. +the astrometry model. As images are loaded into this table, they are +assigned an internal ID (a running sequence in the table), stored in +the field \ippdbcolumn{imageID}. Images may also be assigned an ID +derived from the external source of the image (field +\ippdbcolumn{externID}): in the case of the GPC1 images, this ID is +defined by the processing mysql database and is guaranteed to be +unique within the processing system. In the case of GPC1 exposures, +the external image ID is set to the database field +\ippdbtable{chipImfile}.\ippdbcolumn{chip_imfile_id}. A second field +(\ippdbcolumn{sourceID}) identifies which of the possible image-like +tables supplied this image, guaranteeing uniqueness of image IDs +across the different IPP stages. %% Data from GPC1 (and other cameras processed by the IPP) are loaded @@ -1491,5 +1491,4 @@ flat-field corrections determined by the astrometry calibration analysis \citep[see][]{magnier2017.calibration}. -\note{use names and match DVO schema table} \subsubsection{Sky Partition} @@ -1513,25 +1512,27 @@ files. Level 0 is a single region covering the full sky. Level 1 divides the sky in declination into bands 7.5\degree\ high, as defined -by the HST GSC. Level 2 subdivides these declination bands in the RA -direction, with spacing related to the stellar density. Level 3 -divides these RA chunks into 4 - 8 smaller partitions. This level -exactly matches the HST GSC layout, and uses the same naming -convention to identify the partitions: \code{n0000/0000}, etc. Level 4 -further divides these regions by a factor of 16. In the -\ippdbtable{SkyTable}, a region at one level has a pointer to its -parent region (the one which contains it) and a sequence pointing to -its children (regions it contains). The \ippdbtable{SkyTable} enables -fast lookups of the on-disk partitions which map to a specific -coordinate on the sky. In general, a single DVO will have the full -sky represented with tables at a single level, although it is possible -for mixed levels to be used. For the PV3 master database, the -partitioning is at Level 4, resulting in \approx 150,000 regions to -cover the full sky, of which \approx 110,000 are used for the PV3 -$3\pi$ data. The densest portions of the bulge contain at most -\approx 300,000 astronomical objects in the database files, with an -associated maximum of \approx 30 million measurements in these files. -With the compression scheme described below, the largest database -files are \approx 3GB, which can be loaded into memory in 30 seconds -on the processing machines that contain partition data. +by the HST Guide Star Catalog +\citep[GSC][]{1988IAUS..133..239J,1990AJ.....99.2019L}. Level 2 +subdivides these declination bands in the RA direction, with spacing +related to the stellar density. Level 3 divides these RA chunks into +4 - 8 smaller partitions. This level exactly matches the HST GSC +layout, and uses the same naming convention to identify the +partitions: \code{n0000/0000}, etc. Level 4 further divides these +regions by a factor of 16. In the \ippdbtable{SkyTable}, a region at +one level has a pointer to its parent region (the one which contains +it) and a sequence pointing to its children (regions it contains). +The \ippdbtable{SkyTable} enables fast lookups of the on-disk +partitions which map to a specific coordinate on the sky. In general, +a single DVO will have the full sky represented with tables at a +single level, although it is possible for mixed levels to be used. +For the PV3 master database, the partitioning is at Level 4, resulting +in \approx 150,000 regions to cover the full sky, of which \approx +110,000 are used for the PV3 $3\pi$ data. The densest portions of the +bulge contain at most \approx 300,000 astronomical objects in the +database files, with an associated maximum of \approx 30 million +measurements in these files. With the compression scheme described +below, the largest database files are \approx 3GB, which can be loaded +into memory in 30 seconds on the processing machines that contain +partition data. % parallel partitions @@ -1559,4 +1560,86 @@ reasonable timescale. +\subsubsection{Object and Measurement IDs} + +Within the DVO system, certain integer fields are used to provide unique +identifiers for measurements and objects. The original implementation +of DVO was limited to 32-bit integer fields, but since the maximum +number of objects and measurements was expected to be larger than +$2^{32}$, two 32-bit integer fields are joined together to make +sufficiently large IDs. + +In the table of objects (\ippdbtable{Average}), the fields +\ippdbcolumn{objID} and \ippdbcolumn{catID} together form a unique +64-bit integer value to identify the objects. The \ippdbcolumn{catID} +field is a sequence number for the sky partition table (the +`catalog') in which the object is contained, while \ippdbcolumn{objID} +is an incrementing sequence number within that sky partition +table. As long as no sky partition tables contain more that +$2^{32}$ objects, these fields will not overflow. These two fields +are included in the \ippdbtable{Measure}, \ippdbtable{GalPhot}, +\ippdbtable{StarPar}, \ippdbtable{Lensing}, and \ippdbtable{LensObj} +tables to link the entries in those tables back their corresponding +object. Note that \ippdbtable{SecFilt} does {\em not} contain these +ID fields; the rows in this table are maintained in the correct +sequence to match the \ippdbtable{Average} table entries. + +The \ippdbtable{Measure} table, containing the detections of objects +from individual exposures or stack, or the (potentially +non-signficant) measurements from a warp, uses the 32-bit integer +fields \ippdbcolumn{detID} and \ippdbcolumn{imageID} to uniquely +identify each entry. The \ippdbcolumn{imageID} is the running +sequence number of the ``image'' (GPC1 OTA, stack, warp, or other +other source of the measurement) in which the object was measured. +The \ippdbcolumn{imageID} is a value internal to DVO, and is unique +across all types of images. The \ippdbcolumn{detID} field is a 32-bit +integer giving the sequence number of the detection within the image. +For images processed by the IPP (e.g., using \ippprog{psphot}), the +\ippdbcolumn{detID} corresponds to the output field labeled as +\ippmisc{IPP_IDET} in those data products. Since measurements from +the same image may be spread across multiple sky partition tables, +both \ippdbcolumn{detID} and \ippdbcolumn{imageID} much be used to +uniquely identify a detection within the database. + +In the \ippdbtable{Measure} table, the field \ippdbcolumn{averef} +specifies the row number in the \ippdbtable{Average} table of the +associated object. The \ippdbtable{Measure} table may be unsorted, in +which case it is slow to find the measurements associated with a +specific object (a full table scan is required, referencing +\ippdbcolumn{objID}). After the table is sorted and indexed, the +\ippdbcolumn{Measure} rows for a given object are grouped together. +In this case, the fields \ippdbtable{Average}.\ippdbcolumn{measureOffset} and +\ippdbcolumn{Average}.\ippdbcolumn{Nmeasure} define an index for the +code to jump to the list of measurements for a single object. The +field \ippdbtable{Measure}.\ippdbcolumn{imageID} defines the link from +the measurement to the image which supplied the measurement. + +DVO is also used to construct the unique object and detection IDs used +by the Published Science Products Subsystem (PSPS). Within the PSPS, +the field named \ippdbcolumn{objID} in that database is used to +allows valid joins between tables to select the different kinds of +attributes of the same astronomical objects. This 64-bit integer ID +is constructed based on the coordinates of the object, as described by +\cite[][]{flewelling2017}. In short, the digits of the right +ascension and declination coordinates are used to define a single +64-bit integer with spatial resolution of roughly 3 milliarcseconds. +This values used by this field are generated by the DVO system and +stored in the \ippdbtable{Average} table in the field +\ippdbcolumn{extID}. + +Within the PSPS, the \ippdbtable{Detection} table carries an ID which +is unique for each measurement, equivalent to the DVO +\ippdbcolumn{det_id}, \ippdbcolumn{image_id} pair. In this case, the +PSPS \ippdbcolumn{detectID} is constructed from the MJD of the +exposure, the number of the OTA (e.g., OTA64), and the detection +sequence within the image to form a single unique 64-bit integer value. +For detections from the stack images, the MJD is not unique, so a +different rubrick is used to define IDs for those detections. The +field \ippdbcolumn{XstackDetectID} (where '\ippdbcolumn{X}' is one of +g,r,i,z,y) is constructed from the GPC1 stack ID +(\ippdbtable{stackRun.stack_id}), the detection sequence within the +stack image, and the same value used to define \ippdbcolumn{sourceID} +above. These two types of detection IDs are generated by the program +\ippprog{addstar} when the images and stacks are ingested into DVO. + \subsubsection{DVO Data Storage} @@ -1567,5 +1650,5 @@ of files for tables which are spatially partitioned. The binary FITS tables are compressed using the (to date) experimental FITS binary -table compression strategy outlined by \citet{RickWhite}. Table compression +table compression strategy outlined by \citet{2012arXiv1201.1340P}. Table compression is an option in DVO; for the PV3 database, the large data volume (70TB compressed) drove the decision to compress the tables. @@ -1573,15 +1656,17 @@ % FITS table compression details The FITS binary table compression scheme uses a strategy similar to -that used for FITS image compression (\note{REF}). The binary tabular -data is compressed and stored in the ``HEAP'' section of the FITS table -extension, with pointers to the compressed data stored in the regular -data section. Each column in the FITS table is compressed as one (or -more) blocks. The standard header keywords which describe the data -column format (e.g., TFORM1) are replaced with keywords which describe -the location and size of the compressed data in the HEAP section; the -information about the uncompressed data is moved to a keyword with ``Z'' -prepended (e.g., ZFORM1) and an additional field is added to define -the compression algorithm (e.g., ZCTYP1). The column names (e.g., -TTYPE1) and units (e.g., TUNIT1) are retained in their original form. +that used for FITS image compression +\citep[][]{1999ASPC..172..125W,2000ASPC..216..551P}. The binary +tabular data is compressed and stored in the ``HEAP'' section of the +FITS table extension, with pointers to the compressed data stored in +the regular data section. Each column in the FITS table is compressed +as one (or more) blocks. The standard header keywords which describe +the data column format (e.g., TFORM1) are replaced with keywords which +describe the location and size of the compressed data in the HEAP +section; the information about the uncompressed data is moved to a +keyword with ``Z'' prepended (e.g., ZFORM1) and an additional field is +added to define the compression algorithm (e.g., ZCTYP1). The column +names (e.g., TTYPE1) and units (e.g., TUNIT1) are retained in their +original form. % FITS table compression details @@ -1698,10 +1783,10 @@ catalog files (``smf files'') and determined the zero points of those exposures which were believed to be obtained in photometric -conditions. This process, called ``\"ubercal'', is described in detail -by \cite{2012ApJ...756..158S} for the first (PV1) version. In brief, photometric -periods, with time-scales of at least \note{half of a night}, are -identified by a combination of automatic analysis and manual -inspection. A single solution for all images in a given filter is -determined to minimize scatter for individual stars. The free +conditions. This process, called ``\"ubercal'', is described in +detail by \cite{2012ApJ...756..158S} for the first (PV1) version. In +brief, photometric periods, with time-scales of a large fraction of a +night, are identified by a combination of automatic analysis and +manual inspection. A single solution for all images in a given filter +is determined to minimize scatter for individual stars. The free parameters in this solution consist of a single zero point and airmass slope for each photometric period along with a collection of @@ -1709,8 +1794,9 @@ seasons''). For the PV3 \"ubercal analysis, the flat-field offsets were determined on a $2\times2$ grid for each chip and 5 flat-field -seasons were chosen (listed in Table~\ref{tab:flat-field-seasons}). -The boundaries of the flat-field seasons were determined by -independent inspection of the residuals observed in the Medium Deep -fields. +seasons were identified. The boundaries of the flat-field seasons +were determined by independent inspection of the residuals observed in +the Medium Deep fields. + +%% (listed in Table~\ref{tab:flat-field-seasons}) XXX add this table After the \"ubercal analysis of the photometric periods is completed, @@ -1741,10 +1827,10 @@ Telescope Sciences Institute through their Mikulski Archive for Space Telescopes (MAST). The underying database at MAST is a copy of a -database generated at the IfA by the subsystem -called PSPS : the \note{define PSPS}. The construction of the PSPS -version of the PS1 database starts once the PS1 photometry and -astrometry measurements have been calibrated within the DVO system. -The construction takes place in several stages, described in detail by -\cite{flewelling2017}. We summarize those steps here. +database generated at the IfA by the Published Science Products +Subsystem (PSPS). The construction of the PSPS version of the PS1 +database starts once the PS1 photometry and astrometry measurements +have been calibrated within the DVO system. The construction takes +place in several stages, described in detail by \cite{flewelling2017}. +We summarize those steps here. The first stage of constructing the PSPS database consists of the @@ -1824,11 +1910,11 @@ collection of ``tasks'' which describe the concept of a command which might be run and to regularly generate new commands based on that -concept. The ``tasks'' are defined using the opihi scripting language +concept. The ``tasks'' are defined using the \ippprog{opihi} scripting language (also shared by DVO and other user-interactive programs within the IPP). -\ippprog{Pantasks} repeatedly checks each task in an attempt to generate a new -command: we say \ippprog{pantasks} attempts to ``execute'' the task in each of -these attempts. Tasks may specify the time between execution +\ippprog{Pantasks} repeatedly checks each task in an attempt to +generate a new command: we say \ippprog{pantasks} attempts to +``execute'' the task. Tasks may specify the time between execution attempts, with a 1 second default. @@ -1839,89 +1925,53 @@ executed. A dynamic command is defined within a special block of the task, called \code{task.exec}. This block is a snipet of code (in the -opihi language) which is run each time the task is executed. The +\ippprog{opihi} language) which is run each time the task is executed. The \code{task.exec} code may refer to variables or other data structures -defined by the opihi language within the \ippprog{pantasks} environment. Within -a single \ippprog{pantasks} instance, all opihi variables and data -structures have global context (\ie, all are visible to all tasks). -Variables are by default global, but within the context of an opihi -macro (equivalent of a function call), variables may be -locally-scoped. Other data structures (see below) are global and must -be protected with name space choices. - -Within the \ippprog{task.exec} macro, the command to be run must be -defined with the function ``command''. Once the \ippprog{task.exec} -macro exits successfully, the defined command is then added to the list of jobs -to be run within the UNIX environment. Jobs may be run in one of two -ways: locally or via the parallel processing system. The task, or the -\ippprog{task.exec} macro, uses the ``host'; command to define how to -run the job. If the host is set to ``local'', then the job is run in -the background by \ippprog{pantasks} itself (using the C \code{execvp} -function). Otherwise, the job is sent to the parallel processing -system to be run on another machine within the cluster. If the host -is set to the special value ``anyhost'', then the parallel processing -system is allowed to choose the processing computer arbitrarily. Any -other value is taken to be the DNS name of the computer on which this -job should run. If the option \code{-required} is supplied to the -\code{host} command, then the parallel processing system must ensure -that the job only runs on the specifically named computer. Otherwise, -the parallel processing system may choose to redirect the command to -another computer using its own rules, e.g. to balance processing load -across the cluster. - -When the \ippprog{task.exec} macro is run, the code may choose (e.g., +defined by the \ippprog{opihi} language within the \ippprog{pantasks} +environment. Within a single \ippprog{pantasks} instance, all \ippprog{opihi} +variables and data structures have global context by default (\ie, all +are visible to all tasks). Within the context of an \ippprog{opihi} macro +(equivalent of a function call), variables may be locally-scoped. +Other data structures (see below) are global and must be protected +with name space choices. + +Within the \code{task.exec} macro, the command to be run is defined by +the script. Once the \code{task.exec} macro exits successfully, the +defined command is then added to the list of jobs to be run within the +UNIX environment. Jobs may be run in one of two ways: locally or via +the parallel processing system. The task, or the \code{task.exec} +macro, uses the \code{host} command to define how to run the job. If +the host is set to ``local'', then the job is run in the background by +\ippprog{pantasks} itself (using the C \code{execvp} function). +Otherwise, the job is sent to the parallel processing system to be run +on another machine within the cluster. If the host is set to the +special value ``anyhost'', then the parallel processing system is +allowed to choose the processing computer arbitrarily. Any other +value is taken to be the DNS name of the computer on which this job +should run. The host may (optionally) be required for the command, in +which case the parallel processing system must ensure that the job +only runs on the specifically-named computer. Otherwise, the parallel +processing system may choose to redirect the command to another +computer using its own rules, e.g. to balance processing load across +the cluster. + +When the \code{task.exec} macro is run, the code may choose (e.g., based on tests of some global variables) to exit the macro with an -error condition, e.g., with the ``break'' command. In this -circumstance, no job is produced by the task. The task will be tried -again the next time it is executed. This feature allows for the user -to set processing blocks which depend on some external tests. For -example, some task may check external network connectivity and set a -variable based on the network status; other tasks may then choose to -wait until the network is available before attempting to run. - -Other task options discussed below exist to control the system -behavior in detail. Note that the options below may be dynamically -reset by the \ippprog{task.exec} macro. - -\note{this section probably has too much detail; move this into an - online user guide?} - -The option ``npending'' may be used to limit the number of jobs which -are simultaneously executed for a specific task. For example, some -classes of jobs should only be run one-at-a-time because they are not -protected against collisions or they may overload a resource. The use -of ``npending'' allows these situations to be handled cleanly within -\ippprog{pantasks} (avoiding cumbersome coding within with program or supporting -script). - -The option ``nmax'' limits the total number of jobs which a task -generates. This option may be useful in cases where -\ippprog{pantasks} is used to perform a limited set of operations. -\note{do we actually use this in IPP?} - -The option ``trange'' allows the user to restrict the time period during -which the specific tasks is executed. This option is given with a -start and an end time for the limiting time range. These times may be -of one of several forms: ``HH:MM:SS'' specifies a time within a day -(in UT or local time?). ``Day[@HH:MM:SS]'' specifies a time on a -specific day, e.g., \code{trange Mon@13:00 Tue@09:00} says the task -should be run from 1pm on Mondays to 9am on Tuesdays. ``YYYY/MM/DD,HH:MM:SS'' -specifies a time on a specific date within the year. The start and -end times must be of the same class. The \code{trange} command has -some optional arguments as well. The option \code{-nmax NNN} defines -the maximum number of jobs which may be run in that time range. The -option \code{-exclude} specifies that the time range is a period when -the task should {\em not} be executed. An arbirary number of time -ranges may be specified \note{how are they evaluated?} - -The option \code{nice} specifies the ``nice'' level at which the job is -run when it is executed. The parallel processing system must respect -this concept. - -The option \code{active} can be used to turn on and off a task for -periods. Since a user command or a macro run by \ippprog{pantasks} can -re-define task options, the \code{active} state may be changed -independently of the task execute. This is useful for keeping tasks -defined by a \ippprog{pantasks} instance, but allowing the user to -prevent them from running for some reason. +error condition. In this circumstance, no job is produced by the +task. The task will be tried again the next time it is executed. +This feature allows for the user to set processing blocks which depend +on some external tests. For example, some task may check external +network connectivity and set a variable based on the network status; +other tasks may then choose to wait until the network is available +before attempting to run. + +Other task options exist to control the system behavior in detail. +These options may be dynamically reset by the \code{task.exec} macro. +Some options control the number of jobs, such as limiting the number +of currently-outsanding jobs for a given task, or limiting the total +number generated. Other options can be used to control the time when +jobs of a certain task are allowed to run. It is also possible to +specify the UNIX ``nice'' level at which the job is +run when it is executed. Finally, individual tasks may be disabled +while the system is still running. \subsubsection{pcontrol} @@ -1931,7 +1981,6 @@ across many machines in the computing cluster. The parallel processing system used by \ippprog{pantasks} is an independent -software system. The default parallel processing system is a program -called \ippprog{pcontrol}\footnote{Alternatives are possible: e.g., - {\tt condor} has been experimentally integrated with +software system called \ippprog{pcontrol}\footnote{Alternatives are + possible: e.g., {\tt condor} has been experimentally integrated with \ippprog{pantasks} for tests}. @@ -1946,6 +1995,6 @@ one of several states: pending (ready to run), running (jobs which are running), exit (job has completed), busy (job is being checked by -\ippprog{pcontrol}), crash (job has exited with a signal(?), normally -segv). +\ippprog{pcontrol}), crash (job has exited with a signal, normally +\code{segv}). Similarly, the hosts may also have one of several states: off, down, @@ -2001,7 +2050,7 @@ The \ippprog{pantasks} program can be run as a stand-alone program -which presents an opihi shell interface to the user when it is +which presents an \ippprog{opihi} shell interface to the user when it is started. This mode is useful for testing as all errors are reported -back to the opihi shell. However, when the user exits the shell, the +back to the \ippprog{opihi} shell. However, when the user exits the shell, the \ippprog{pantasks} instance exits, shutting down \ippprog{pcontrol} and all remote client connections. In standard operations, \ippprog{pantasks} is run in a client @@ -2025,6 +2074,6 @@ \end{verbatim} \caption{\label{fig:task_example} Example of a simple static - task in the opihi-based scripting language used by ippprog{pantasks}. In - this example, ippprog{pantasks} would run a single instance of the command + task in the opihi-based scripting language used by pantasks. In + this example, pantasks would run a single instance of the command ({\tt ls /tmp}) every 5 seconds, sending the stdout and stderr to the listed files. } @@ -2036,12 +2085,12 @@ \subsubsection{Pantasks scripts: ippTasks} -\ippprog{Pantasks} provides an environment in which commands can be generated -and extensive parallel processing managed. The details of how to -implement the different stages of IPP processing are captured in a -collection of scripts written for \ippprog{pantasks} in the \code{opihi} -language. In general, each stage is defined by an associated script -collected together under the \ippmisc{ippTasks} collection. While -each script has its own details, there are a number of common -elements. +\ippprog{Pantasks} provides an environment in which commands can be +generated and extensive parallel processing managed. The details of +how to implement the different stages of IPP processing are captured +in a collection of scripts written for \ippprog{pantasks} in the +\ippprog{opihi} language. In general, each stage is defined by an +associated script collected together under the \ippmisc{ippTasks} +collection. While each script has its own details, there are a number +of common elements. Most stages consist of two related tasks: a \ippmisc{load} task, which @@ -2059,5 +2108,5 @@ job is permitted to run simultaneously, preventing race conditions. -The results from the database query job are stored in an opihi data +The results from the database query job are stored in an \ippprog{opihi} data structure called a \ippmisc{book} within the \ippprog{pantasks} environment. Each row in the result set is saved to a separate entry @@ -2077,6 +2126,7 @@ \ippmisc{book} for any pages with \ippdbcolumn{pantasksState} set to \ippmisc{DONE}, and removes them from the book, as these represent -jobs that have finished. \note{the manipulation above takes place in - the task.exit subscript} +jobs that have finished. + +% \note{the manipulation above takes place in the task.exit subscript} The associated \ippmisc{run} task generates jobs constructed from the @@ -2099,5 +2149,5 @@ program to do the data analysis work and a supporting Perl script which performs the database update upon completion. Upon completion, -the \ippprog{pantasks} \ippmisc{RUN} tasks is responsible for updating the +the \ippprog{pantasks} \ippmisc{RUN} task is responsible for updating the status within the book, but not within the processing database. This split keeps the interactions at the \ippprog{pantasks} level relatively light, @@ -2147,9 +2197,10 @@ \label{sec:automation} -Outside of the basic sequence of \ippstage{chip} to \ippstage{warp}, there is no single -natural next step. For example: a stack can be generated with any -number of input warps; a difference image can be generated between a -warp and any one of many other warps or stacks. Without a single -sequence, more complex and sophisticated decisions much be made. +Beyond of the basic sequence of \ippstage{chip} to \ippstage{warp}, +there is no single natural ``next step''. For example: a stack can be +generated with any number of input warps; a difference image can be +generated between a warp and any one of many other warps or stacks. +Without a single sequence, more complex and sophisticated decisions +much be made. For nightly processing of data obtained at the summit, this is handled @@ -2252,5 +2303,5 @@ generated by the GPC1 camera. The \ippprog{Nebulous} system was designed to aid in thie process. \ippprog{Nebulous} is not a file -system per-se, but only method of tracking the locations of files +system per-se, but only a method of tracking the locations of files within the file system, and of tracking duplicate copies of the same file. The core of \ippprog{Nebulous} is a mysql database which tracks @@ -2276,29 +2327,29 @@ All of the analysis stages which interact with that chip could then be preferentially targeted to be run on that computer. The localization -in \ippprog{Nebulous} and the host targeted processing in \ippprog{pantasks} -can therefore work together to encourage processing to require only -local disk access, reducing the I/O local on the network -infrastructure. In the early stages of the Pan-STARRS project, this -was important because network bandwidth was an expensive resource. In -practice, the as-built IPP has had sufficient network bandwidth that -this targetting was not required. In practice, due to the timing of -hardware acquisition, occasional hardware failures, and other -organizational details, targeted processing has only been used to a -moderate degree within the Pan-STARRS cluster. +in \ippprog{Nebulous} and the host targeted processing in +\ippprog{pantasks} can therefore work together to encourage processing +to require only local disk access, reducing the I/O local on the +network infrastructure. In the early stages of the Pan-STARRS +project, this was important because network bandwidth was an expensive +resource. In practice, the as-built IPP has had sufficient network +bandwidth that this targetting was not completely required. In +practice, due to the timing of hardware acquisition, occasional +hardware failures, and other organizational details, targeted +processing has only been used to a moderate degree within the +Pan-STARRS cluster. \subsubsection{Implementation Details} The user interfaces to Nebulous consist of command-line programs as -well as APIs in both C and Perl. - -The basic user commands to interact with Nebulous are to 1) query the -database for an existing storage object, and find a valid file -instance associated with that object; 2) create a new storage object, -which instantiates an empty file that can be opened for writing; 3) -replicate an existing storage object to create more file instances; 4) -cull a single file instance of storage object from the cluster; and 5) -remove a storage object, and ensure that all file instances are -removed. The filehandles returned for newly created instances can -then be opened for reading and writing data to that instance. +well as APIs in both C and Perl. The basic user commands to interact +with Nebulous are to 1) query the database for an existing storage +object, and find a valid file instance associated with that object; 2) +create a new storage object, which instantiates an empty file that can +be opened for writing; 3) replicate an existing storage object to +create more file instances; 4) cull a single file instance of storage +object from the cluster; and 5) remove a storage object, and ensure +that all file instances are removed. The filehandles returned for +newly created instances can then be opened for reading and writing +data to that instance. % The basic user commands to interact @@ -2429,12 +2480,12 @@ Requests to this server may restrict to the latest by time. Each row in the listing includes basic information about the exposure: an -exposure identifier (e.g., o5432g0123o; see~\ref{GPC1.names} for -details), the date and time of the exposure, the telescope commanded -pointing, the filter and exposure time, and the observation comment -for that exposure. The row also provides a link to a listing of the -chips associated with that exposure. This listing includes a link to -the individual chip FITS files as well as an md5 checksum. Systems -which are allowed access may download the raw chip FITS files via http -requests to the provided links. +exposure identifier \citep[e.g., o5432g0123o; see][for + details]{chambers2017}, the date and time of the exposure, the +telescope commanded pointing, the filter and exposure time, and the +observation comment for that exposure. The row also provides a link +to a listing of the chips associated with that exposure. This listing +includes a link to the individual chip FITS files as well as an md5 +checksum. Systems which are allowed access may download the raw chip +FITS files via http requests to the provided links. % \note{add a discussion of gpc1 filenames?} Index: trunk/doc/release.2015/ps1.datasystem/dvo.sh =================================================================== --- trunk/doc/release.2015/ps1.datasystem/dvo.sh (revision 40578) +++ trunk/doc/release.2015/ps1.datasystem/dvo.sh (revision 40578) @@ -0,0 +1,9 @@ + +macro figure2 + + resize 1000 510; region 180 0 83 ait; box -c black -tickpad 0.5 -pad 0.5 -ticks 0000 -labels 0000; skycat -all -depth 3 -c blue + style -sz 0.4 -c green; ecliptic; style -c red; galactic + style -sz 0.8 -c darkgreen; ecliptic; style -c red; galactic + png -name skypartition.png + +end