Index: trunk/doc/release.2015/ps1.datasystem/datasystem.tex =================================================================== --- trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 40298) +++ trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 40559) @@ -1,8 +1,10 @@ % \documentclass[iop,floatfix]{emulateapj} % \documentclass[iop,floatfix,onecolumn]{emulateapj} -\documentclass[12pt,preprint]{aastex} -% \documentclass[10pt,preprint]{aastex} +% \documentclass[12pt,preprint]{aastex} +\documentclass[10pt,preprint]{aastex} % \pdfoutput=1 +%\RequirePackage{deluxetable} -- included by aastex? +\RequirePackage{nsfprop} \RequirePackage{color} \RequirePackage{code} @@ -93,6 +95,4 @@ \label{sec:intro} -\note{missing figures: analysis elements, DVO schema} - The 1.8m Pan-STARRS\,1 telescope is located on the summit of Haleakala on the Hawaiian island of Maui. The wide-field optical design of the @@ -185,5 +185,5 @@ \citet[][Paper VII]{huber2017} %Huber et al. 2017 (Paper VII) -describes the Medium Deep Survey in detail, including the unique issues and data products specific to that survey. The Medium Deep Survey is not part of Data Release 1. (DR1) +describes the Medium Deep Survey in detail, including the unique issues and data products specific to that survey. The Medium Deep Survey is not part of Data Release 1 or 2. Section~\ref{sec:overview} provides an overview of the full data @@ -204,11 +204,11 @@ reducing data from other cameras and telescopes. -{\color{red} {\em Note: These papers are being placed on arXiv.org to - provide crucial support information at the time of the public - release of Data Release 1 (DR1). We expect the arXiv versions to - be updated prior to submission to the Astrophysical Journal in - January 2017. Feedback and suggestions for additional information - from early users of the data products are welcome during the - submission and refereeing process.}} +%% {\color{red} {\em Note: These papers are being placed on arXiv.org to +%% provide crucial support information at the time of the public +%% release of Data Release 1 (DR1). We expect the arXiv versions to +%% be updated prior to submission to the Astrophysical Journal in +%% January 2017. Feedback and suggestions for additional information +%% from early users of the data products are welcome during the +%% submission and refereeing process.}} \section{Overview of Pan-STARRS Data Processing} @@ -243,17 +243,18 @@ \item PSPS : this system ingests the calibrated measurements from the IPP, MOPS, and others and generates a high-availability database - with web-based interactions for public consumption. + with web-based interactions for public consumption \citet[][]{flewelling2017}. + \end{itemize} -The above set of analysis stages take place at the IfA within the -scope of responsibility of the Pan-STARRS Observatory. Across the -wider Pan-STARRS colloboration(s), additional data analysis operations -are performed to support science results. These collaboration-wide -analysis operations range from those which are tightly-coupled to the -Pan-STARRS Observatory system, such as the analysis of the transient -search teams and the public archive database at MAST, to those which -perform offline analysis for eventual ingest back into the Pan-STARRS -databases and archive. The latter category includes the ubercal -photometric analysis \citep{ubercal}, the photo-z analysis -\citep{photoz}, and the QSO / RR Lyra search efforts +Management of the above set of analysis stages takes place at the IfA +within the scope of responsibility of the Pan-STARRS Observatory. +Across the wider Pan-STARRS colloboration(s), additional data analysis +operations are performed to support science results. These +collaboration-wide analysis operations range from those which are +tightly coupled to the Pan-STARRS Observatory system, such as the +analysis of the transient search teams and the public archive database +at MAST, to those which perform offline analysis for eventual ingest +back into the Pan-STARRS databases and archive. The latter category +includes the ubercal photometric analysis \citep{ubercal}, the photo-z +analysis \citep{photoz}, and the QSO / RR Lyra search efforts \citep{hernitschek2016}. In addition, collaborations within the wider Pan-STARRS community have implemented a variety of science-level @@ -263,5 +264,5 @@ Figure~\ref{fig:analysis.elements} illustrates the many elements of the Pan-STARRS data analysis system. This figure focuses on the data -analysis steps which occur within the Pan-STARRS observatory, with an +analysis steps which occur within the Pan-STARRS Observatory, with an emphasis on the analysis, calibration, and database ingest stages. The MOPS is described in detail by \cite{2013PASP..125..357D}, while @@ -276,5 +277,5 @@ external groups (``customers''). The arrows show a simplified representation of the major flow of data between the analysis stages and data - processing elements.} + processing elements. \note{arrow types are unclear for on-demand vs DVO}} \end{center} \end{figure*} @@ -320,12 +321,14 @@ analysis stages listed above which are shared with the nightly processing, these large-scale reprocessing analyses include additional -processing. A more detailed photometric analysis is performed on the -stacks, including morphological analysis appropriate to galaxies. The -results of the stack photometry analysis are used to drive a -forced-photometry analysis of the warp images. The data products from -the camera, stack photometry, and forced-warp photometry analysis -stages are ingested into the internal calibration database (DVO, the -Desktop Virtual Observatory) and used for photometric and astrometric -calibrations (see Section~\ref{sec:DVO}). +processing steps. A more detailed photometric analysis is performed +on the stacks, including morphological analysis appropriate to +galaxies. The results of the stack photometry analysis are used to +drive a forced-photometry analysis of the warp images. These analysis +steps are discussed in detail by +\citet[][]{magnier2017.analysis}. The data products from the +camera, stack photometry, and forced-warp photometry analysis stages +are ingested into the internal calibration database (DVO, the Desktop +Virtual Observatory) and used for photometric and astrometric +calibrations \citet[see Section~\ref{sec:DVO} and][]{magnier2017.calibration}. \subsection{Data Access and Distribution} @@ -345,5 +348,6 @@ (PV1 \& PV2), the data were ingested into the PSPS database system and made available to the PS1SC community through a web portal based at -the IfA as well as the MAST portal. +the IfA as well as the MAST portal \citep[see][for full + details]{flewelling2017}. \section{IPP Data Processing Stages} @@ -354,50 +358,60 @@ \begin{table*} -\caption{\label{tab:database_schema} GPC1 Database Schema Outline}\vspace{-0.5cm} +\caption{GPC1 Database Schema Outline} %\vspace{-0.5cm} \begin{center} -\begin{tabular}{lllll} +\begin{tabular}{llll} \hline \hline -{\bf Stage} & {\bf Primary Table} & {\bf Secondary Table(s)} & {\bf Key} & {\bf Notes} \\ -\hline - \ippstage{summitcopy} & \ippdbtable{pzDataStore} & & & Lists locations to check for new exposures.\\ - & \ippdbtable{summitExp} & \ippdbtable{summitImfile} & \ippdbcolumn{summit_id} & Exposures available at the telescope.\\ - & \ippdbtable{pzDownloadExp}& \ippdbtable{pzDownloadImfile} & & Exposures that are being downloaded.\\ - & \ippdbtable{newExp} & \ippdbtable{newImfile} & \ippdbcolumn{exp_id} & Exposures that have been saved to IPP cluster.\\ - - \ippstage{registration} & \ippdbtable{rawExp} & \ippdbtable{rawImfile} & \ippdbcolumn{exp_id} & \\ - \ippstage{chip} & \ippdbtable{chipRun} & \ippdbtable{chipProcessedImfile} & \ippdbcolumn{chip_id} & \\ - \ippstage{camera} & \ippdbtable{camRun} & \ippdbtable{camProcessedExp} & \ippdbcolumn{cam_id} & \\ - \ippstage{fake} & \ippdbtable{fakeRun} & \ippdbtable{fakeProcessedImfile} & \ippdbcolumn{fake_id} & \\ - \ippstage{warp} & \ippdbtable{warpRun} & \ippdbtable{warpImfile} & \ippdbcolumn{warp_id} & \\ - & & \ippdbtable{warpSkyCellMap} & & Mapping of input chips to projection skycells.\\ - & & \ippdbtable{warpSkyfile} & & \\ - \ippstage{stack} & \ippdbtable{stackRun} & \ippdbtable{stackInputSkyfile} & \ippdbcolumn{stack_id} & \\ - & & \ippdbtable{stackSumSkyfile} & & \\ - \ippstage{staticsky} & \ippdbtable{staticskyRun} & \ippdbtable{staticskyInput} & \ippdbcolumn{sky_id} & \\ - & & \ippdbtable{staticskyResult} & & \\ - \ippstage{skycal} & \ippdbtable{skycalRun} & \ippdbtable{skycalResult} & \ippdbcolumn{skycal_id} & \\ - \ippstage{fullforce} & \ippdbtable{fullForceRun} & \ippdbtable{fullForceInput} & \ippdbcolumn{ff_id} & \\ - & & \ippdbtable{fullForceResult} & & \\ - & & \ippdbtable{fullForceSummary} & & Properties about average parameters from all results.\\ - \ippstage{diff} & \ippdbtable{diffRun} & \ippdbtable{diffSkyfile} & \ippdbcolumn{diff_id} & \\ - & & \ippdbtable{diffInputSkyfile} & & \\ - \ippstage{detrend} & \ippdbtable{detRun} & \ippdbtable{detRunSummary} & \ippdbcolumn{det_id} & \\ - & & \ippdbtable{detInputExp} & & \\ - & & \ippdbtable{detRegisteredImfile} & & Information about detrends produced externally.\\ - & & \ippdbtable{detStackedImfile} & & \\ - & \ippdbtable{detProcessedExp} & \ippdbtable{detProcessedImfile} & & \\ - & \ippdbtable{detResidExp} & \ippdbtable{detResidImfile} & & \\ - & \ippdbtable{detNormalizedExp} & \ippdbtable{detNormalizedImfile} & & \\ - \ippstage{addstar} & \ippdbtable{addRun} & \ippdbtable{addProcessedExp} & \ippdbcolumn{add_id} & \\ - \ippstage{distribution} & \ippdbtable{distRun} & \ippdbtable{distComponent} & \ippdbcolumn{dist_id} & \\ - & & \ippdbtable{distTarget} & & \\ - \ippstage{publish} & \ippdbtable{publishRun} & \ippdbtable{publishDone} & \ippdbcolumn{pub_id} & \\ - & & \ippdbtable{publishClient} & & \\ - \ippstage{lap} & \ippdbtable{lapSequence} & \ippdbtable{lapRun} & \ippdbcolumn{seq_id} & Sequence of full reprocessing\\ - & \ippdbtable{lapRun} & \ippdbtable{lapExp} & \ippdbcolumn{lap_id} & \\ - \ippstage{remote} & \ippdbtable{remoteRun} & \ippdbtable{remoteComponent} & \ippdbcolumn{remote_id} & \\ +{\bf Stage} & {\bf Primary Table} & {\bf Secondary Table(s)} & {\bf Key} \\% & {\bf Notes} \\ +%%D \begin{deluxetable}{llll} +\footnotesize +%%D \tablecolumns{5} +%%D \tablewidth{0pc} +%%D \tablecaption{GPC1 Database Schema Outline} +%%D \tablehead{\colhead{Stage} & \colhead{Primary Table} & \colhead{Secondary Table} & \colhead{Key}} % & \colhead{Notes}} +%%D \startdata +%\hline + \ippstage{summitcopy} & \ippdbtable{pzDataStore} & & \\% & Lists locations to check for new exposures.\\ + & \ippdbtable{summitExp} & \ippdbtable{summitImfile} & \ippdbcolumn{summit_id} \\% & Exposures available at the telescope.\\ + & \ippdbtable{pzDownloadExp}& \ippdbtable{pzDownloadImfile} & \\% & Exposures that are being downloaded.\\ + & \ippdbtable{newExp} & \ippdbtable{newImfile} & \ippdbcolumn{exp_id} \\% & Exposures that have been saved to IPP cluster.\\ + + \ippstage{registration} & \ippdbtable{rawExp} & \ippdbtable{rawImfile} & \ippdbcolumn{exp_id} \\% & \\ + \ippstage{chip} & \ippdbtable{chipRun} & \ippdbtable{chipProcessedImfile} & \ippdbcolumn{chip_id} \\% & \\ + \ippstage{camera} & \ippdbtable{camRun} & \ippdbtable{camProcessedExp} & \ippdbcolumn{cam_id} \\% & \\ + \ippstage{fake} & \ippdbtable{fakeRun} & \ippdbtable{fakeProcessedImfile} & \ippdbcolumn{fake_id} \\% & \\ + \ippstage{warp} & \ippdbtable{warpRun} & \ippdbtable{warpImfile} & \ippdbcolumn{warp_id} \\% & \\ + & & \ippdbtable{warpSkyCellMap} & \\% & Mapping of input chips to projection skycells.\\ + & & \ippdbtable{warpSkyfile} & \\% & \\ + \ippstage{stack} & \ippdbtable{stackRun} & \ippdbtable{stackInputSkyfile} & \ippdbcolumn{stack_id} \\% & \\ + & & \ippdbtable{stackSumSkyfile} & \\% & \\ + \ippstage{staticsky} & \ippdbtable{staticskyRun} & \ippdbtable{staticskyInput} & \ippdbcolumn{sky_id} \\% & \\ + & & \ippdbtable{staticskyResult} & \\% & \\ + \ippstage{skycal} & \ippdbtable{skycalRun} & \ippdbtable{skycalResult} & \ippdbcolumn{skycal_id} \\% & \\ + \ippstage{fullforce} & \ippdbtable{fullForceRun} & \ippdbtable{fullForceInput} & \ippdbcolumn{ff_id} \\% & \\ + & & \ippdbtable{fullForceResult} & \\% & \\ + & & \ippdbtable{fullForceSummary} & \\% & Properties about average parameters from all results.\\ + \ippstage{diff} & \ippdbtable{diffRun} & \ippdbtable{diffSkyfile} & \ippdbcolumn{diff_id} \\% & \\ + & & \ippdbtable{diffInputSkyfile} & \\% & \\ + \ippstage{detrend} & \ippdbtable{detRun} & \ippdbtable{detRunSummary} & \ippdbcolumn{det_id} \\% & \\ + & & \ippdbtable{detInputExp} & \\% & \\ + & & \ippdbtable{detRegisteredImfile} & \\% & Information about detrends produced externally.\\ + & & \ippdbtable{detStackedImfile} & \\% & \\ + & \ippdbtable{detProcessedExp} & \ippdbtable{detProcessedImfile} & \\% & \\ + & \ippdbtable{detResidExp} & \ippdbtable{detResidImfile} & \\% & \\ + & \ippdbtable{detNormalizedExp} & \ippdbtable{detNormalizedImfile} & \\% & \\ + \ippstage{addstar} & \ippdbtable{addRun} & \ippdbtable{addProcessedExp} & \ippdbcolumn{add_id} \\% & \\ + \ippstage{distribution} & \ippdbtable{distRun} & \ippdbtable{distComponent} & \ippdbcolumn{dist_id} \\% & \\ + & & \ippdbtable{distTarget} & \\% & \\ + \ippstage{publish} & \ippdbtable{publishRun} & \ippdbtable{publishDone} & \ippdbcolumn{pub_id} \\% & \\ + & & \ippdbtable{publishClient} & \\% & \\ + \ippstage{lap} & \ippdbtable{lapSequence} & \ippdbtable{lapRun} & \ippdbcolumn{seq_id} \\% & Sequence of full reprocessing\\ + & \ippdbtable{lapRun} & \ippdbtable{lapExp} & \ippdbcolumn{lap_id} \\% & \\ + \ippstage{remote} & \ippdbtable{remoteRun} & \ippdbtable{remoteComponent} & \ippdbcolumn{remote_id} \\% & \\ +%%D \enddata \hline \end{tabular} +\label{tab:database_schema} +%%D \end{deluxetable} \end{center} \end{table*} @@ -428,12 +442,12 @@ either to be done, in progress, or completed. An associated secondary table (or set of tables) lists the details of component elements which -have been processed for each top-level item. Table \ref{tab: database - schema} contains an outline of the database schema, showing the -relations between tables organized by processing stage. As an -example, one critical stage is the \ippstage{chip} processing stage -(see \S\ref{sec:chip}) in which the individual chips from an exposure -are detrended and sources are detected. Within the gpc1 database, the -primary table is called \ippdbtable{chipRun} in which each exposure -has a single entry. Associated with this table is the +have been processed for each top-level item. Table +\ref{tab:database_schema} contains an outline of the database schema, +showing the relations between tables organized by processing stage. +As an example, one critical stage is the \ippstage{chip} processing +stage (see \S\ref{sec:chip}) in which the individual chips from an +exposure are detrended and sources are detected. Within the gpc1 +database, the primary table is called \ippdbtable{chipRun} in which +each exposure has a single entry. Associated with this table is the \ippdbtable{chipProcessedImfile} table, which contains one row for each of the chips associated with the exposure (up to 60 for gpc1). @@ -550,15 +564,15 @@ database tables (\ippdbtable{rawExp} and \ippdbtable{rawImfile}). -For GPC1, the \ippstage{registration} stage is also the stage at which the -\ippprog{burntool} analysis is run. This analysis is more completely -described in \citet{waters2017}. In brief, the \ippprog{burntool} -program identifies bright sources on the image, and identifies -persistence trails that result from the incomplete transfer of charge. -As this charge can leak out in subsequent exposures, the burntool -analysis is run sequentially on the exposures, based on the +For GPC1, the \ippstage{registration} stage is also the stage at which +the \ippprog{burntool} analysis is run. This analysis is more +completely described in \citet{waters2017}. In brief, the +\ippprog{burntool} program identifies bright sources on the image, and +identifies persistence trails that result from the incomplete transfer +of charge. As this charge can leak out in subsequent exposures, the +burntool analysis is run sequentially on the exposures, based on the observation date and time listed in the headers, with the results -stored in an text table. As a result of the sequential nature of this -analysis, the \ippstage{registration} of exposures is blocked until the -\ippprog{burntool} has been run on the previous exposures. +stored on disk. As a result of the sequential nature of this +analysis, the \ippstage{registration} of exposures is blocked until +the \ippprog{burntool} has been run on the previous exposures. Once the \ippstage{registration} process has finished, new science @@ -591,15 +605,15 @@ majority of stages operate on smaller segments of a full exposure, allowing the processing tasks to be spread over the machines in the -processing cluster. The \ippprog{pantasks} environment, which manages -the jobs, attempts to target the processing to a computer which is -assigned to host data for the particular OTA. This capability is -implemented to reduce the network I/O load by minimizing the number of -operations done on non-local data. In practice, this targeted -processing has not had as large of an impact as was originally -intended: the data volume and operational details of the hardware has -reduced the ability of any one node to reliably contain a particular -OTA. The targeted processing has probably reduced the network load -somewhat but it has not been as critical of a requirement as -originally expected. +processing cluster. The \ippprog{pantasks} environment (the system +which manages the processing jobs, see Section~\ref{sec:pantasks}) +attempts to target the processing to a computer which is assigned to +host data for the particular OTA. This capability is implemented to +reduce the network I/O load by minimizing the number of operations +done on non-local data. In practice, this targeted processing has not +had as large of an impact as was originally intended: the data volume +and operational details of the hardware has reduced the ability of any +one node to reliably contain a particular OTA. The targeted +processing has probably reduced the network load somewhat but it has +not been as critical of a requirement as originally expected. %% In the \ippstage{chip} stage, @@ -623,12 +637,12 @@ program. This program reads the raw data into memory and applies the detrend corrections \citep[see][]{waters2017} to each cell in the OTA -(which are stored as different extensions in the FITS file format), -and then mosaics the cells into a single contiguous \ippstage{chip} -stage image. This step also creates in memory additional images to -hold the mask data, which indicates which pixels may not be valid, and -the variance image, constructed as the Poissonian noise on the number -of electrons detected based on the original pixel value and the -detector gain. A background model is then fit across the image and -subtracted to remove the expected contribution from the sky +(stored as different extensions in the FITS file format), and then +mosaics the cells into a single contiguous \ippstage{chip} stage +image. This step also creates in memory additional images to hold the +mask data, which indicates which pixels may not be valid, and the +variance image, constructed as the Poissonian noise on the number of +electrons detected based on the original pixel value and the detector +gain. A background model is then fit across the image and subtracted +to remove the expected contribution from the sky \citep[see][]{waters2017} for details. @@ -706,5 +720,5 @@ The guess astrometry is used to match the reference catalog to the observed stellar positions in the focal plane coordinate system -\citep[see][]{magnier2017.calibration}). +\citep[see][]{magnier2017.calibration}. Once an acceptable match is found, the astrometric calibration of the @@ -838,9 +852,9 @@ generated for the nightly groups and for the full depth using all exposures, producing ``deep stacks''. In addition, a ``best seeing'' -set of stacks have been produced \note{using image quality cuts to be - described: need input from MEH}. We have also generated -out-of-season stacks for the Medium Deep fields, in which all images -not from a particular observing season for a field are combined into a -stack. These later stacks are useful as deep templates when studying +set of stacks have been produced using image quality cuts described by +\citet[][Paper VII]{huber2017}. We have also generated out-of-season +stacks for the Medium Deep fields, in which all images {\em not} from a +particular observing season for a field are combined into a stack. +These later stacks are useful as deep templates when studying long-term transient events in the Medium Deep fields as they are not (or less) contaminated by the flux of the transients from a given @@ -879,12 +893,12 @@ Although images are generated in the \ippstage{stack} stage of the -IPP, the source detection and extraction analysis of those images is -deferred to the \ippstage{staticsky} stage. This separation is -maintained because the photometry analysis of the \ippstage{stack} -images, including convolved galaxy model fitting, is performed on all -5 filters simultaneously. By deferring this analysis, the processing -system may also decouple the generation of the pixels from the source -detection. This makes the sequencing of analysis somewhat easier and -less subject to blocks due to a failure in the stacking analysis. +IPP, the source detection and analysis of those images is deferred to +the \ippstage{staticsky} stage. This separation is maintained because +the photometry analysis of the \ippstage{stack} images, including +convolved galaxy model fitting, is performed on all 5 filters +simultaneously. By deferring this analysis, the processing system may +also decouple the generation of the pixels from the source detection. +This makes the sequencing of analysis somewhat easier and less subject +to blocks due to a failure in the long-running stacking analysis. Similar to the \ippstage{stack} stage, an entry is created in the \ippdbtable{staticskyRun} table, linked to a series of rows in the @@ -893,20 +907,19 @@ entries for the skycell under consideration. -The input images are passed to the \ippprog{psphotStack} program, -which does the analysis. The stack photometry algorithms are -described in detail in \cite{magnier2017.analysis}. In short, sources are -detected in all 5 filter images down to the $5\sigma$ significance. -The collection of detected sources is merged into a single master -list. If a source is detected in at least two bands, or only in -\yps{} band, then a PSF model is fitted to the pixels of the other -bands in which the source was not detected. This forced photometry -results in lower significance measurements of the flux at the -positions of objects which are thought to be real sources, by virtue -of triggering a detection in at least two bands. The relaxed limit -for \yps{} band is included to allow for searches of \yps{} dropout -objects: it is known that faint, high-redshift quasars may be detected -in \yps{} band only. Sources detected only in \yps{} band are -therefore more likely to have a higher false-positive rate than the -other stack sources. +The input images are passed to the \ippprog{psphotStack} program which +does the analysis. The stack photometry algorithms are described in +detail in \cite{magnier2017.analysis}. In short, sources are detected +in all 5 filter images down to the $5\sigma$ significance. The +collection of detected sources is merged into a single master list. +If a source is detected in at least two bands, or only in \yps{} band, +then a PSF model is fitted to the pixels of the other bands in which +the source was not detected. This forced photometry results in lower +significance measurements of the flux at the positions of objects +which are thought to be real sources, by virtue of triggering a +detection in at least two bands. The relaxed limit for \yps{} band is +included to allow for searches of \yps{} dropout objects: it is known +that faint, high-redshift quasars may be detected in \yps{} band only. +Sources detected only in \yps{} band are therefore more likely to have +a higher false-positive rate than the other stack sources. The stack photometry output files consist of a set of FITS table @@ -946,6 +959,4 @@ \subsection{Forced Warp Photometry} \label{sec:fullforce} - -\note{too much detail in this section; balance relative to psphot} Traditionally, projects which use multiple exposures to increase the @@ -977,5 +988,5 @@ degraded. The highly textured PSF variations make this a very challenging problem: not only would such a PSF model need to be highly -fine-grained, there would likely not be enough PSF stars in a given +fine-grained, there would likely not be enough stars in a given \ippstage{stack} image to determine the model at the resolution required. The IPP photometry analysis code uses a PSF model with 2D @@ -986,5 +997,5 @@ images. -Thus PSF photometry as well as convolved galaxy models in the stack +Thus PSF photometry and convolved galaxy model analysis in the stack are degraded by the PSF variations. Aperture-like measurements are in general not as affected by the PSF variations, as long as the aperture @@ -1043,6 +1054,5 @@ the PSF-convolved galaxy models are of limited accuracy. -Upon completion of the forced photometry (for point sources as well as -galaxies, discussed below), an entry is added to the +Upon completion of the forced photometry, an entry is added to the \ippdbtable{fullForceResult} table with the processing statistics for that combination of \ippdbcolumn{ff_id} and \ippdbcolumn{warp_id}. @@ -1075,5 +1085,5 @@ epoch. The quality of such a difference image can be enhanced by convolving one or both of the images so that the PSFs in the two -images are matched. \note{discuss Alard-Lupton}. +images are matched \citep[e.g.,][]{AlardLupton}. In the \ippstage{diff} stage, the IPP generates difference images for @@ -1082,16 +1092,17 @@ images, from a \ippstage{warp} and a \ippstage{stack} of some variety, or from a pair of \ippstage{stack} stage images. During the PS1 -survey, pairs of exposures, called TTI pairs (see~\note{Survey - Strategy in Chambers et al}), were obtained for each pointing within a $\approx$ 1 -hour period in the same filter, and to the extent possible with the -same orientation and boresite position. The standard PS1 nightly -processing generated difference images from the resulting pairs of -\ippstage{warp} images. The nightly processing generated -\ippstage{stack} images for the Medium Deep fields, and these were -combined with a template reference \ippstage{stack} image to generate -``stack-stack diffs'' each night they were observed. For the PV3 -$3\pi$ processing, the entire collection of \ippstage{warp} stage -images for the survey were combined with images generated by the -\ippstage{stack} processing to generate ``warp-stack diffs''. +survey, pairs of exposures, called TTI pairs \citep[see Survey + Strategy in][]{chambers2017}, were obtained for each pointing within +a $\approx$ 1 hour period in the same filter, and to the extent +possible with the same orientation and boresite position. The +standard PS1 nightly processing generated difference images from the +resulting pairs of \ippstage{warp} images. The nightly processing +generated \ippstage{stack} images for the Medium Deep fields, and +these were combined with a template reference \ippstage{stack} image +to generate ``stack-stack diffs'' each night they were observed. For +the PV3 $3\pi$ processing, the entire collection of \ippstage{warp} +stage images for the survey were combined with images generated by the +\ippstage{stack} processing to generate ``warp-stack diffs'', for +eventual public released. When a \ippstage{diff} processing is defined, an entry is added to the @@ -1132,11 +1143,7 @@ \label{sec:postprocessing} -\note{introduction to this section: data ingested into DVO database, - database gets calibrated, data ingested into PSPS via IPP to PSPS} - \begin{table}[hb] \begin{center} -\caption{DVO Database Tables\label{tab:DVO_schema} \note{fix order, - drop invalid tables}} +\caption{DVO Database Tables\label{tab:DVO_schema} \note{fix names, include missing}} \begin{tabular}{ll} \hline @@ -1145,17 +1152,17 @@ \hline Images & The images that have objects in the DB. \\ -Image Overlaps & Image regions which are touched by specific images. \\ +% Image Overlaps & Image regions which are touched by specific images. \\ Objects & The objects --- average properties of multiple detections of the same object. \\ -Average Magnitudes & Average photometry in multiple filters \\ -Solar System Objects & Identification of solar system objects \\ -Matched Detections & Detections of sources in an image identified with an Object. \\ -Orphaned Detections & Detections of sources in an image not identified with an Object. \\ -Non-detections & Non-detections of objects in an image. \\ +Average & Average photometry in multiple filters \\ +% Solar System Objects & Identification of solar system objects \\ +Measure & Detections of sources in an image identified with an Object. \\ +% Orphaned Detections & Detections of sources in an image not identified with an Object. \\ +% Non-detections & Non-detections of objects in an image. \\ SkyRegions & spatial distribution of tables \\ -Filters & Filters understood by the system. \\ +% Filters & Filters understood by the system. \\ Photcodes & Transformations between different photometric systems \\ -Zero Points & History of Zero-point \& Airmass terms \\ -Distortion Models & History of Optical Distortion terms \\ -Database Hosts & computers used to store the tables \\ +% Zero Points & History of Zero-point \& Airmass terms \\ +% Distortion Models & History of Optical Distortion terms \\ +Hosts & computers used to store the tables \\ \hline \end{tabular} @@ -1210,18 +1217,18 @@ which store supporting information (metadata). -DVO includes two major classes of database tables: those containing -information about astronomical objects in the sky and those containing -other supporting information. The object-related tables are -partitioned on the basis of position in the sky: objects within a -region bounded by lines of constant RA,DEC are contained in a specific -file. The boundaries and the associated partition names are stored in -one of the supporting tables, \ippdbtable{SkyTable}. This table -contains the definitions of the boundaries for each sky region -(\ippdbcolumn{R_MIN}, \ippdbcolumn{R_MAX}, \ippdbcolumn{D_MIN}, -\ippdbcolumn{D_MAX}), the name of the sky region, an ID -(\ippdbcolumn{INDEX}, equal to the sequence number of the region in -the table), and index entries to enable navigation within the table. -The regions are defined in a hierarchical sense, with a series of -levels each containing a finer mesh of regions covering the sky. +%% DVO includes two major classes of database tables: those containing +%% information about astronomical objects in the sky and those containing +%% other supporting information. The object-related tables are +%% partitioned on the basis of position in the sky: objects within a +%% region bounded by lines of constant RA,DEC are contained in a specific +%% file. The boundaries and the associated partition names are stored in +%% one of the supporting tables, \ippdbtable{SkyTable}. This table +%% contains the definitions of the boundaries for each sky region +%% (\ippdbcolumn{R_MIN}, \ippdbcolumn{R_MAX}, \ippdbcolumn{D_MIN}, +%% \ippdbcolumn{D_MAX}), the name of the sky region, an ID +%% (\ippdbcolumn{INDEX}, equal to the sequence number of the region in +%% the table), and index entries to enable navigation within the table. +%% The regions are defined in a hierarchical sense, with a series of +%% levels each containing a finer mesh of regions covering the sky. \subsubsubsection{Photcodes} @@ -1257,7 +1264,7 @@ transform a measurement in the specific photcode to a common system. For example, a \ippmisc{DEP} photcode GPC1.g.X01 would have the -nominal zero point (25.XX) and airmass term (0.14). The structures +nominal zero point (24.563) and airmass term (0.147). The structures allow for individual chips to have different color terms to bring them -to a common filter system. +to a common filter system. Beyond the basic use, DVO has the ability to accept data from other @@ -1281,5 +1288,6 @@ processed by the IPP may also be included similarly in a DVO database. Measurements from other sources, such as SDSS, 2MASS, or WISE, can -also be included in this table. +also be included in this table, distinguished by their different +photcodes. The \ippdbtable{Measure} table includes the instrumental magnitudes @@ -1296,12 +1304,12 @@ discussed below) and the astrometrically calibrated position. Astrometric offsets for several systematic corrections discussed below -are also defined for each measurement. Photometry from \ippstage{chip}, \ippstage{warp}, -and \ippstage{stack} are all placed in the same table with photcodes -distinguishing the source \note{show example of stack and warp - photcodes}. Since stacks and forced warp fluxes may have -non-significant values, the table is somewhat de-normalized: it also -carries both magnitudes as well as instrumental flux values for the -PSF, aperture, and Kron photometry. In this case, we have chosen to -trade storage space for computing time. +are also defined for each measurement. Photometry from +\ippstage{chip}, \ippstage{warp}, and \ippstage{stack} are all placed +in the same table with photcodes distinguishing the source. Since +stacks and forced warp fluxes may have non-significant values, the +table is somewhat de-normalized: it also carries both magnitudes as +well as instrumental flux values for the PSF, aperture, and Kron +photometry. In this case, we have chosen to trade storage space for +computing time. For the warp images, we also measure the weak lensing KSB parameters @@ -1310,15 +1318,16 @@ along with the radial aperture fluxes for radii numbers 5, 6, \& 7 (respectively 3.0, 4.63, and 7.43 arcsec). This table contains one -row for every warp row. \note{warp row hasn't been defined anywhere.} -Similarly to the \ippdbtable{Measure} table, the fields -\ippdbcolumn{objID}, \ippdbcolumn{catID}, and \ippdbcolumn{averef} -define links from the \ippdbtable{Lensing} table to the -\ippdbtable{Average} table. In a similar fashion, the fields -\ippdbtable{Average}.\ippdbcolumn{lensingOffset} and -\ippdbtable{Average}.\ippdbcolumn{Nlensing} are the index into the -sorted \ippdbtable{Lensing} table entries. \note{discuss failure of - the Lensing to Measure indexing} - -\note{Average used above but defined below} +row for every warp image on which the object was measured. + +%% Similarly to the \ippdbtable{Measure} table, the fields +%% \ippdbcolumn{objID}, \ippdbcolumn{catID}, and \ippdbcolumn{averef} +%% define links from the \ippdbtable{Lensing} table to the +%% \ippdbtable{Average} table. In a similar fashion, the fields +%% \ippdbtable{Average}.\ippdbcolumn{lensingOffset} and +%% \ippdbtable{Average}.\ippdbcolumn{Nlensing} are the index into the +%% sorted \ippdbtable{Lensing} table entries. \note{discuss failure of +%% the Lensing to Measure indexing} + +% \note{Average used above but defined below} \subsubsubsection{Object Tables} @@ -1332,7 +1341,9 @@ new detections are loaded, they are compared to the objects already stored in the database. If an object is already found in the database -within the match radius of \note{one arcsecond}, the new detection is -assigned to that object. If more than one object exists within the -database, the detection is associated with the closest object. +within the match radius, the new detection is assigned to that +object. If more than one object exists within the database, the +detection is associated with the closest object. For most data +sources, a match radius of 1.0 arcsecond is used, but this may be +adjusted in special cases. Two tables carry the most important information about the astronomical @@ -1343,5 +1354,5 @@ \pi$) and associated errors, data quality flags for each object, links to the other tables, and a number of IDs, with one row for each -astronomical object. \note{go into complete detail here on the IDs?}. +astronomical object. The \ippdbtable{SecFilt} table\footnote{The name \ippdbtable{SecFilt} is a bit of a historical misnomer: originally, DVO was designed for @@ -1391,5 +1402,5 @@ The \ippdbtable{Starpar} table carries measurements provide by Greg Green \& Eddie Schlafly from their analysis of the SED of objects in -the PS1 $3\pi$ data, using the \note{PV1?} version of the analysis +the PS1 $3\pi$ data, using the PV1 version of the analysis \citep{2015ApJ...810...25G}. In this work, the goal was a 3D model of the dust in the Galaxy based on Pan-STARRS and 2MASS photometry. As @@ -1399,14 +1410,16 @@ these photometric distance modulus measurements are not extremely precise (see below), they provide a constraint on the distance is used -in our analysis of the astrometry \citep[see][]{magnier2017.calibration}. +in our analysis of the astrometry +\citep[see][]{magnier2017.calibration}. In the \ippdbtable{Measure} table, there are three fields which provide two independent links from the specific measurement to the associated object: \ippdbtable{Measure}.\ippdbcolumn{catID} specifies -the spatial partition to which the measurement belongs; +the spatial partition to which the measurement belongs (see +Section~\ref{sec:SkyPartition} below); \ippdbtable{Measure}.\ippdbcolumn{objID} specifies to which entry in the \ippdbtable{Average} table the measurement belongs. These two 32 bit fields can thus be combined into a single 64 bit ID unique for all -objects in the database. \note{PSPS IDs} In addition, the field +objects in the database. In addition, the field \ippdbtable{Measure}.\ippdbcolumn{averef} specifies the row number in the \ippdbtable{Average} table of the associated object. The @@ -1421,4 +1434,6 @@ field \ippdbtable{Measure}.\ippdbcolumn{imageID} defines the link from the measurement to the image which supplied the measurement. + +\note{Discuss PSPS IDs} \subsubsubsection{Image Tables} @@ -1460,9 +1475,8 @@ \subsubsection{Sky Partition} - -\note{re-word this sentence} DVO includes two major classes of database tables: those containing -information about astronomical objects in the sky and those containing -other supporting information. The object-related tables are -partitioned on the basis of position in the sky: objects within a +\label{sec:SkyPartition} + +Tables within DVO containing information about astronomical objects +are partitioned on the basis of position in the sky: objects within a region bounded by lines of constant RA,DEC are contained in a specific file. The boundaries and the associated partition names are stored in @@ -1478,56 +1492,51 @@ In the default used by the PV3 DVO, the partitioning scheme is based on the one used by the Hubble Space Telescope Guide Star Catalog -files. \note{add figure} Level 0 is a single region covering the full -sky. Level 1 divides the sky in declination into bands -7.5\degree\ high. Level 2 subdivides these declination bands in the -RA direction, with spacing related to the stellar density. Level 3 +files. Level 0 is a single region covering the full sky. Level 1 +divides the sky in declination into bands 7.5\degree\ high, as defined +by the HST GSC. Level 2 subdivides these declination bands in the RA +direction, with spacing related to the stellar density. Level 3 divides these RA chunks into 4 - 8 smaller partitions. This level exactly matches the HST GSC layout, and uses the same naming -convention to identify the partitions: \code{n0000/0000}, -etc. \note{more on the names?}. Level 4 further divides these regions -by a factor of 16. In the \ippdbtable{SkyTable}, a region at one -level has a pointer to its parent region (the one which contains it) -and a sequence pointing to its children (regions it contains). The -\ippdbtable{SkyTable} enables fast lookups of the on-disk partitions -which map to a specific coordinate on the sky. In general, a single -DVO will have the full sky represented with tables at a single -level. Although it is possible for mixed levels to be used, this mode -is not well tested and is avoided in the PV3 DVO database. For the -PV3 master database, the partitioning at the \note{should this be - 4th?} 5th level results in \approx 150,000 regions to cover the full -sky, of which \approx 110,000 are used for the PV3 $3\pi$ data. The -densest portions of the bulge contain at most \approx 300,000 -astronomical objects in the database files, with an associated maximum -of \approx 30 million measurements in these files. With the compression -scheme described below, the largest database files are \approx -3GB, which can be loaded into memory in 30 seconds on the processing -machines that contain partition data. - -\note{is the use of the term `partition host' consistent in this paper - and the calibration paper?} +convention to identify the partitions: \code{n0000/0000}, etc. Level 4 +further divides these regions by a factor of 16. In the +\ippdbtable{SkyTable}, a region at one level has a pointer to its +parent region (the one which contains it) and a sequence pointing to +its children (regions it contains). The \ippdbtable{SkyTable} enables +fast lookups of the on-disk partitions which map to a specific +coordinate on the sky. In general, a single DVO will have the full +sky represented with tables at a single level, although it is possible +for mixed levels to be used. For the PV3 master database, the +partitioning is at Level 4, resulting in \approx 150,000 regions to +cover the full sky, of which \approx 110,000 are used for the PV3 +$3\pi$ data. The densest portions of the bulge contain at most +\approx 300,000 astronomical objects in the database files, with an +associated maximum of \approx 30 million measurements in these files. +With the compression scheme described below, the largest database +files are \approx 3GB, which can be loaded into memory in 30 seconds +on the processing machines that contain partition data. % parallel partitions The DVO software system allows the tables which are partitioned across the sky to also be distributed across multiple computers, which we -call partition hosts. A single file defines the names of these -partition hosts and the location of the database partition on the -disks of that machine. The \ippdbtable{SkyTable} contains elements to -define by ID the parition host to which a partitioned set of tables -has been assigned. Operations which query the database, or perform -other operations on the database, are aware of the partitioning scheme -and will launch their operations as remote processes on the machines -which contain the data they need. For example, a query for data from -a small region will launch sub-query operations on the machines which -contain the data overlapping the region of interest. These remote -query operations will select the database information which matches -the query request (i.e., applying restrictions as defined) and return -the results to the master process. The results from the various -partition hosts are then merged into a single result by the master -process. When the parallel partitioning for a DVO instance is -defined, the tables are randomly assigned to the partition hosts. As -a result, queries which span more than a single parition are likely to -spread the I/O load across a large number of machines. This -parallelization is critical to querying and manipulating the enormous -database on a reasonable timescale. +call partition hosts. A single file identifies these partition hosts +and the location of the database partition on the disks of that +machine. The \ippdbtable{SkyTable} contains elements to define by ID +the parition host to which a set of tables has been assigned. +Operations which query the database, or perform other operations on +the database, are aware of the partitioning scheme and will launch +their operations as remote processes on the machines which contain the +data they need. For example, a query for data from a small region +will launch sub-query operations on the machines which contain the +data overlapping the region of interest. These remote query +operations will select the database information which matches the +query request (i.e., applying restrictions as defined) and return the +results to the master process. The results from the various partition +hosts are then merged into a single result by the master process. +When the parallel partitioning for a DVO instance is defined, the +tables are randomly assigned to the partition hosts. As a result, +queries which span more than a single parition are likely to spread +the I/O load across a large number of machines. This parallelization +is critical to querying and manipulating the enormous database on a +reasonable timescale. \subsubsection{DVO Data Storage} @@ -1537,7 +1546,7 @@ the database tables are stored on disk using binary FITS tables. Each type of database table is stored as a separate file, or a collection -of files for table which are spatially partitioned. The binary FITS +of files for tables which are spatially partitioned. The binary FITS tables are compressed using the (to date) experimental FITS binary -table compression strategy outlined by \note{REF}. Table compression +table compression strategy outlined by \citet{RickWhite}. Table compression is an option in DVO; for the PV3 database, the large data volume (70TB compressed) drove the decision to compress the tables. @@ -1587,12 +1596,11 @@ is associated with the the \ippstage{addstar} processing stage. The measurement catalogs generated by the \ippstage{camera}, -\ippstage{staticsky}, \ippstage{skycal}, \ippstage{fullforce}, and -\ippstage{diff} stages are processed loaded into DVOs in this fashion, -although not every measurement in each catalog are included in the -master DVO that is constructed. For a particular re-processing -version, a single master DVO is constructed for the positive image -stages (\ippstage{camera}, \ippstage{staticsky}, \ippstage{skycal}, -\ippstage{fullforce}) and a separate one is constructed for the -difference image analysis stage results. +\ippstage{skycal}, \ippstage{fullforce}, and \ippstage{diff} stages +are loaded into DVOs in this fashion, although not every measurement +in each catalog are included in the master DVO that is constructed. +For a particular re-processing version, a single master DVO is +constructed for the positive image stages (\ippstage{camera}, +\ippstage{skycal}, \ippstage{fullforce}) and a separate one is +constructed for the difference image analysis stage results. The construction of the master DVO is performed in a hierarchical @@ -1620,14 +1628,14 @@ some stages, such as the \ippstage{diff} stage, create more than a single catalog, multiple entries with the \ippdbcolumn{stage_id} are -created, with the \ippdbcolumn{stage_extra1} field containing an -index to the individual components. The catalog specified by the -entry is added to the target \ippmisc{minidvo} by the -\ippprog{addstar} program, \note{describe what's done?}. When this -completes, an entry containing the statistics of the job is added to -the \ippdbtable{addProcessedExp} table. +created, with the \ippdbcolumn{stage_extra1} field containing an index +to the individual components. The catalog specified by the entry is +added to the target \ippmisc{minidvo} by the \ippprog{addstar} +program, updating the measurements in the appropriate DVO tables. +When this completes, an entry containing the statistics of the job is +added to the \ippdbtable{addProcessedExp} table. After the master DVO is contructed containing the PS1 data, data from other sources are also added to the database. For the PV3 DVO -database, data was added from 2MASS, WISE, Gaia, and Tycho. These +database, data was added from 2MASS, WISE, Gaia DR1, and Tycho. These external data sources are added by first generating a DVO database containing just the particular data source, then using the same DVO @@ -1665,6 +1673,4 @@ astrometry is again performed this time using the corrected positions. -\note{have eddie suggest wording here?} - Photometric calibration consists of determination of zero points for each exposure along with corrections for systematic effects. In this @@ -2122,8 +2128,4 @@ \label{sec:automation} -\note{start with a discussion of the standard sequencing (end-stage)} - -\note{then discuss the addstar sequences with manual triggering} - Outside of the basic sequence of \ippstage{chip} to \ippstage{warp}, there is no single natural next step. For example: a stack can be generated with any @@ -2136,5 +2138,5 @@ \ippmisc{ippScript}. These scripts have a well-defined and restricted set of goals: to ensure that difference images are generated for each -exposures (either by pairing together warps or pairs warps with +exposure (either by pairing together warps or pairs warps with pre-defined stacks), that nightly stacks are generated for MD fields, and that the stacks are also differenced against an appropriate @@ -2190,9 +2192,5 @@ The automatic nightly processing ensures that data is processed as soon as it is downloaded from the summit, reducing the lag between an -observation and the reduced data. \note{some numbers here about - completion times and such? Words about getting data to MOPS and SN - transient folks} - -\note{re-read paragraph below and cleanup} +observation and the reduced data. The other processing task that requires automation is the reprocessing @@ -2208,5 +2206,5 @@ unit of sky defined to be a square four degrees on each side which has a single tangent plane projection \citep[][see]{waters2017}. -\note{does waters2017 discuss RINGS.V3? if not, where?} Once this +Once this entry is defined, it is populated with all exposures (stored in the \ippdbtable{lapExp} table in the database) that are located @@ -2268,6 +2266,5 @@ hardware acquisition, occasional hardware failures, and other organizational details, targeted processing has only been used to a -moderate degree within the Pan-STARRS cluster. \note{can we get a - number here?} +moderate degree within the Pan-STARRS cluster. \subsubsection{Implementation Details} @@ -2276,5 +2273,5 @@ well as APIs in both C and Perl. -"The basic user commands to interact with Nebulous are to 1) query the +The basic user commands to interact with Nebulous are to 1) query the database for an existing storage object, and find a valid file instance associated with that object; 2) create a new storage object, @@ -2408,20 +2405,19 @@ Transferring data between the IPP and other parts of the Pan-STARRS system is generally accomplished via a ``datastore'', an http service -that exposes data in a common form. \note{add Isani / Hoblitt - reference?} One of the main datastores used by the IPP is the one -located at the summit. This datastore exposes a list of the -exposures obtained since the start of the PS1 operations. Requests to -this server may restrict to the latest by time. Each row in the -listing includes basic information about the exposure: an exposure -identifier (e.g., o5432g0123o; see~\ref{GPC1.names} for details), the -date and time of the exposure, the telescope commanded pointing, the -filter and exposure time, and the observation comment for that -exposure. The row also provides a link to a listing of the chips -associated with that exposure. This listing includes a link to the -individual chip FITS files as well as an md5 checksum. Systems which -are allowed access may download the raw chip FITS files via http requests to -the provided links. - -\note{add a discussion of gpc1 filenames?} +that exposes data in a common form. One of the main datastores used +by the IPP is the one located at the summit. This datastore exposes a +list of the exposures obtained since the start of the PS1 operations. +Requests to this server may restrict to the latest by time. Each row +in the listing includes basic information about the exposure: an +exposure identifier (e.g., o5432g0123o; see~\ref{GPC1.names} for +details), the date and time of the exposure, the telescope commanded +pointing, the filter and exposure time, and the observation comment +for that exposure. The row also provides a link to a listing of the +chips associated with that exposure. This listing includes a link to +the individual chip FITS files as well as an md5 checksum. Systems +which are allowed access may download the raw chip FITS files via http +requests to the provided links. + +% \note{add a discussion of gpc1 filenames?} The IPP also uses datastores to provide access to its own data @@ -2543,10 +2539,6 @@ library. -\note{This likely needs cleaning up and more information.} - \section{IPP Hardware Systems} \label{sec:hardware} - -\note{what about psps hardware? mops hardware?} \subsection{Kihei Processing Cluster} @@ -2567,10 +2559,9 @@ with a variety of individual specifications due to the cluster being assembled from multiple generations of purchases. The data storage -nodes contain \note{check, as this contains B nodes} approximately 10 -petabytes of storage space that are used to store both the raw -exposure data downloaded from the telescope as well as processed data -products. These nodes are also used to do processing, and have jobs -targeted to them in an effort to reduce the network I/O demands -(see~\ref{sec:chip} for more on this process). +nodes contain several petabytes of storage space that are used to +store both the raw exposure data downloaded from the telescope as well +as processed data products. These nodes are also used to do +processing, and have jobs targeted to them in an effort to reduce the +network I/O demands (see~\ref{sec:chip} for more on this process). These storage nodes are not fully capable of completing all processing @@ -2584,10 +2575,11 @@ The final type of computer in the cluster are the database servers. -These special purpose computers \note{have lots of memory and disk - space? Is that it?} are used to store and manage both the IPP gpc1 -and \ippprog{Nebulous} databases. In addition to the main master -servers, we have duplicate servers used as database replicants, which -allow for quick switching from the main to backup servers in case of a -hardware issue that cannot be resolved immediately. +These computers have large memory capacity and high-speed disk access +(originally fast spindle spinning disks, now migrated to SSDs) are +used to store and manage both the IPP gpc1 and \ippprog{Nebulous} +databases. In addition to the main master servers, we have duplicate +servers used as database replicants, which allow for quick switching +from the main to backup servers in case of a hardware issue that +cannot be resolved immediately. \subsection{Los Alamos National Labs} @@ -2643,12 +2635,30 @@ values used for the various IPP processing stages. -\begin{table} -\caption{\label{tab:SC_processing_parameters} Cost values for remote processing}\vspace{-0.5cm} -\begin{center} -\begin{tabular}{lcc} -\hline -\hline -{\bf IPP Stage} & {\bf $t_\mathrm{task}$ (s)} & {\bf $S_\mathrm{task}$} \\ -\hline +%% \begin{table} +%% \caption{\label{tab:SC_processing_parameters} Cost values for remote processing}\vspace{-0.5cm} +%% \begin{center} +%% \begin{tabular}{lcc} +%% \hline +%% \hline +%% {\bf IPP Stage} & {\bf $t_\mathrm{task}$ (s)} & {\bf $S_\mathrm{task}$} \\ +%% \hline +%% \ippstage{chip} & 150 & 2 \\ +%% \ippstage{camera} & 1700 & 2 \\ +%% \ippstage{warp} & 110 & 2 \\ +%% \ippstage{stack} & 1500 & 6 \\ +%% \ippstage{staticsky} & 7200 & 6 \\ +%% % \ippstage{diff} & 300 & 2 \\ +%% \ippstage{fullforce} & 300 & 2 \\ +%% \hline +%% \end{tabular} +%% \end{center} +%% \end{table} + +\begin{deluxetable}{lcc} + \tablecolumns{3} + \tablewidth{0pc} + \tablecaption{Cost values for remote processing} + \tablehead{\colhead{IPP Stage}&\colhead{$t_\mathrm{task}$ (s)}&\colhead{$S_\mathrm{task}$}} + \startdata \ippstage{chip} & 150 & 2 \\ \ippstage{camera} & 1700 & 2 \\ @@ -2657,26 +2667,8 @@ \ippstage{staticsky} & 7200 & 6 \\ % \ippstage{diff} & 300 & 2 \\ - \ippstage{fullforce} & 300 & 2 \\ -\hline -\end{tabular} -\end{center} -\end{table} - -%% \begin{deluxetable}{lcc} -%% \tablecolumns{3} -%% \tablewidth{0pc} -%% \tablecaption{Cost values for remote processing} -%% \tablehead{\colhead{IPP Stage}&\colhead{$t_\mathrm{task}$ (s)}&\colhead{$S_\mathrm{task}$}} -%% \startdata -%% \ippstage{chip} & 150 & 2 \\ -%% \ippstage{camera} & 1700 & 2 \\ -%% \ippstage{warp} & 110 & 2 \\ -%% \ippstage{stack} & 1500 & 6 \\ -%% \ippstage{staticsky} & 7200 & 6 \\ -%% % \ippstage{diff} & 300 & 2 \\ -%% \ippstage{fullforce} & 300 & 2 -%% \enddata -%% \label{tab:SC processing parameters} -%% \end{deluxetable} + \ippstage{fullforce} & 300 & 2 + \enddata + \label{tab:SC processing parameters} +\end{deluxetable} Once the preparation for the job is complete, the input and output @@ -2764,24 +2756,16 @@ %\input{datasystem.bbl} -\appendix - -\section{GPC1 Database Schema Outline} -\label{sec:database.schema} - -Table \ref{tab: database schema} provides a list of a majority of the -tables in the GPC1 database schema. Tables that have been excluded -are either no longer used in IPP processing, or are used for rare -reductions that were not used for the PV3 data release. The tables -are grouped into stages, with the primary table and any secondary -tables for that stage listed together, along with the primary key -column that link the tables together. - -\note{logical or alphabetical sequence?} +% \appendix + +% Table \ref{tab: database schema} provides a list of a majority of the +% tables in the GPC1 database schema. Tables that have been excluded +% are either no longer used in IPP processing, or are used for rare +% reductions that were not used for the PV3 data release. The tables +% are grouped into stages, with the primary table and any secondary +% tables for that stage listed together, along with the primary key +% column that link the tables together. \end{document} -Figures needed for this document: - -* \begin{center} \begin{deluxetable}{lllll}