Index: trunk/doc/release.2015/ps1.datasystem/datasystem.tex =================================================================== --- trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 40714) +++ trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 40715) @@ -1100,84 +1100,5 @@ eventual public released. -When a \ippstage{diff} processing is defined, an entry is added to the -\ippdbtable{diffRun} table, and the appropriate input images are added -to the \ippdbtable{diffInputSkyfile} table, with one entry for each -skycell that is covered by the images. For a \ippstage{diff} -generated from two \ippstage{warp} stage products, the input images -have their \ippdbcolumn{warp_id} values recorded in the -\ippdbcolumn{warp1} and \ippdbcolumn{warp2} for each skycell that -overlaps. If two \ippstage{stack} stages are to be used in the -difference, their \ippdbcolumn{stack_id} entries are recorded in the -\ippdbcolumn{stack1} and \ippdbcolumn{stack2} fields. As each -\ippstage{stack} only covers a single skycell, the \ippstage{diff} is -usually defined indirectly, using other information from the -\ippdbtable{stackRun} table to select appropriate -\ippdbcolumn{stack_id} values. Similarly, \ippstage{diff} processing -is defined for the mixed case by creating entries that populate one of -\ippdbcolumn{warp1} and \ippdbcolumn{stack1} and populating one of -\ippdbcolumn{warp2} and \ippdbcolumn{stack2}. In all cases, the -minuend of the subtraction to be performed is the ``1'' entry, and the -subtrahend is the ``2'' entry. - -Jobs are created based on the entries of -\ippdbtable{diffInputSkyfile}, with the appropriate images and -catalogs passed to the \ippprog{ppSub} program. This does the -subtraction, as well as the photometry of any sources detected in the -\ippstage{diff} image. Sources may be detected as a positive source -(flux in the minuend is higher than the subtrahend) or as a negative -source (flux in the subtrahend is higher). The algorithm used for PSF -matching is described in Paper III. Upon completion of these -jobs, statistics about the processing are written to an entry in the -\ippdbtable{diffSkyfile} table. An \ippmisc{advance} checks for the -completion of all of the components listed in -\ippdbtable{diffInputSkyfile}, and marks the \ippdbtable{diffRun} -entry as such. - -\section{Post-Processing : Database Ingest and Calibration} -\label{sec:postprocessing} - -\subsection{DVO} -\label{sec:DVO} - -\subsubsection{Overview} - -% intro -The Pan-STARRS IPP uses an internal database system, distinct from the -publicly visible database system, to determine the association -between multiple detections of the same astronomical object and as -part of the astrometric and photometric calibration process. This -database system, called the ``Desktop Virtual Observatory'' (DVO) was -developed originally for the LONEOS project -\citep{1995DPS....27.0110B}, and used as part of the CFHT Elixir -system \citep{2004PASP..116..449M}. The capabilities of this -databasing system have been somewhat expanded for the Pan-STARRS -context. - -% overview -DVO tracks three main classes of information: 1) average properties of -astronomical objects; 2) measurements of those objects (from which the -average properties are derived); 3) properties of the images which -provided some or all of the measurements. In addition, certain -metadata tables define general features of the database. -Table~\ref{tab:DVO_schema} lists the full collection of database -tables used by DVO. - -In the most basic implementation, a collection of measurements for -detections from a set of images is loaded into DVO along with the -metadata describing the images. The latter includes properties such -as the exposure time, airmass, filter, time \& date of the exposure, -etc. Critically, the image metadata includes an astrometric -transformation relating the detection coordinate on the image to the -coordinate on the sky. As the collection of measurements are loaded -into DVO, the software constructs astronomical objects based on those -detections. If images overlap, multiple observations of the same -astronomical object are grouped together. Thus, a single DVO database -will contain a one-to-many relationship between the images and the -measurements and a many-to-one relationship between the measurements -and the derived astronomical objects. - -\subsubsection{DVO Schema} - -\begin{table*}[hb] +\begin{table*} \begin{center} \caption{DVO Database Tables\label{tab:DVO_schema}} @@ -1202,4 +1123,83 @@ \end{center} \end{table*} + +When a \ippstage{diff} processing is defined, an entry is added to the +\ippdbtable{diffRun} table, and the appropriate input images are added +to the \ippdbtable{diffInputSkyfile} table, with one entry for each +skycell that is covered by the images. For a \ippstage{diff} +generated from two \ippstage{warp} stage products, the input images +have their \ippdbcolumn{warp_id} values recorded in the +\ippdbcolumn{warp1} and \ippdbcolumn{warp2} for each skycell that +overlaps. If two \ippstage{stack} stages are to be used in the +difference, their \ippdbcolumn{stack_id} entries are recorded in the +\ippdbcolumn{stack1} and \ippdbcolumn{stack2} fields. As each +\ippstage{stack} only covers a single skycell, the \ippstage{diff} is +usually defined indirectly, using other information from the +\ippdbtable{stackRun} table to select appropriate +\ippdbcolumn{stack_id} values. Similarly, \ippstage{diff} processing +is defined for the mixed case by creating entries that populate one of +\ippdbcolumn{warp1} and \ippdbcolumn{stack1} and populating one of +\ippdbcolumn{warp2} and \ippdbcolumn{stack2}. In all cases, the +minuend of the subtraction to be performed is the ``1'' entry, and the +subtrahend is the ``2'' entry. + +Jobs are created based on the entries of +\ippdbtable{diffInputSkyfile}, with the appropriate images and +catalogs passed to the \ippprog{ppSub} program. This does the +subtraction, as well as the photometry of any sources detected in the +\ippstage{diff} image. Sources may be detected as a positive source +(flux in the minuend is higher than the subtrahend) or as a negative +source (flux in the subtrahend is higher). The algorithm used for PSF +matching is described in Paper III. Upon completion of these +jobs, statistics about the processing are written to an entry in the +\ippdbtable{diffSkyfile} table. An \ippmisc{advance} checks for the +completion of all of the components listed in +\ippdbtable{diffInputSkyfile}, and marks the \ippdbtable{diffRun} +entry as such. + +\section{Database Ingest and Calibration} +\label{sec:postprocessing} + +\subsection{DVO} +\label{sec:DVO} + +\subsubsection{Overview} + +% intro +The Pan-STARRS IPP uses an internal database system, distinct from the +publicly visible database system, to determine the association +between multiple detections of the same astronomical object and as +part of the astrometric and photometric calibration process. This +database system, called the ``Desktop Virtual Observatory'' (DVO) was +developed originally for the LONEOS project +\citep{1995DPS....27.0110B}, and used as part of the CFHT Elixir +system \citep{2004PASP..116..449M}. The capabilities of this +databasing system have been somewhat expanded for the Pan-STARRS +context. + +% overview +DVO tracks three main classes of information: 1) average properties of +astronomical objects; 2) measurements of those objects (from which the +average properties are derived); 3) properties of the images which +provided some or all of the measurements. In addition, certain +metadata tables define general features of the database. +Table~\ref{tab:DVO_schema} lists the full collection of database +tables used by DVO. + +In the most basic implementation, a collection of measurements for +detections from a set of images is loaded into DVO along with the +metadata describing the images. The latter includes properties such +as the exposure time, airmass, filter, time \& date of the exposure, +etc. Critically, the image metadata includes an astrometric +transformation relating the detection coordinate on the image to the +coordinate on the sky. As the collection of measurements are loaded +into DVO, the software constructs astronomical objects based on those +detections. If images overlap, multiple observations of the same +astronomical object are grouped together. Thus, a single DVO database +will contain a one-to-many relationship between the images and the +measurements and a many-to-one relationship between the measurements +and the derived astronomical objects. + +\subsubsection{DVO Schema} \paragraph{Photcodes} @@ -2594,4 +2594,24 @@ the Moab job control file was constructed. +\begin{table} +\caption{\label{tab:SC_processing_parameters} Cost values for remote processing} +\begin{center} +\begin{tabular}{lcc} +\hline +\hline +{\bf IPP Stage} & {\bf $t_\mathrm{task}$ (s)} & {\bf $S_\mathrm{task}$} \\ +\hline + \ippstage{chip} & 150 & 2 \\ + \ippstage{camera} & 1700 & 2 \\ + \ippstage{warp} & 110 & 2 \\ + \ippstage{stack} & 1500 & 6 \\ + \ippstage{staticsky} & 7200 & 6 \\ +% \ippstage{diff} & 300 & 2 \\ + \ippstage{fullforce} & 300 & 2 \\ +\hline +\end{tabular} +\end{center} +\end{table} + The control file contains the resource requests for the job, as well as the commands to be executed to complete it. The resource request @@ -2612,24 +2632,4 @@ values used for the various IPP processing stages. -\begin{table*} -\caption{\label{tab:SC_processing_parameters} Cost values for remote processing} -\begin{center} -\begin{tabular}{lcc} -\hline -\hline -{\bf IPP Stage} & {\bf $t_\mathrm{task}$ (s)} & {\bf $S_\mathrm{task}$} \\ -\hline - \ippstage{chip} & 150 & 2 \\ - \ippstage{camera} & 1700 & 2 \\ - \ippstage{warp} & 110 & 2 \\ - \ippstage{stack} & 1500 & 6 \\ - \ippstage{staticsky} & 7200 & 6 \\ -% \ippstage{diff} & 300 & 2 \\ - \ippstage{fullforce} & 300 & 2 \\ -\hline -\end{tabular} -\end{center} -\end{table*} - Once the preparation for the job is complete, the input and output file lists, the task list, and the job control file are transferred @@ -2690,6 +2690,4 @@ cluster as other users started to utilize the system, with 168,685 / 994,890 runs processed there. - -%% add a discussion of lessons-learned? \section{Conclusion}