Index: trunk/doc/release.2015/ps1.datasystem/Makefile =================================================================== --- trunk/doc/release.2015/ps1.datasystem/Makefile (revision 40612) +++ trunk/doc/release.2015/ps1.datasystem/Makefile (revision 40613) @@ -2,5 +2,5 @@ DO_PDFLATEX = 1 -DO_BIBTEX = 1 +DO_BIBTEX = 0 help: @@ -19,4 +19,6 @@ ../inputs/apj.bst \ ../inputs/lib.bib \ +PS1_Data_Analysis_System_Overview.pdf \ +skypartition.png \ datasystem.tex Index: trunk/doc/release.2015/ps1.datasystem/datasystem.tex =================================================================== --- trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 40612) +++ trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 40613) @@ -1,10 +1,12 @@ -% \documentclass[iop,floatfix]{emulateapj} +% \documentclass[preprint2]{emulateapj} % works for 2-column +\documentclass[iop,floatfix]{emulateapj} % \documentclass[iop,floatfix,onecolumn]{emulateapj} % \documentclass[12pt,preprint]{aastex} -\documentclass[10pt,preprint]{aastex} +% \documentclass[10pt,preprint]{aastex} % use for 1-column +% \documentclass[preprint]{aastex} % \pdfoutput=1 %\RequirePackage{deluxetable} -- included by aastex? -\RequirePackage{nsfprop} +%\RequirePackage{nsfprop} % defines \subsubsubsection but breaks 2-col \RequirePackage{color} \RequirePackage{code} @@ -1226,5 +1228,76 @@ entry as such. -\begin{table}[hb] +\section{Post-Processing : Database Ingest and Calibration} +\label{sec:postprocessing} + +\subsection{DVO} +\label{sec:DVO} + +\subsubsection{Overview} + +% intro +The Pan-STARRS IPP uses an internal database system, distinct from the +publicly visible database system, to determine the association +between multiple detections of the same astronomical object and as +part of the astrometric and photometric calibration process. This +database system, called the ``Desktop Virtual Observatory'' (DVO) was +developed originally for the LONEOS project +\citep{1995DPS....27.0110B}, and used as part of the CFHT Elixir +system \citep{2004PASP..116..449M}. The capabilities of this +databasing system have been somewhat expanded for the Pan-STARRS +context. + +% overview +DVO tracks three main classes of information: 1) average properties of +astronomical objects; 2) measurements of those objects (from which the +average properties are derived); 3) properties of the images which +provided some or all of the measurements. In addition, certain +metadata tables define general features of the database. +Table~\ref{tab:DVO_schema} lists the full collection of database +tables used by DVO. + +%Figure~\ref{fig:DVO_schema} +%illustrates the schematic relationship between these types of +%measurements. + +In the most basic implementation, a collection of measurements for +detections from a set of images is loaded into DVO along with the +metadata describing the images. The latter includes properties such +as the exposure time, airmass, filter, time \& date of the exposure, +etc. Critically, the image metadata includes an astrometric +transformation relating the detection coordinate on the image to the +coordinate on the sky. As the collection of measurements are loaded +into DVO, the software constructs astronomical objects based on those +detections. If images overlap, multiple observations of the same +astronomical object are grouped together. Thus, a single DVO database +will contain a one-to-many relationship between the images and the +measurements and a many-to-one relationship between the measurements +and the derived astronomical objects. + +% +%% These tables fall into one of several classes: +%% those which store information about the average properties of +%% astronomical objects; those which store information about individual +%% measurements; those which store information about the images; those +%% which store supporting information (metadata). + +%% DVO includes two major classes of database tables: those containing +%% information about astronomical objects in the sky and those containing +%% other supporting information. The object-related tables are +%% partitioned on the basis of position in the sky: objects within a +%% region bounded by lines of constant RA,DEC are contained in a specific +%% file. The boundaries and the associated partition names are stored in +%% one of the supporting tables, \ippdbtable{SkyTable}. This table +%% contains the definitions of the boundaries for each sky region +%% (\ippdbcolumn{R_MIN}, \ippdbcolumn{R_MAX}, \ippdbcolumn{D_MIN}, +%% \ippdbcolumn{D_MAX}), the name of the sky region, an ID +%% (\ippdbcolumn{INDEX}, equal to the sequence number of the region in +%% the table), and index entries to enable navigation within the table. +%% The regions are defined in a hierarchical sense, with a series of +%% levels each containing a finer mesh of regions covering the sky. + +\subsubsection{DVO Schema} + +\begin{table*}[hb] \begin{center} \caption{DVO Database Tables\label{tab:DVO_schema}} @@ -1248,78 +1321,7 @@ \end{tabular} \end{center} -\end{table} - -\section{Post-Processing : Database Ingest and Calibration} -\label{sec:postprocessing} - -\subsection{DVO} -\label{sec:DVO} - -\subsubsection{Overview} - -% intro -The Pan-STARRS IPP uses an internal database system, distinct from the -publicly visible database system, to determine the association -between multiple detections of the same astronomical object and as -part of the astrometric and photometric calibration process. This -database system, called the ``Desktop Virtual Observatory'' (DVO) was -developed originally for the LONEOS project -\citep{1995DPS....27.0110B}, and used as part of the CFHT Elixir -system \citep{2004PASP..116..449M}. The capabilities of this -databasing system have been somewhat expanded for the Pan-STARRS -context. - -% overview -DVO tracks three main classes of information: 1) average properties of -astronomical objects; 2) measurements of those objects (from which the -average properties are derived); 3) properties of the images which -provided some or all of the measurements. In addition, certain -metadata tables define general features of the database. -Table~\ref{tab:DVO_schema} lists the full collection of database -tables used by DVO. - -%Figure~\ref{fig:DVO_schema} -%illustrates the schematic relationship between these types of -%measurements. - -In the most basic implementation, a collection of measurements for -detections from a set of images is loaded into DVO along with the -metadata describing the images. The latter includes properties such -as the exposure time, airmass, filter, time \& date of the exposure, -etc. Critically, the image metadata includes an astrometric -transformation relating the detection coordinate on the image to the -coordinate on the sky. As the collection of measurements are loaded -into DVO, the software constructs astronomical objects based on those -detections. If images overlap, multiple observations of the same -astronomical object are grouped together. Thus, a single DVO database -will contain a one-to-many relationship between the images and the -measurements and a many-to-one relationship between the measurements -and the derived astronomical objects. - -% -%% These tables fall into one of several classes: -%% those which store information about the average properties of -%% astronomical objects; those which store information about individual -%% measurements; those which store information about the images; those -%% which store supporting information (metadata). - -%% DVO includes two major classes of database tables: those containing -%% information about astronomical objects in the sky and those containing -%% other supporting information. The object-related tables are -%% partitioned on the basis of position in the sky: objects within a -%% region bounded by lines of constant RA,DEC are contained in a specific -%% file. The boundaries and the associated partition names are stored in -%% one of the supporting tables, \ippdbtable{SkyTable}. This table -%% contains the definitions of the boundaries for each sky region -%% (\ippdbcolumn{R_MIN}, \ippdbcolumn{R_MAX}, \ippdbcolumn{D_MIN}, -%% \ippdbcolumn{D_MAX}), the name of the sky region, an ID -%% (\ippdbcolumn{INDEX}, equal to the sequence number of the region in -%% the table), and index entries to enable navigation within the table. -%% The regions are defined in a hierarchical sense, with a series of -%% levels each containing a finer mesh of regions covering the sky. - -\subsubsection{DVO Schema} - -\subsubsubsection{Photcodes} +\end{table*} + +\paragraph{Photcodes} % photcodes @@ -1366,5 +1368,5 @@ photcode of the measurement. -\subsubsubsection{Measurement Tables} +\paragraph{Measurement Tables} In most cases, the individual measurements of the astronomical objects @@ -1438,5 +1440,5 @@ % \note{Average used above but defined below} -\subsubsubsection{Object Tables} +\paragraph{Object Tables} \label{sec:object} @@ -1501,5 +1503,5 @@ calculated. -\subsubsubsection{Image Tables} +\paragraph{Image Tables} Measurements which are loaded into DVO may be associated with a @@ -1536,5 +1538,5 @@ %% \ippdbtable{Measure} and similar tables, -\subsubsubsection{Other Tables} +\paragraph{Other Tables} Other tables are used to track information used by the calibration @@ -2767,33 +2769,15 @@ \rfloor + 1$, where $\mathrm{nodes}_\mathrm{max}$ is the maximum number of nodes that can be requested in a single job (1000 for -Mustang). Table \ref{tab:SC processing parameters} contains the cost +Mustang). Table \ref{tab:SC_processing_parameters} contains the cost values used for the various IPP processing stages. -%% \begin{table} -%% \caption{\label{tab:SC_processing_parameters} Cost values for remote processing}\vspace{-0.5cm} -%% \begin{center} -%% \begin{tabular}{lcc} -%% \hline -%% \hline -%% {\bf IPP Stage} & {\bf $t_\mathrm{task}$ (s)} & {\bf $S_\mathrm{task}$} \\ -%% \hline -%% \ippstage{chip} & 150 & 2 \\ -%% \ippstage{camera} & 1700 & 2 \\ -%% \ippstage{warp} & 110 & 2 \\ -%% \ippstage{stack} & 1500 & 6 \\ -%% \ippstage{staticsky} & 7200 & 6 \\ -%% % \ippstage{diff} & 300 & 2 \\ -%% \ippstage{fullforce} & 300 & 2 \\ -%% \hline -%% \end{tabular} -%% \end{center} -%% \end{table} - -\begin{deluxetable}{lcc} - \tablecolumns{3} - \tablewidth{0pc} - \tablecaption{Cost values for remote processing} - \tablehead{\colhead{IPP Stage}&\colhead{$t_\mathrm{task}$ (s)}&\colhead{$S_\mathrm{task}$}} - \startdata +\begin{table*} +\caption{\label{tab:SC_processing_parameters} Cost values for remote processing} +\begin{center} +\begin{tabular}{lcc} +\hline +\hline +{\bf IPP Stage} & {\bf $t_\mathrm{task}$ (s)} & {\bf $S_\mathrm{task}$} \\ +\hline \ippstage{chip} & 150 & 2 \\ \ippstage{camera} & 1700 & 2 \\ @@ -2802,8 +2786,26 @@ \ippstage{staticsky} & 7200 & 6 \\ % \ippstage{diff} & 300 & 2 \\ - \ippstage{fullforce} & 300 & 2 - \enddata - \label{tab:SC processing parameters} -\end{deluxetable} + \ippstage{fullforce} & 300 & 2 \\ +\hline +\end{tabular} +\end{center} +\end{table*} + +%%\begin{deluxetable}{lcc} +%% \tablecolumns{3} +%% \tablewidth{0pc} +%% \tablecaption{Cost values for remote processing} +%% \tablehead{\colhead{IPP Stage}&\colhead{$t_\mathrm{task}$ (s)}&\colhead{$S_\mathrm{task}$}} +%% \startdata +%% \ippstage{chip} & 150 & 2 \\ +%% \ippstage{camera} & 1700 & 2 \\ +%% \ippstage{warp} & 110 & 2 \\ +%% \ippstage{stack} & 1500 & 6 \\ +%% \ippstage{staticsky} & 7200 & 6 \\ +%%% \ippstage{diff} & 300 & 2 \\ +%% \ippstage{fullforce} & 300 & 2 +%% \enddata +%% \label{tab:SC processing parameters} +%%\end{deluxetable} Once the preparation for the job is complete, the input and output @@ -2900,6 +2902,6 @@ \bibliographystyle{apj} -\bibliography{lib}{} -%\input{datasystem.bbl} +%\bibliography{lib}{} +\input{datasystem.bbl} % \appendix