Index: trunk/doc/manual/manual.tex =================================================================== --- trunk/doc/manual/manual.tex (revision 13869) +++ trunk/doc/manual/manual.tex (revision 13881) @@ -1,4 +1,4 @@ - %%% $Id: manual.tex,v 1.8 2007-06-19 03:34:03 eugene Exp $ -\documentclass[panstarrs,spec]{panstarrs} + %%% $Id: manual.tex,v 1.9 2007-06-19 21:38:53 eugene Exp $ +\documentclass[panstarrs,psreport,spec]{panstarrs} % basic document variables @@ -24,5 +24,7 @@ % version Date Description DR & 2007-01-31 & First draft \\ \hline -\RevisionsEnd +\hline +\end{tabular} +\end{center} \inserttbd @@ -409,7 +411,60 @@ \note{specifying the dvo output database} -\subsubsection{Pipeline Reduction : CFHT Queue Runs} - -\subsubsection{Pipeline Reduction : PS1 Survey Operations} +\subsubsection{Pipeline Reduction : PI-Oriented Observatory} + +An PI-oriented observatory has the opportunity to operate the analysis +system on a somewhat larger scale than the individual user. At a +PI-oriented observatory, there will normally be a rotating suite of +instruments, each of which is mounted on the telescope for a +substantial number of nights. Over the course of months and years, +the observatory will collect enough data from the instruments to +perform a much more detailed level of analysis of the instrument than +is possible in a single observing run by a single PI. In such a +situation, the observatory may use the IPP to generate and track the +high-quality master detrend images, to apply the detrend images to the +data as it is being collected, and to monitor the instrumental zero +points and other calibration information. The basic concepts of +running the IPP at a PI observatory are identical to that of a single +user. There are, however, a few modifications which may be +interesting. + +First, it is natural to automatically inject and register the data as +it is obtained at the telescope. The telescope readout software could +be modified to call \code{ipp_serial_inject.pl} after each image is +obtained. Alternatively, this step could take place downstream +wherever the data are archived. In an observatory environment, more +so than in a single-user situation, it may be perfereable to use the +Nebulous infrastructure to manage the location and copies of the image +data. \tdb{define an inject mechanism that uses Nebulous}. + +Second, it is interesting to consider on what timescale to test and +re-build the master detrend images. In the single user case, the +possible input detrend images are fairly limited. As illustrated +above, the single user may use the IPP to define master detrend images +and test the consistency of the inputs against these masters. In an +observatory environment, it may be more appropriate to build the +masters only on regular intervals when there is a suspicion the +instrument may have changed. The periods in which the instrument is +mounted define an obvious possible starting point for this period. +Alternatively, the detrend creation steps in the IPP can be defined to +run in verify mode on a nightly basis to test the current state of the +instrument. For the PI-oriented observatory, the number of variables +may make it difficult to rely on these results in an automatic +fashion; user examination of the results with \code{ippMonitor} may be +needed to decide if a new master should be built. + +\subsubsection{Pipeline Reduction : Survey Observatory} + +A telescope used as a survey instrument has a somewhat different, more +limited range of circumstances, than a PI-oriented observatory. There +is generally a pre-defined schedule, perhaps with only a single +instrument. The instrument configuration is more controlled, and the +types of observations are more consistently defined. In this case, an +automatic validation process can be more reliably used. The IPP can +be set up to run the detrend validation stage at the beginning of +every night, to automatically generate a new master if any of the +input detrend images are out of spec, and to start automatic +processing of the night's data with the new master images when they +are ready. \section{IPP Components} @@ -419,61 +474,148 @@ \subsection{Analysis Programs} + \subsubsection{psphot} + +The detection and characterization of astronomical objects within an +image is performed by the IPP component called \code{psphot}. This +software may be used as a stand-alone program, or it may be called by +other IPP programs to operate on the images they generate. A more +detailed guide is available at REF. Here we summarize the basic +operation and command-line options. + +The basic call to the stand-alone version of \code{psphot} is: +% +\code{psphot -file input.fits output [options]} +% +where \code{[options]} +represents options to the IPP configuration system as well as some +specific to \code{psphot}. In the case of a ``split'' camera format, +where separate chips or amps are stored in separate files, the +``input.fits'' may be a UNIX file glob which will expand to the set of +file; note that the glob must be protected with double quotes for +psphot. Alternatively, the input file list may be written to a text +file and the \code{-file input.fits} entry replace with \code{-list + file.txt}. The output entry is used as the root of a number of +possible output files selected by the user. + +Standard IPP-wide command-line options (document elsewhere): +\begin{itemize} +\item C{-site site.mdc} +\item C{-camera camera-name} +\item C{-recipe NAME VALUE} +\item C{-D KEY VALUE} +\item C{-Df KEY VALUE} +\item C{-Db KEY VALUE} +\item C{-Di KEY VALUE} +\end{itemize} + +Here are other possible command-line options specific to psphot: +\begin{itemize} +\item -version : report version info and exit +\item -mask mask.fits +\item -weight weight.fits +\item -modeltest X Y : run the object model on the object at the given coordinates +\begin{itemize} +\item -model +\item -fitmode +\item -fitset +\end{itemize} +\item -photcode : manually specify the photcode for this image +\item -region : limit the analysis to a specific image area +\item -chip : limit the analysis to the given list of chips +\item -psf : use the specified psf model (do not build a psf model) +\item -src : perform photometry on the positions given in the list +\end{itemize} + +There are many configuration options in the \code{psphot.config} +recipe file. These are defined in the \code{psphot} manual. + \subsubsection{psastro} + \subsubsection{ppStats} + \subsubsection{ppImage} + \subsubsection{ppMerge} + \subsubsection{ppNorm} \subsection{Pipeline Infrastructure} -\subsubsection{ippdb} + +\subsubsection{ippdb / dbconfig} + +The IPP uses a mysql database to track the data to be processed, +the current state of the processing steps, and summary information +from each of the analysis steps. The database schema is defined by +the set of MDC files in \code{dbconfig}. These files are used to +generate a set of C-level APIs to access the database tables. + \subsubsection{ippTools} + \subsubsection{ippScripts} + \subsubsection{ippTasks and panTasks} + \subsubsection{ippMonitor} -\subsubsection{Nebulous} +\subsection{Nebulous} \subsection{DVO} + \subsubsection{DVO Shell} + \subsubsection{Adding and Removing Data} + \subsubsubsection{addstar} + \subsubsubsection{delstar} + \subsubsubsection{getstar} + \subsubsection{Database Level Calibrations} + \subsubsubsection{relphot} + \subsubsubsection{uniphot} + \subsubsubsection{relastro} + \subsubsection{Other Tools} + \subsubsubsection{sky cell tools} \subsection{Software Architecture} -\subsection{psLib} -\subsection{psModules} -\subsection{Perl Modules} + +\subsubsection{psLib} + +\subsubsection{psModules} + +\subsubsection{Perl Modules} \section{Configuration} -Correct use of the IPP depends on several configuration files. We -distinguish below between configuration files required for the image -processing and those for running the process scheduler, PanTasks. -Note, however, that the Perl scripts called by PanTasks to run the -processing do use the site and camera configuration files principally -used for the image processing. - -\subsection{Image Processing} +Correct use of the IPP depends on a collection of configuration files. +These files define information about the processing environment, such +as the location of reference data or the name of computers available +for analysis or storage. They also define the operating parameters +for the different programs, and choices about which analysis steps to +perform. Configuration information for the image processing is provided on four -levels: the site, camera, format and recipe configurations. Each uses -the ``MetaData Configuration'' (MDC) file format, which is briefly -described below; for a more detailed description, see the psLib SDRS -(PSDC-430-007). +levels: the site-specific information, camera-specific details, +information related to different formats of data from the same camera, +and recipe configurations for the different analysis programs or +steps. The configuration information is stored in files using the +``psMetadataConfig'' (MDC) file format, which is briefly described +below (\S\ref{sec:MDC}); for a more detailed description, see the +psLib SDRS (PSDC-430-007). The configuration levels for the image processing components of the IPP are: \begin{itemize} -\item Options for the particular site installation of the - pipeline: the {\it site}; -\item Options specifying the instrument setup: the {\it camera}; +\item Options for the particular installation of the + IPP: the {\it site}; +\item Options specifying the details about the instrument: the {\it +camera}; \item Options specifying the format of the FITS file: the {\it format}; and @@ -493,5 +635,30 @@ (PSDC-430-???) has all the detailed information. +\subsubsection{Setting up configuration files} + +When the IPP software is installed, it generates a copy of the +top-level {\it site} configuration file: +\code{PREFIX/share/ippconfig/ipprc.config}. Programs look for this +information in the user's home directory at \code{~/.ipprc}. A new +user may want to link \code{~/.ipprc} to the installed copy +\code{PREFIX/share/ippconfig/ipprc.config}, in which case new +installations will be immediately available. Alternatively, the user +may want to copy the file in order to make their own modifications, if +for example the user needs to modify the recipes for a specific +camera. + +After the {\it site} configuration file is read by IPP programs, they +then attempt to read the {\it camera} configuration files defined in +the {\it site} file by the list of cameras. These entries refer to +files for each camera. The IPP configuration system searches for these +files (and others referred there in) in the PATH defined at the top of +the {\it site} file. This value mimics the UNIX PATH variable, and +consists of a colon-separated list of locations. The first entry +found from this list is used by the configuration system, allowing a +user to override, for example, the globally installed camera +configuration for some camera. + \subsubsection{Overview of MDC format} +\label{sec:MDC} psLib defines a \code{psMetadata} structure which can carry labeled @@ -577,23 +744,23 @@ current level. - -\subsubsection{Setting up configuration files} - -You will generally want to link \code{~/.ipprc} to the site -configuration file (\code{ipprc.config} which gets installed in -\code{PREFIX/share/ippconfig/} where \code{PREFIX} is your -installation prefix). Then link \code{~/.ipp} to the \code{ippconfig} -directory to save hacking the \code{PATH} in the site configuration. - -\subsection{PanTasks} - -\subsubsection{Paths} - -Throughout PanTasks, a file may be referred to as: +\subsection{Filenames : UNIX Paths, Abstract Paths, Nebulous} + +The IPP programs recognize three types of file names: a standard UNIX +path, an abstract path with a top-level location defined by the IPP +configuration system, and a Nebulous path, in which the file location +is completely abstracted by the Nebulous file management system: \begin{itemize} -\item \code{/path/to/file.ext} --- not a URI, but it should work. -\item \code{file:///path/to/file.ext} --- the URI version of the above. -\item \code{path://PATH/file.ext} --- Uses the \code{DATAPATH} in the site configuration to set the path. +\item \code{file:///path/to/file.ext} : a UNIX path, always relative + to the root of the file system. It is valid to drop the + \code{file:/} component of the name. The IPP accepts an arbitrary + number of slashes after \code{file:}. +\item \code{path://PATH/file.ext} : An abstract path. The value of + PATH is iterpolated from the corresponding entry in the + \code{DATAPATH} table given in the site configuration file. +\item \code{neb://full/nebulous/name} : A nebulous filename. The IPP + programs query Nebulous for the true path of (one copy of) the file. \end{itemize} +If you wish to expand a file name in any of these forms to a standard +UNIX path, use the program \code{ipp_datapath.pl}. \section{Installation} @@ -653,5 +820,5 @@ When building software -\subsubsection{jhbuild} +\subsection{jhbuild} JH uses \code{jhbuild} even though the 'jh' in \code{jhbuild} doesn't @@ -935,5 +1102,5 @@ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -\subsubsection{Aliases} +\subsection{Aliases} PAP puts the following in his \code{~/.tcshrc}: @@ -966,5 +1133,5 @@ \end{itemize} -\subsection{Perl} +\subsection{Manual Perl Module Installation} Here we describe setting up the Perl dependencies followed by the