Index: /trunk/doc/manual/manual.tex =================================================================== --- /trunk/doc/manual/manual.tex (revision 13846) +++ /trunk/doc/manual/manual.tex (revision 13847) @@ -1,3 +1,3 @@ - %%% $Id: manual.tex,v 1.6 2007-06-15 15:33:38 eugene Exp $ + %%% $Id: manual.tex,v 1.7 2007-06-15 20:52:48 eugene Exp $ \documentclass[panstarrs,spec]{panstarrs} @@ -6,5 +6,5 @@ \subtitle{User Guide} \shorttitle{PS IPP UG} -\author{Paul A. Price, Joshua Hoblitt} +\author{Eugene A. Magnier, Paul A. Price, Joshua Hoblitt} \audience{Pan-STARRS IPP Users} \group{Pan-STARRS Image Processing Pipeline} @@ -124,4 +124,1084 @@ %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +\section{Using the IPP} + +\subsection{Conceptual Overview} + +The operating goals of the IPP can be broken down into several major categories: +\begin{itemize} +\item reduction, analysis, and calibration of individual astronomical science images. +\item combinations (additions and subtractions) of groups of science image, along with their analysis and calibration. +\item analysis of the ensemble properties of astronomical objects detected from many images, including improved astrometric and photometric calibrations. +\item construction of the necessary master detrend images and other calibration information needed to perform the science analysis. +\item enabling further investigation into the data characteristics by tracking sufficient metadata and providing tools for this analysis. +\end{itemize} +Furthermore, the IPP is designed to perform these tasks with a high +degree of automation and for large collections of data from multiple +instruments. + +A cartoon of the user's view on the IPP is seen in Figure~\ref{}. +Images are injected into the IPP, either automatically by software +interacting with the telescope systems or manually by the user. The +images are processed by the IPP through any number of analysis steps; +the specifics of the analysis will depend on the type of image, the +overall configuration, and additional requests the user may make. The +user may use the program \code{pantasks} to control and monitor the +ongoing processing, or view summary result information in the +ippMonitor web browser. As science images are processed, the +information about the objects detected in these images is passed to +the DVO database. The user may explore the results or perform further +science analysis by interacting with the DVO database via the DVO +Shell. From this tool, the user may, for example, create +color-magnitude diagrams of regions in the sky, or plot light curves +of specific objects. The DVO Shell makes it possible to performing +data extraction and data visualization on the DVO database. + +Below, we discuss in more detail how a user would use the IPP in +several different operating scenarios: to reduce the results from an +individual observing run; to perform automatic analysis of data from a +telescope with a blocked-out schedule, as for example the CFHT Megacam +Queue; to perform automatic analysis of data from a continuous survey +instrument such as the Pan-STARRS PS1 Telescope. These difference +scenarios have slightly different conditions and underlying +assumptions, and make a good set of examples for the types of +decisions a user much make. + +All of these scenarios have a few common elements. The first goal of +the user is to explore the characteristics of the instrument in +question and to define initial master detrend images. The intial +master detrend images may be used to test the validity of new detrend +images and thereby test the evolution of the instrument response. +Next, the user will make an initial analysis of science images. This +initial analysis may be used to construct further master detrend or +calibration information (eg, flat-field correction images; reference +astrometric parameters; deeper astrometric reference catalog). +Finally, the user may begin large-scale automatic analysis of the +complete data collection or the real-time data stream. + +\subsection{Projects and Databases} + +The IPP has the concept of a 'project': a related group of data that +are analysed within the same context. A project may consist of data +from many different camera, or the user may find it more convenient to +divide data even from the same camera into multiple independent +projects. A single installation of the IPP may operate on many +separate projects at the same time. + +Within a single project, the user may also choose to define more than +one output DVO database. For example, within the Pan-STARRS project, +the detections from the different survey modes, with different depths +and cadences, will initially be saved in separate DVO databases, and +only later joined for improved calibration and more in-depth analysis. + +\subsection{User Interface Tools} + +In this section, we introduce the user to the IPP user interface +tools. + +\subsubsection{ippMonitor} + +The \code{ippMonitor} is a web-based tool that allows the end-user to +explore at a high level the results of the IPP analysis. There is an +initial login page, after which the user is asked to choose a project. +Once a project is selected, the user may view pages which show summary +information of the images that have been processed at each of the +different stages. There are top-level sections for the detrend +analysis, the single science image analysis, and the different image +combinations. Further clicking takes the user to more detailed +information about the individual images: jpegs of the reduced image, +plots showing the psf modeling and the astrometric analysis, etc. + +\note{to be added to ippMonitor: pages showing summary plots from DVO} + +\subsubsection{DVO Shell} + +The DVO database is the primary output destination for results from +the IPP. The DVO shell is the basic user interface to the DVO +database. Within the DVO shell, the user may extract measurments from +the database, create plots, or view the locations of objects and +images on projections of the sky. + +A complete user's guide is available at DVO REF. Here are a few +quick-start details: + +Start the DVO shell by typing: \code{dvo}. You will be given a +prompt, at which you may type commands. This is an interactive shell, +with standard readline comand-line editing features. The \code{help} +command gives further information about the commands. Commands will +give additional information if they are given a '-help' option, or in +many cases if the command is typed without arguments. + +\begin{verbatim} +catdir demo +region -n view1 0 0 80 ait +images + +region -n view2 10 40 2 +avextract ra,dec where (g - i > 2.0) +cplot ra dec -c red +avextract ra,dec where (g - i < 0.5) +cplot ra dec -c blue +\end{verbatim} + +Choose the DVO database with the command \code{catdir}. \note{you +must specify the path to the top of the database directories}. Define +a view port for plotting objects and images on a sky projection with: +\code{region RA DEC range [projection]}. The projection may be +\code{ait} (aitoff projection), \code{sin} (sin projection), .... + +\subsubsection{pantasks} + +Pantasks is the program which schedules the analysis to be performed +and distributes the analysis jobs across the parallel processing +environemnt. A more complete user's guide is available at REF. Here +we give a basic overview of starting up the IPP. In this example, we +are using the stand-alone pantasks program. It is also possible to +run pantasks in a client / server mode. In this mode, the server runs +continuously in the background, and multiple users may interact with +the same pantasks server via the pantasks client tool. Most of the +commands are the same in both contexts. However, for security reasons +in the client / server mode, the clients are not able to issue certain +commands to the server. Thus, there are some differences in the details. + +Start \code{pantasks}. Load the collection of IPP tasks with the +command: \code{module pantasks.pro}. Add remote analysis hosts to the +parallel processing controller (pcontrol): \code{controller host add +NAME}, where NAME is the name of a machine set up to run the IPP. Add +the IPP project(s) of interest: \code{add.database (project)}. Start +the analysis: \code{run}. Check the current status with +\code{status}. If something goes wrong, stop the processing with +\code{stop} or \code{halt}. The former will continue to accept the +results from the jobs already launched, while the latter halts all +processing. \note{halt does not current stop pcontrol}. + +\subsubsection{ippTools} + +\subsection{Operational Scenarios} + +\subsubsection{Reducing an Observing Run} + +\subsubsection{Pipeline Reduction : CFHT Queue Runs} + +\subsubsection{Pipeline Reduction : PS1 Survey Operations} + +\section{IPP Components} + +\tbd{Need to document command-line arguments, perhaps even +algorithms.} + +\subsection{Analysis Programs} +\subsubsection{psphot} +\subsubsection{psastro} +\subsubsection{ppStats} +\subsubsection{ppImage} +\subsubsection{ppMerge} +\subsubsection{ppNorm} + +\subsection{Pipeline Infrastructure} +\subsubsection{ippdb} +\subsubsection{ippTools} +\subsubsection{ippScripts} +\subsubsection{ippTasks and panTasks} +\subsubsection{ippMonitor} + +\subsubsection{Nebulous} + +\subsection{DVO} +\subsubsection{DVO Shell} +\subsubsection{Adding and Removing Data} +\subsubsubsection{addstar} +\subsubsubsection{delstar} +\subsubsubsection{getstar} +\subsubsection{Database Level Calibrations} +\subsubsubsection{relphot} +\subsubsubsection{uniphot} +\subsubsubsection{relastro} +\subsubsection{Other Tools} +\subsubsubsection{sky cell tools} + +\subsection{Software Architecture} +\subsection{psLib} +\subsection{psModules} +\subsection{Perl Modules} + +\section{Configuration} + +Correct use of the IPP depends on several configuration files. We +distinguish below between configuration files required for the image +processing and those for running the process scheduler, PanTasks. +Note, however, that the Perl scripts called by PanTasks to run the +processing do use the site and camera configuration files principally +used for the image processing. + +\subsection{Image Processing} + +Configuration information for the image processing is provided on four +levels: the site, camera, format and recipe configurations. Each uses +the ``MetaData Configuration'' (MDC) file format, which is briefly +described below; for a more detailed description, see the psLib SDRS +(PSDC-430-007). + +The configuration levels for the image processing components of the +IPP are: +\begin{itemize} +\item Options for the particular site installation of the + pipeline: the {\it site}; +\item Options specifying the instrument setup: the {\it camera}; +\item Options specifying the format of the FITS file: the {\it + format}; and +\item Options specifying the particular parameter choices that affect + the details of an analysis: the {\it recipe}. +\end{itemize} +Note that these are arranged in an hierarchical order, with the site +configuration being the most general, and the recipe configurations +the most specific. For example, not all sites will have to deal with +all cameras, and different cameras may require different recipes at +different times according to their particular quirks, analysis +experimentations, or their evolution. + +We have provided examples of each of these configurations in the +\code{config} component of the IPP, which should be a useful guide for +setting up your own. The Pan-STARRS IPP Configuration Guide +(PSDC-430-???) has all the detailed information. + +\subsubsection{Overview of MDC format} + +psLib defines a \code{psMetadata} structure which can carry labeled +data of arbitrary types. Originally designed for carrying the data in +FITS headers, the \code{psMetadata} have proved so generally useful +that we use them for our configurations (and a multitude of other +uses!). We have designed a human-readable text-based format --- the +``MetaData Configuration'' (MDC) format --- which we use for this end. + +Each simple entry in an MDC file must contain the name, type and +value. Each of these is on a single line, separated by whitespace, +and in that order. Comments may be placed at the end of the line (or +on a blank line), after a hash mark (\code{#}). Whitespace at the +beginning and end of strings (either the name, value or comment) are +stripped. + +The simple types follow the psLib types. Integers are specified by a +letter indicating if the integer is signed (\code{S}) or unsigned +(\code{U}) and a number indicating the dynamic range in bits (8, 16, +32 or 64); e.g., \code{U8} is commonly used for bit mask values, +\code{S32} is commonly used for ordinary integer values. Floating +point values are specified by the letter \code{F} and a number +indicating the precision in bits (32 or 64): \code{F32} (single +precision) or \code{F64} (double precision). Strings are specified by +\code{STR}. Times may be specified with the following types: +\code{UTC,UT1,TAI,TT}; values for the time are expected to be in +ISO8601 format (\code{YYYY-MM-DDTHH:MM:SS.sZ}). + +Names are traditionally all-caps, though there is no reason why they +must be; the names are case-sensitive. A name may not be repeated +unless it has previously been declared to be of type \code{MULTI} (no +value should be provided with this declaration): +\begin{verbatim} +COMMENT MULTI +COMMENT STR Having more than one COMMENT like this +COMMENT STR is permitted because of the MULTI. +\end{verbatim} + +A hierarchy can be made using the \code{METADATA} type, which signals +a new level: +\begin{verbatim} +JANITOR METADATA + NAME STR John Doe + PAY F32 1234.56 + ECCENTRICITY STR 9.87 +END +\end{verbatim} +Note that a \code{METADATA} block is closed by an \code{END}. No +identing need be done within a \code{METADATA} block, but it is useful +to be able to see the levels at a glance (just like in a C program). +\code{METADATA} blocks may be nested within \code{METADATA} blocks, +probably down as far as you have the patience to try. Note that +\code{MULTI} declarations only apply to the current level --- there is +no inheritance. + +The above format can be long if there are many \code{METADATA}s with +similar contents. For this reason, we provide the \code{TYPE} +declaration, which generates a \code{METADATA} with the contents each +of type \code{STR}: +\begin{verbatim} +TYPE EMPLOYEE NAME PAY ECCENTRICITY +\end{verbatim} +Now, the type \code{EMPLOYEE} may be used, with string values (NB: no +spaces allowed!) to specify multiple entries: +\begin{verbatim} +JANITOR EMPLOYEE JohnDoe 1234.56 9.87 +PROGRAMMER EMPLOYEE FooBar 2345.67 1.00 +\end{verbatim} +This is the same as the much longer block: +\begin{verbatim} +JANITOR METADATA + NAME STR JohnDoe + PAY STR 1234.56 + ECCENTRICITY STR 9.87 +END +PROGRAMMER METADATA + NAME STR FooBar + PAY STR 2345.67 + ECCENTRICITY STR 1.00 +END +\end{verbatim} +Like the \code{MULTI}, \code{TYPE} declarations only apply to the +current level. + + +\subsubsection{Setting up configuration files} + +You will generally want to link \code{~/.ipprc} to the site +configuration file (\code{ipprc.config} which gets installed in +\code{PREFIX/share/ippconfig/} where \code{PREFIX} is your +installation prefix). Then link \code{~/.ipp} to the \code{ippconfig} +directory to save hacking the \code{PATH} in the site configuration. + +\subsection{PanTasks} + +\subsubsection{Paths} + +Throughout PanTasks, a file may be referred to as: +\begin{itemize} +\item \code{/path/to/file.ext} --- not a URI, but it should work. +\item \code{file:///path/to/file.ext} --- the URI version of the above. +\item \code{path://PATH/file.ext} --- Uses the \code{DATAPATH} in the site configuration to set the path. +\end{itemize} + +\section{Installation} + +After the dependencies (\S\ref{sec:dependencies}) have been satisfied, +the IPP packages should be installed in the following order: +\begin{itemize} +\item \code{Ohana} +\item \code{psLib} +\item \code{psModules} +\item \code{psphot} +\item \code{psastro} +\item \code{ppStats} +\item \code{ppImage} +\item \code{ppMerge} +\item \code{ppNorm} +\item \code{pois} +\item \code{stac} +\item \code{pswarp} +\item \code{ppStac} +\item \code{ippdb} +\item \code{ippTools} +\item \code{PS-IPP-Metadata-Config} +\item \code{PS-IPP-Metadata} +\item \code{ippScripts} +\item \code{ippTasks} +\item \code{config} +\end{itemize} + +\subsection{psconfig} + +* pscheckperl : search for and install, if needed, external Perl modules +* pschecklibs : search for and install, if needed, external C libraries +* psconfig : set up the UNIX shell environment +* psbuild : build and install the software +* psdist : build IPP distributions (requires CVS access) +* tagsets : tables defining the C and Perl components to be built + +The IPP is a large and complex software system. A major goal of the +IPP build system is to be user-friendly for those end users which do +not have root access on their machines. Using the IPP build tools, it +is possible to install the complete system as a non-priviledged user. +The build system also makes it possible to maintain multiple +simultaneous installations with different versions of the +software. This latter feature is particularly important for developers +who need to be able to make tests and comparisons of different +versions. + +\subsubsubsection{UNIX environment} + +With the psconfig system, the complete collection of libraries and + +When building software + +\subsubsection{jhbuild} + +JH uses \code{jhbuild} even though the 'jh' in \code{jhbuild} doesn't +really refer to him. + +\subsubsubsection{What is it?} + +According to the introduction on the \code{jhbuild} website: + +\begin{quote} +\code{jhbuild} is a program that can be used to pull a number of +modules from CVS and build them in the correct order. Unlike some +build scripts, \code{jhbuild} lets you specify what modules you want +built and it will then go and build those modules plus dependencies. + +Although \code{jhbuild} was originally developed to build +\code{[WWW]Gnome}, it is now able to build a number of the modules in +\code{freedesktop.org} CVS. Extending it to handle new modules is +usually trivial (assuming the build infrastructure matches the other +modules it handles). +\end{quote} + +In additional to retrieving source code from various SCM's (CVS, SVN, +arch, etc.), jhbuild has the ability to download tarballs via HTTP or +FTP. + +\code{jhbuild} has been adopted as an official \code{freedesktop.org} +project. You can find more information on the project's homepage +(\code{http://www.freedesktop.org/Software/jhbuild}). Bugs can be +filed in the Gnome Bugzilla (\code{http://bugzilla.gnome.org}). + +\subsubsubsection{Where to get it} + +It was necessary to slightly modify \code{jhbuild} for use with IPP +software. Therefore, you must checkout the \code{jhbuild} module from +the Pan-STARRS CVS tree. Please see the Pan-STARRS CVS Guide for help +on setting up and using CVS. \code{jhbuild} will need to be able to +find it's own source tree even after installation so you should choose +a checkout path that can be permanent. Something along the lines of +\code{$HOME/src} is recommended. + +\begin{verbatim} +cd +mkdir -p src +cd src +cvs co jhbuild +\end{verbatim} + +After running CVS you should see something like this: + +\begin{verbatim} +$ cvs co jhbuild +cvs checkout: Updating jhbuild +U jhbuild/.cvsignore +U jhbuild/COPYING +U jhbuild/ChangeLog +U jhbuild/HACKING +U jhbuild/Makefile +U jhbuild/README +U jhbuild/install-check.c +. +. +\end{verbatim} + +\subsubsubsection{Installing jhbuild into your home directory} + +\code{jhbuild} should be installed locally under your home +directory. This will require that you modify the \code{PATH} +environment variable so that you can run jhbuild after it has been +installed. + +\begin{verbatim} +cd jhbuild +make +make install +\end{verbatim} + +Which should look something like this: + +\begin{verbatim} +$ make +gcc -Wall -O2 -o install-check install-check.c +Run "make install" to install. +$ make install +Creating /home/moanui/jhoblitt/bin/jhbuild +Creating /home/moanui/jhoblitt/.gnome2/vfolders/applications/jhbuild.desktop +install -m755 install-check /home/moanui/jhoblitt/bin/install-check +install -m755 config.guess /home/moanui/jhoblitt/bin/config.guess +\end{verbatim} + +That will install the \code{jhbuild} executable under +\code{$HOME/bin}. You are responsible for including this path in your +\code{PATH} environment variable. It is highly recommended that you +add this to your \code{.bashrc} or equivalent shell login script. + +For the \code{bash} shell, place this line in your \code{.bashrc}: +\begin{verbatim} +export PATH=${HOME}/bin:${PATH} +\end{verbatim} +For the \code{tcsh} shell, place this line in your \code{.tschrc}: +\begin{verbatim} +setenv PATH ${HOME}/bin:${PATH} +\end{verbatim} + +\subsubsubsection{Configuring jhbuild} + +\code{jhbuild} is configured via an rc file that lives at +\code{${HOME}/.jhbuildrc}. Please note that this rc file is executed +as Python code; be careful! + +Example \code{.jhbuildrc}, suitable for cut and paste: + +\begin{verbatim} +# what profile to build? +moduleset = 'http://pan-starrs.ifa.hawaii.edu/project/IPP/software/modulesets/ipp12.modules' + +# modules to build by default +modules = [ 'pslib', 'psmodules' ] + +# where should working copies go? +jhroot = os.environ['HOME'] + '/jhroot' + +# where should tarballs be kept? +tarballdir = jhroot + '/src' + +# in what prefix should things be installed? (must be writable) +target = os.popen('config.guess').read().rstrip() +prefix = jhroot + '/' + target +checkoutroot = prefix + '/build' + +# extra arguments to pass to the autogen.sh script? +autogenargs = '--enable-maintainer-mode --disable-static' + +# use an alternative install program that preserves the +# mtime on header files if they haven't changed. Speeds +# up rebuilds. +os.environ['INSTALL'] = os.environ['HOME'] + '/bin/install-check' + +# don't try to use /usr/ucb/cc on Solaris +import sys +if sys.platform == 'sunos5': + os.environ['CC'] = 'gcc' +\end{verbatim} + +\subsubsubsection{Running jhbuild} + +\code{Jhbuild} can be executed as \code{jhbuild build +[modulename]}. Just \code{jhbuild} will build the packages specified +in the \code{modules} variable from your rc file. +\begin{verbatim} +jhbuild +\end{verbatim} +or +\begin{verbatim} +jhbuild build pslib +\end{verbatim} + +Run \code{jhbuild} list to get a list of the packages \code{jhbuild} +knows how to build. + +\begin{verbatim} +$ jhbuild list +cfitsio +gsl +fftw +libxml2 +mysql +pslib +psmodules +\end{verbatim} + +\code{jhbuild} supports many other commands. Please see \code{jhbuild +--help} for a complete list of options. + +\begin{verbatim} +$ jhbuild --help +usage: jhbuild [ -f config ] command [ options ... ] +Build a set of CVS modules (such as GNOME). + +Global options: + -f, --file=CONFIG use a non default configuration file + -m, --moduleset=URI use a non default module set + --no-interact do not prompt for input + +Commands: + gui build targets from a gui app + update update from cvs + updateone modules update a fixed set of modules + build [ opts... ] [modules] update and compile (the default) + buildone [ opts... ] modules build a single module + tinderbox [ opts... ] build non-interactively with logging + run program [ args... ] run a command in the build environment + shell start a shell in the build environment + sanitycheck check that required support tools exists + bootstrap build required support tools + list [ opts ... ] [modules] list what modules would be built + dot [ modules ] output a dot file of dependencies suitable + for processing with graphviz + info modules... prints information about modules + +Options valid for the build, buildone, tinderbox and update commands: + -s, --skip=MODULES treat the given modules as up to date + -t, --start-at=MODULE start building at the given module + -D date_spec set a sticky date when checking out modules + +Options valid for the build, buildone and tinderbox commands: + -a, --autogen always run autogen.sh + -c, --clean run make clean before make + -n, --no-network skip cvs update + +Options valid for the tinderbox command: + -o, --output=DIR directory to save build logs in + +Options valid for the list command: + -r, --show-revision show which revision will be built +\end{verbatim} + +\subsubsubsection{Dependancies} + +\code{jhbuild} has a fairly minimal set of dependencies --- far less +than what may be required to actually compile and install any +packages. However, if your system can meet the base requirements, +\code{jhbuild} should be able to bootstrap your build environment. +\begin{itemize} +\item A working C compiler (eg. \code{gcc}) +\item A working \code{libc} (eg. \code{glibc}) +\item Perl 5 with the \code{XML::Parser} module (needed by + \code{libtool}) +\item Python 2.? +\item Either \code{wget} or \code{curl} +\item GNU \code{M4} 1.4 +\item \code{tar} +\item \code{gzip} +\item \code{bzip2} +\end{itemize} + +\subsubsubsection{Bootstrapping} + +\code{jhbuild} has a limited ability to install some of the necessary +tools for maintaining software that configure its build environment +with the GNU autotools. + +This step is probably required on OSX and Solaris. Your mileage will +vary per Linux distribution but you can probably skip this step if +your distribution is around RedHat 9 vintage or newer. + +\begin{verbatim} +jhbuild bootstrap +\end{verbatim} + +\code{jhbuild} will then will begin to build a series of packages. + +\subsubsubsection{Using the jhbuild enviroment} + +As you've already seen, \code{jhbuild} is capable of setting up an +independent build environment under the (configurable) directory of +your choice. In order to link non-\code{jhbuild} management software +against this build environment a number of your shell's environment +variable have to be modified. \code{jhbuild} is capable of doing this +for you. The syntax for this is \code{jhbuild shell}, which as the +syntax implies, spawns a new shell with the proper environment +variables. + +This example demonstrates \code{jhbuild} setting up the dynamic +linkers default search path for you. + +\begin{verbatim} +$ echo $LD_LIBRARY_PATH + +$ jhbuild shell +$ echo $LD_LIBRARY_PATH +/home/moanui/jhoblitt/jhroot/i686-pc-linux-gnu/lib +\end{verbatim} + +%$ --- Emacs needs this to balance out the dollar signs + +A fair number of other variables are also adjusted for you. Enough so +that most (all?) \code{autoconf} configured software will be able to +find it's dependencies. + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{Aliases} + +PAP puts the following in his \code{~/.tcshrc}: +\begin{verbatim} +setenv SWDIR $HOME/local/`$HOME/bin/config.guess`/ +if (! -d $SWDIR) mkdir --parents $SWDIR +alias ./autogen.sh './autogen.sh --prefix=$SWDIR CFLAGS="-I$SWDIR/include/ -g" LDFLAGS=-L$SWDIR/lib/' +alias autogen.sh './autogen.sh --prefix=$SWDIR CFLAGS="-I$SWDIR/include/ -g" LDFLAGS=-L$SWDIR/lib/' +alias ./configure './configure --prefix=$SWDIR CFLAGS="-I$SWDIR/include/ -g" LDFLAGS=-L$SWDIR/lib/' +alias configure './configure --prefix=$SWDIR CFLAGS="-I$SWDIR/include/ -g" LDFLAGS=-L$SWDIR/lib/' +setenv PATH ${PATH}:$SWDIR/bin/ +setenv LD_LIBRARY_PATH $SWDIR/lib/:$SWDIR/lib/mysql:$LD_LIBRARY_PATH +setenv MANPATH $SWDIR/man:$MANPATH +setenv PKG_CONFIG_PATH $SWDIR/lib/pkgconfig/:$PKG_CONFIG_PATH +\end{verbatim} + +%$ --- Emacs needs this to balance out the dollar signs + +Here, \code{config.guess} is the common GNU script for guessing the +build system triplet (e.g., \code{i686-pc-linux-gnu}). + +There are a couple of notes: +\begin{itemize} +\item To compile a binary, simply do \code{./configure}, then + \code{make && make install}. +\item \code{Ohana} doesn't like this setup, so you need to build it + with: \code{\./configure --prefix=$SWDIR} +\item Perl modules can be installed: \code{./Build install + --prefix=$SWDIR}. +\end{itemize} + +\subsection{Perl} + +Here we describe setting up the Perl dependencies followed by the +IPP components. + +\subsubsection{Dependencies} +\label{sec:installing-perl-dependencies} + +If you have access to the \code{root} account, installation as +\code{root} is much easier. If not, you will have to go through +the more flaky installation as an unprivileged user. + +\subsubsubsection{Installation as root} + +Many of the Perl dependencies are available from the Comprehensive +Perl Archive Network (CPAN) at www.cpan.org. If you have root access +on your target machines, they can be very simply retrieved, built and +installed (replacing \code{MODULE_NAME} for each module): + +\begin{verbatim} +> su - +Password: +> cpan +[...] +cpan> install MODULE_NAME +[...] +cpan> quit +\end{verbatim} + +Follow the prompts. It's usually safe to accept the default (simply +hit enter) in response to most questions. + +If you get into trouble, try: \code{force install MODULE_NAME}. + +You can also try to use the \code{Bundle::PS} as described below if +you're feeling adventurous. + +\subsubsubsection{Installation as unprivileged user} + +To install modules from CPAN with the \code{CPAN.pm} interface, you +need to setup a CPAN configuration file in your home directory. Then +\code{CPAN.pm} can walk you through setting up the most important +configuration values. Unfortunately, there is some variation in the +behavior of the various versions of \code{CPAN.pm} that have shipped +with Perl. Some (most) of these variants will not correctly create a +configuration files that allows a non-\code{root} user to install +modules outside of "system" paths. In order to make sure that you get +a "correct" CPAN configuration file you need to ``prime'' it with a +few values. + +First you need to create the directory in which the CPAN configuration file will live. +\begin{verbatim} +> mkdir -p .cpan/CPAN/ +\end{verbatim} + +Then we need to create a partial configuration file. Note that this example +assumes that you want to install your perl modules under +\code{$HOME/local/lib/perl5}. + +\begin{verbatim} +> echo "\$CPAN::Config = {" >> .cpan/CPAN/MyConfig.pm +> echo " makepl_arg => q[PREFIX=$HOME/local/]," >> .cpan/CPAN/MyConfig.pm +> echo " mbuildpl_arg => q[--install_base $HOME/local/]," >> .cpan/CPAN/MyConfig.pm +> echo "};" >> .cpan/CPAN/MyConfig.pm +> echo "1;" >> .cpan/CPAN/MyConfig.pm +> echo "__END__" >> .cpan/CPAN/MyConfig.pm +\end{verbatim} + +%$ --- Emacs needs this to balance the previous dollar sign + +Now you need to invoke \code{CPAN.pm} so it can walk you through +configuring the rest of the required values. This is an example of +one possible configuration with \code{CPAN.pm} version 1.8802. +\textbf{Your version of CPAN.pm may present you with different +prompts.} Use your common sense. If in doubt, it is generally safe +to simply hit enter (and accept the default). + +\begin{verbatim} +> perl -MCPAN -e shell +CPAN: File::HomeDir loaded ok +Sorry, we have to rerun the configuration dialog for CPAN.pm due to +the following indispensable but missing parameters: + +build_cache, build_dir, cache_metadata, cpan_home, ftp_proxy, http_proxy, +index_expire, inhibit_startup_message, keep_source_where, make_arg, +make_install_arg, mbuild_arg, mbuild_install_arg, mbuild_install_build_command, +no_proxy, prerequisites_policy, scan_cache, urllist + + +The following questions are intended to help you with the +configuration. The CPAN module needs a directory of its own to cache +important index files and maybe keep a temporary mirror of CPAN files. +This may be a site-wide directory or a personal directory. + + + +I see you already have a directory + /home/moanui/jhoblitt/.cpan +Shall we use it as the general CPAN build and cache directory? + +CPAN build and cache directory? [/home/moanui/jhoblitt/.cpan] + + +Unless you are accessing the CPAN via the filesystem directly CPAN.pm +needs to keep the source files it downloads somewhere. Please supply a +directory where the downloaded files are to be kept. [/home/moanui/jhoblitt/.cpan/sources] +Directory where the build process takes place? [/home/moanui/jhoblitt/.cpan/build] + + +How big should the disk cache be for keeping the build directories +with all the intermediate files? + +Cache size for build directory (in MB)? [100] + + +The CPAN indexes are usually rebuilt once or twice per hour, but the +typical CPAN mirror mirrors only once or twice per day. Depending on +the quality of your mirror and your desire to be on the bleeding edge, +you may want to set the following value to more or less than one day +(which is the default). It determines after how many days CPAN.pm +downloads new indexes. + +Let the index expire after how many days? [1] + + +By default, each time the CPAN module is started, cache scanning is +performed to keep the cache size in sync. To prevent this, answer +'never'. + +Perform cache scanning (atstart or never)? [atstart] + + +To considerably speed up the initial CPAN shell startup, it is +possible to use Storable to create a cache of metadata. If Storable +is not available, the normal index mechanism will be used. + +Cache metadata (yes/no)? [yes] + + +The CPAN module can detect when a module which you are trying to build +depends on prerequisites. If this happens, it can build the +prerequisites for you automatically ('follow'), ask you for +confirmation ('ask'), or just ignore them ('ignore'). Please set your +policy to one of the three values. + +Policy on building prerequisites (follow, ask or ignore)? [ask] follow + + +Every Makefile.PL is run by perl in a separate process. Likewise we +run 'make' and 'make install' in separate processes. If you have +any parameters (e.g. PREFIX, LIB, UNINST or the like) you want to +pass to the calls, please specify them here. + +If you don't understand this question, just press ENTER. +Parameters for the 'make' command? +Typical frequently used setting: + + -j3 # dual processor system + +Your choice: [] +Parameters for the 'make install' command? +Typical frequently used setting: + + UNINST=1 # to always uninstall potentially conflicting files + +Your choice: [] UNINST=1 + + +The next questions deal with Module::Build support. + +A Build.PL is run by perl in a separate process. Likewise we run +'./Build' and './Build install' in separate processes. If you have any +parameters you want to pass to the calls, please specify them here. + +Parameters for the './Build' command? +Setting might be: + + --extra_linker_flags -L/usr/foo/lib # non-standard library location + +Your choice: [] +Do you want to use a different command for './Build install'? +Sudo users will probably prefer: + + su root -c ./Build +or + sudo ./Build +or + /path1/to/sudo -u admin_account ./Build + +or some such. Your choice: [./Build] +Parameters for the './Build install' command? +Typical frequently used setting: + + --uninst 1 # uninstall conflicting files + +Your choice: [] --uninst 1 + + +If you're accessing the net via proxies, you can specify them in the +CPAN configuration or via environment variables. The variable in +the $CPAN::Config takes precedence. + +Your ftp_proxy? [] +Your http_proxy? [] +Your no_proxy? [] +You have no /home/moanui/jhoblitt/.cpan/sources/MIRRORED.BY + I'm trying to fetch one +CPAN: LWP::UserAgent loaded ok +Fetching with LWP: + http://www.perl.org/CPAN/MIRRORED.BY + + +Now we need to know where your favorite CPAN sites are located. Push +a few sites onto the array (just in case the first on the array won't +work). If you are mirroring CPAN to your local workstation, specify a +file: URL. + +First, pick a nearby continent and country by typing in the number(s) +in front of the item(s) you want to select. You can pick several of +each, separated by spaces. Then, you will be presented with a list of +URLs of CPAN mirrors in the countries you selected, along with +previously selected URLs. Select some of those URLs, or just keep the +old list. Finally, you will be prompted for any extra URLs -- file:, +ftp:, or http: -- that host a CPAN mirror. + +(1) Africa +(2) Asia +(3) Central America +(4) Europe +(5) North America +(6) Oceania +(7) South America +Select your continent (or several nearby continents) [] 5 + +(1) Bahamas +(2) Canada +(3) Mexico +(4) United States +Select your country (or several nearby countries) [] 4 + +(1) ftp://carroll.cac.psu.edu/pub/CPAN/ +(2) ftp://cpan-du.viaverio.com/pub/CPAN/ +(3) ftp://cpan-sj.viaverio.com/pub/CPAN/ +(4) ftp://cpan.calvin.edu/pub/CPAN +(5) ftp://cpan.cs.utah.edu/pub/CPAN/ +(6) ftp://cpan.cse.msu.edu/ +(7) ftp://cpan.erlbaum.net/CPAN/ +(8) ftp://cpan.glines.org/pub/CPAN/ +(9) ftp://cpan.hostrack.net/pub/CPAN +(10) ftp://cpan.llarian.net/pub/CPAN/ +(11) ftp://cpan.mirrors.redwire.net/pub/CPAN/ +(12) ftp://cpan.mirrors.tds.net/pub/CPAN +(13) ftp://cpan.netnitco.net/pub/mirrors/CPAN/ +(14) ftp://cpan.pair.com/pub/CPAN/ +(15) ftp://cpan.teleglobe.net/pub/CPAN +(16) ftp://cpan.uchicago.edu/pub/CPAN/ +40 more items, hit RETURN to show them +Select as many URLs as you like (by number), +put them on one line, separated by blanks, hyphenated ranges allowed + e.g. '1 4 5' or '7 1-4 8' [] 14 11 12 + +Enter another URL or RETURN to quit: [] +New set of picks: + ftp://cpan.pair.com/pub/CPAN/ + ftp://cpan.mirrors.redwire.net/pub/CPAN/ + ftp://cpan.mirrors.tds.net/pub/CPAN + + +Please remember to call 'o conf commit' to make the config permanent! + + +cpan shell -- CPAN exploration and modules installation (v1.8802) +ReadLine support enabled + + cpan[1]> o conf commit +commit: wrote '/home/moanui/jhoblitt/.cpan/CPAN/MyConfig.pm' +\end{verbatim} + +%$ --- Emacs needs this to balance the previous dollar sign + +Now we need to install the module that installs the other modules. + +\begin{verbatim} +cpan> install Module::Build +\end{verbatim} + +Exit out of cpan: + +\begin{verbatim} +cpan> exit +\end{verbatim} + +In order to use of the installed modules, we need to setup an +environment variable called \code{PERL5LIB} so that 'perl' can find +them. To do this, we need to know where under 'perl5' our modules +were actually installed. This will set variable with the version of +Perl that you are using. The easiest way to do this is just just look +in the root of the path where we did the install. + +\begin{verbatim} +> ls local/lib/perl5/ +5.8.8 site_perl +\end{verbatim} + +That means we're using perl 5.8.8 and \code{PERL5LIB} needs to be +setup as following: +\begin{verbatim} +export PERL5LIB=$HOME/local/lib/perl5/5.8.8:$HOME/local/lib/perl5/site_perl/5.8.8 +\end{verbatim} + +Now we should install the basic compliment of helper modules that +\code{CPAN.pm} needs to function fully. Go back into CPAN (\code{perl +-MCPAN -e shell}) and: + +\begin{verbatim} +cpan> install Bundle::CPAN +\end{verbatim} + +You can quit out of the CPAN shell at this point with the `exit` +command or do the following few steps in another shell We're ready to +install the full set Perl module dependencies for IPP software. In +order to make this process a bit easier on the end user a "Bundle" +module has been created. In order to use it you need to create a +directory (if it doesn't already exist) called Bundle under your .cpan +directory. + +\begin{verbatim} +> mkdir -p .cpan/Bundle +\end{verbatim} + +The file \code{PS.pm} should copied into this directory: + +\begin{verbatim} +cp /path/to/PS.pm .cpan/Bundle/ +\end{verbatim} + +Back in the CPAN shell, 'force' the install of the PS Bundle. The +'force' keyword instructs the shell to ignore any tests failures. +This is necessary as some of the modules 'DBD::mysql'/etc. require a +properly working database setup in order for the tests to pass. You +will most likely be prompted for input by several the modules. It is +safe to answer with a carriage return to all questions. If it insists +on a path to \code{httpd}, hit \code{CTRL-C} and it will go on to the +next step. + +\begin{verbatim} +cpan> force install Bundle:PS +\end{verbatim} + +For further instructions on installing Perl modules from CPAN ''by +hand', see: +\begin{verbatim} +http://www.cs.ucsc.edu/~you/notes/perl-module-install.html +\end{verbatim} + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +\subsubsection{Modules and scripts} + +Pan-STARRS Perl modules and scripts are installed by executing the +following in the directory containing the source: +\begin{verbatim} +> perl Build.PL +> ./Build +> ./Build install --prefix=/path/to/install/ +\end{verbatim} + +Make sure you set your \code{PERL5LIB} environment variable to the +installation path (here, \code{/path/to/install/}). + \section{System Requirements} @@ -150,5 +1230,5 @@ binaries) and Perl module dependencies. -\subsubsection{Libraries} +\subsubsubsection{Libraries} \begin{figure} @@ -175,5 +1255,5 @@ -\subsubsection{Perl modules} +\subsubsubsection{Perl modules} \begin{figure} @@ -247,32 +1327,4 @@ \end{itemize} - -\section{Installation} - -After the dependencies (\S\ref{sec:dependencies}) have been satisfied, -the IPP packages should be installed in the following order: -\begin{itemize} -\item \code{Ohana} -\item \code{psLib} -\item \code{psModules} -\item \code{psphot} -\item \code{psastro} -\item \code{ppStats} -\item \code{ppImage} -\item \code{ppMerge} -\item \code{ppNorm} -\item \code{pois} -\item \code{stac} -\item \code{pswarp} -\item \code{ppStac} -\item \code{ippdb} -\item \code{ippTools} -\item \code{PS-IPP-Metadata-Config} -\item \code{PS-IPP-Metadata} -\item \code{ippScripts} -\item \code{ippTasks} -\item \code{config} -\end{itemize} - \subsection{Binaries} @@ -282,876 +1334,4 @@ environment and installing the binaries to deal with this problem. We describe each of these below. Choose one that works for you! - -\subsubsection{Aliases} - -PAP puts the following in his \code{~/.tcshrc}: -\begin{verbatim} -setenv SWDIR $HOME/local/`$HOME/bin/config.guess`/ -if (! -d $SWDIR) mkdir --parents $SWDIR -alias ./autogen.sh './autogen.sh --prefix=$SWDIR CFLAGS="-I$SWDIR/include/ -g" LDFLAGS=-L$SWDIR/lib/' -alias autogen.sh './autogen.sh --prefix=$SWDIR CFLAGS="-I$SWDIR/include/ -g" LDFLAGS=-L$SWDIR/lib/' -alias ./configure './configure --prefix=$SWDIR CFLAGS="-I$SWDIR/include/ -g" LDFLAGS=-L$SWDIR/lib/' -alias configure './configure --prefix=$SWDIR CFLAGS="-I$SWDIR/include/ -g" LDFLAGS=-L$SWDIR/lib/' -setenv PATH ${PATH}:$SWDIR/bin/ -setenv LD_LIBRARY_PATH $SWDIR/lib/:$SWDIR/lib/mysql:$LD_LIBRARY_PATH -setenv MANPATH $SWDIR/man:$MANPATH -setenv PKG_CONFIG_PATH $SWDIR/lib/pkgconfig/:$PKG_CONFIG_PATH -\end{verbatim} - -%$ --- Emacs needs this to balance out the dollar signs - -Here, \code{config.guess} is the common GNU script for guessing the -build system triplet (e.g., \code{i686-pc-linux-gnu}). - -There are a couple of notes: -\begin{itemize} -\item To compile a binary, simply do \code{./configure}, then - \code{make && make install}. -\item \code{Ohana} doesn't like this setup, so you need to build it - with: \code{\./configure --prefix=$SWDIR} -\item Perl modules can be installed: \code{./Build install - --prefix=$SWDIR}. -\end{itemize} - -\subsubsection{psconfig} - -* pscheckperl : search for and install, if needed, external Perl modules -* pschecklibs : search for and install, if needed, external C libraries -* psconfig : set up the UNIX shell environment -* psbuild : build and install the software -* psdist : build IPP distributions (requires CVS access) -* tagsets : tables defining the C and Perl components to be built - -The IPP is a large and complex software system. A major goal of the -IPP build system is to be user-friendly for those end users which do -not have root access on their machines. Using the IPP build tools, it -is possible to install the complete system as a non-priviledged user. -The build system also makes it possible to maintain multiple -simultaneous installations with different versions of the -software. This latter feature is particularly important for developers -who need to be able to make tests and comparisons of different -versions. - -\subsubsubsection{UNIX environment} - -With the psconfig system, the complete collection of libraries and - -When building software - -\subsubsection{jhbuild} - -JH uses \code{jhbuild} even though the 'jh' in \code{jhbuild} doesn't -really refer to him. - -\subsubsubsection{What is it?} - -According to the introduction on the \code{jhbuild} website: - -\begin{quote} -\code{jhbuild} is a program that can be used to pull a number of -modules from CVS and build them in the correct order. Unlike some -build scripts, \code{jhbuild} lets you specify what modules you want -built and it will then go and build those modules plus dependencies. - -Although \code{jhbuild} was originally developed to build -\code{[WWW]Gnome}, it is now able to build a number of the modules in -\code{freedesktop.org} CVS. Extending it to handle new modules is -usually trivial (assuming the build infrastructure matches the other -modules it handles). -\end{quote} - -In additional to retrieving source code from various SCM's (CVS, SVN, -arch, etc.), jhbuild has the ability to download tarballs via HTTP or -FTP. - -\code{jhbuild} has been adopted as an official \code{freedesktop.org} -project. You can find more information on the project's homepage -(\code{http://www.freedesktop.org/Software/jhbuild}). Bugs can be -filed in the Gnome Bugzilla (\code{http://bugzilla.gnome.org}). - -\subsubsubsection{Where to get it} - -It was necessary to slightly modify \code{jhbuild} for use with IPP -software. Therefore, you must checkout the \code{jhbuild} module from -the Pan-STARRS CVS tree. Please see the Pan-STARRS CVS Guide for help -on setting up and using CVS. \code{jhbuild} will need to be able to -find it's own source tree even after installation so you should choose -a checkout path that can be permanent. Something along the lines of -\code{$HOME/src} is recommended. - -\begin{verbatim} -cd -mkdir -p src -cd src -cvs co jhbuild -\end{verbatim} - -After running CVS you should see something like this: - -\begin{verbatim} -$ cvs co jhbuild -cvs checkout: Updating jhbuild -U jhbuild/.cvsignore -U jhbuild/COPYING -U jhbuild/ChangeLog -U jhbuild/HACKING -U jhbuild/Makefile -U jhbuild/README -U jhbuild/install-check.c -. -. -\end{verbatim} - -\subsubsubsection{Installing jhbuild into your home directory} - -\code{jhbuild} should be installed locally under your home -directory. This will require that you modify the \code{PATH} -environment variable so that you can run jhbuild after it has been -installed. - -\begin{verbatim} -cd jhbuild -make -make install -\end{verbatim} - -Which should look something like this: - -\begin{verbatim} -$ make -gcc -Wall -O2 -o install-check install-check.c -Run "make install" to install. -$ make install -Creating /home/moanui/jhoblitt/bin/jhbuild -Creating /home/moanui/jhoblitt/.gnome2/vfolders/applications/jhbuild.desktop -install -m755 install-check /home/moanui/jhoblitt/bin/install-check -install -m755 config.guess /home/moanui/jhoblitt/bin/config.guess -\end{verbatim} - -That will install the \code{jhbuild} executable under -\code{$HOME/bin}. You are responsible for including this path in your -\code{PATH} environment variable. It is highly recommended that you -add this to your \code{.bashrc} or equivalent shell login script. - -For the \code{bash} shell, place this line in your \code{.bashrc}: -\begin{verbatim} -export PATH=${HOME}/bin:${PATH} -\end{verbatim} -For the \code{tcsh} shell, place this line in your \code{.tschrc}: -\begin{verbatim} -setenv PATH ${HOME}/bin:${PATH} -\end{verbatim} - -\subsubsubsection{Configuring jhbuild} - -\code{jhbuild} is configured via an rc file that lives at -\code{${HOME}/.jhbuildrc}. Please note that this rc file is executed -as Python code; be careful! - -Example \code{.jhbuildrc}, suitable for cut and paste: - -\begin{verbatim} -# what profile to build? -moduleset = 'http://pan-starrs.ifa.hawaii.edu/project/IPP/software/modulesets/ipp12.modules' - -# modules to build by default -modules = [ 'pslib', 'psmodules' ] - -# where should working copies go? -jhroot = os.environ['HOME'] + '/jhroot' - -# where should tarballs be kept? -tarballdir = jhroot + '/src' - -# in what prefix should things be installed? (must be writable) -target = os.popen('config.guess').read().rstrip() -prefix = jhroot + '/' + target -checkoutroot = prefix + '/build' - -# extra arguments to pass to the autogen.sh script? -autogenargs = '--enable-maintainer-mode --disable-static' - -# use an alternative install program that preserves the -# mtime on header files if they haven't changed. Speeds -# up rebuilds. -os.environ['INSTALL'] = os.environ['HOME'] + '/bin/install-check' - -# don't try to use /usr/ucb/cc on Solaris -import sys -if sys.platform == 'sunos5': - os.environ['CC'] = 'gcc' -\end{verbatim} - -\subsubsubsection{Running jhbuild} - -\code{Jhbuild} can be executed as \code{jhbuild build -[modulename]}. Just \code{jhbuild} will build the packages specified -in the \code{modules} variable from your rc file. -\begin{verbatim} -jhbuild -\end{verbatim} -or -\begin{verbatim} -jhbuild build pslib -\end{verbatim} - -Run \code{jhbuild} list to get a list of the packages \code{jhbuild} -knows how to build. - -\begin{verbatim} -$ jhbuild list -cfitsio -gsl -fftw -libxml2 -mysql -pslib -psmodules -\end{verbatim} - -\code{jhbuild} supports many other commands. Please see \code{jhbuild ---help} for a complete list of options. - -\begin{verbatim} -$ jhbuild --help -usage: jhbuild [ -f config ] command [ options ... ] -Build a set of CVS modules (such as GNOME). - -Global options: - -f, --file=CONFIG use a non default configuration file - -m, --moduleset=URI use a non default module set - --no-interact do not prompt for input - -Commands: - gui build targets from a gui app - update update from cvs - updateone modules update a fixed set of modules - build [ opts... ] [modules] update and compile (the default) - buildone [ opts... ] modules build a single module - tinderbox [ opts... ] build non-interactively with logging - run program [ args... ] run a command in the build environment - shell start a shell in the build environment - sanitycheck check that required support tools exists - bootstrap build required support tools - list [ opts ... ] [modules] list what modules would be built - dot [ modules ] output a dot file of dependencies suitable - for processing with graphviz - info modules... prints information about modules - -Options valid for the build, buildone, tinderbox and update commands: - -s, --skip=MODULES treat the given modules as up to date - -t, --start-at=MODULE start building at the given module - -D date_spec set a sticky date when checking out modules - -Options valid for the build, buildone and tinderbox commands: - -a, --autogen always run autogen.sh - -c, --clean run make clean before make - -n, --no-network skip cvs update - -Options valid for the tinderbox command: - -o, --output=DIR directory to save build logs in - -Options valid for the list command: - -r, --show-revision show which revision will be built -\end{verbatim} - -\subsubsubsection{Dependancies} - -\code{jhbuild} has a fairly minimal set of dependencies --- far less -than what may be required to actually compile and install any -packages. However, if your system can meet the base requirements, -\code{jhbuild} should be able to bootstrap your build environment. -\begin{itemize} -\item A working C compiler (eg. \code{gcc}) -\item A working \code{libc} (eg. \code{glibc}) -\item Perl 5 with the \code{XML::Parser} module (needed by - \code{libtool}) -\item Python 2.? -\item Either \code{wget} or \code{curl} -\item GNU \code{M4} 1.4 -\item \code{tar} -\item \code{gzip} -\item \code{bzip2} -\end{itemize} - -\subsubsubsection{Bootstrapping} - -\code{jhbuild} has a limited ability to install some of the necessary -tools for maintaining software that configure its build environment -with the GNU autotools. - -This step is probably required on OSX and Solaris. Your mileage will -vary per Linux distribution but you can probably skip this step if -your distribution is around RedHat 9 vintage or newer. - -\begin{verbatim} -jhbuild bootstrap -\end{verbatim} - -\code{jhbuild} will then will begin to build a series of packages. - -\subsubsubsection{Using the jhbuild enviroment} - -As you've already seen, \code{jhbuild} is capable of setting up an -independent build environment under the (configurable) directory of -your choice. In order to link non-\code{jhbuild} management software -against this build environment a number of your shell's environment -variable have to be modified. \code{jhbuild} is capable of doing this -for you. The syntax for this is \code{jhbuild shell}, which as the -syntax implies, spawns a new shell with the proper environment -variables. - -This example demonstrates \code{jhbuild} setting up the dynamic -linkers default search path for you. - -\begin{verbatim} -$ echo $LD_LIBRARY_PATH - -$ jhbuild shell -$ echo $LD_LIBRARY_PATH -/home/moanui/jhoblitt/jhroot/i686-pc-linux-gnu/lib -\end{verbatim} - -%$ --- Emacs needs this to balance out the dollar signs - -A fair number of other variables are also adjusted for you. Enough so -that most (all?) \code{autoconf} configured software will be able to -find it's dependencies. - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsection{Perl} - -Here we describe setting up the Perl dependencies followed by the -IPP components. - -\subsubsection{Dependencies} -\label{sec:installing-perl-dependencies} - -If you have access to the \code{root} account, installation as -\code{root} is much easier. If not, you will have to go through -the more flaky installation as an unprivileged user. - -\subsubsubsection{Installation as root} - -Many of the Perl dependencies are available from the Comprehensive -Perl Archive Network (CPAN) at www.cpan.org. If you have root access -on your target machines, they can be very simply retrieved, built and -installed (replacing \code{MODULE_NAME} for each module): - -\begin{verbatim} -> su - -Password: -> cpan -[...] -cpan> install MODULE_NAME -[...] -cpan> quit -\end{verbatim} - -Follow the prompts. It's usually safe to accept the default (simply -hit enter) in response to most questions. - -If you get into trouble, try: \code{force install MODULE_NAME}. - -You can also try to use the \code{Bundle::PS} as described below if -you're feeling adventurous. - -\subsubsubsection{Installation as unprivileged user} - -To install modules from CPAN with the \code{CPAN.pm} interface, you -need to setup a CPAN configuration file in your home directory. Then -\code{CPAN.pm} can walk you through setting up the most important -configuration values. Unfortunately, there is some variation in the -behavior of the various versions of \code{CPAN.pm} that have shipped -with Perl. Some (most) of these variants will not correctly create a -configuration files that allows a non-\code{root} user to install -modules outside of "system" paths. In order to make sure that you get -a "correct" CPAN configuration file you need to ``prime'' it with a -few values. - -First you need to create the directory in which the CPAN configuration file will live. -\begin{verbatim} -> mkdir -p .cpan/CPAN/ -\end{verbatim} - -Then we need to create a partial configuration file. Note that this example -assumes that you want to install your perl modules under -\code{$HOME/local/lib/perl5}. - -\begin{verbatim} -> echo "\$CPAN::Config = {" >> .cpan/CPAN/MyConfig.pm -> echo " makepl_arg => q[PREFIX=$HOME/local/]," >> .cpan/CPAN/MyConfig.pm -> echo " mbuildpl_arg => q[--install_base $HOME/local/]," >> .cpan/CPAN/MyConfig.pm -> echo "};" >> .cpan/CPAN/MyConfig.pm -> echo "1;" >> .cpan/CPAN/MyConfig.pm -> echo "__END__" >> .cpan/CPAN/MyConfig.pm -\end{verbatim} - -%$ --- Emacs needs this to balance the previous dollar sign - -Now you need to invoke \code{CPAN.pm} so it can walk you through -configuring the rest of the required values. This is an example of -one possible configuration with \code{CPAN.pm} version 1.8802. -\textbf{Your version of CPAN.pm may present you with different -prompts.} Use your common sense. If in doubt, it is generally safe -to simply hit enter (and accept the default). - -\begin{verbatim} -> perl -MCPAN -e shell -CPAN: File::HomeDir loaded ok -Sorry, we have to rerun the configuration dialog for CPAN.pm due to -the following indispensable but missing parameters: - -build_cache, build_dir, cache_metadata, cpan_home, ftp_proxy, http_proxy, -index_expire, inhibit_startup_message, keep_source_where, make_arg, -make_install_arg, mbuild_arg, mbuild_install_arg, mbuild_install_build_command, -no_proxy, prerequisites_policy, scan_cache, urllist - - -The following questions are intended to help you with the -configuration. The CPAN module needs a directory of its own to cache -important index files and maybe keep a temporary mirror of CPAN files. -This may be a site-wide directory or a personal directory. - - - -I see you already have a directory - /home/moanui/jhoblitt/.cpan -Shall we use it as the general CPAN build and cache directory? - -CPAN build and cache directory? [/home/moanui/jhoblitt/.cpan] - - -Unless you are accessing the CPAN via the filesystem directly CPAN.pm -needs to keep the source files it downloads somewhere. Please supply a -directory where the downloaded files are to be kept. [/home/moanui/jhoblitt/.cpan/sources] -Directory where the build process takes place? [/home/moanui/jhoblitt/.cpan/build] - - -How big should the disk cache be for keeping the build directories -with all the intermediate files? - -Cache size for build directory (in MB)? [100] - - -The CPAN indexes are usually rebuilt once or twice per hour, but the -typical CPAN mirror mirrors only once or twice per day. Depending on -the quality of your mirror and your desire to be on the bleeding edge, -you may want to set the following value to more or less than one day -(which is the default). It determines after how many days CPAN.pm -downloads new indexes. - -Let the index expire after how many days? [1] - - -By default, each time the CPAN module is started, cache scanning is -performed to keep the cache size in sync. To prevent this, answer -'never'. - -Perform cache scanning (atstart or never)? [atstart] - - -To considerably speed up the initial CPAN shell startup, it is -possible to use Storable to create a cache of metadata. If Storable -is not available, the normal index mechanism will be used. - -Cache metadata (yes/no)? [yes] - - -The CPAN module can detect when a module which you are trying to build -depends on prerequisites. If this happens, it can build the -prerequisites for you automatically ('follow'), ask you for -confirmation ('ask'), or just ignore them ('ignore'). Please set your -policy to one of the three values. - -Policy on building prerequisites (follow, ask or ignore)? [ask] follow - - -Every Makefile.PL is run by perl in a separate process. Likewise we -run 'make' and 'make install' in separate processes. If you have -any parameters (e.g. PREFIX, LIB, UNINST or the like) you want to -pass to the calls, please specify them here. - -If you don't understand this question, just press ENTER. -Parameters for the 'make' command? -Typical frequently used setting: - - -j3 # dual processor system - -Your choice: [] -Parameters for the 'make install' command? -Typical frequently used setting: - - UNINST=1 # to always uninstall potentially conflicting files - -Your choice: [] UNINST=1 - - -The next questions deal with Module::Build support. - -A Build.PL is run by perl in a separate process. Likewise we run -'./Build' and './Build install' in separate processes. If you have any -parameters you want to pass to the calls, please specify them here. - -Parameters for the './Build' command? -Setting might be: - - --extra_linker_flags -L/usr/foo/lib # non-standard library location - -Your choice: [] -Do you want to use a different command for './Build install'? -Sudo users will probably prefer: - - su root -c ./Build -or - sudo ./Build -or - /path1/to/sudo -u admin_account ./Build - -or some such. Your choice: [./Build] -Parameters for the './Build install' command? -Typical frequently used setting: - - --uninst 1 # uninstall conflicting files - -Your choice: [] --uninst 1 - - -If you're accessing the net via proxies, you can specify them in the -CPAN configuration or via environment variables. The variable in -the $CPAN::Config takes precedence. - -Your ftp_proxy? [] -Your http_proxy? [] -Your no_proxy? [] -You have no /home/moanui/jhoblitt/.cpan/sources/MIRRORED.BY - I'm trying to fetch one -CPAN: LWP::UserAgent loaded ok -Fetching with LWP: - http://www.perl.org/CPAN/MIRRORED.BY - - -Now we need to know where your favorite CPAN sites are located. Push -a few sites onto the array (just in case the first on the array won't -work). If you are mirroring CPAN to your local workstation, specify a -file: URL. - -First, pick a nearby continent and country by typing in the number(s) -in front of the item(s) you want to select. You can pick several of -each, separated by spaces. Then, you will be presented with a list of -URLs of CPAN mirrors in the countries you selected, along with -previously selected URLs. Select some of those URLs, or just keep the -old list. Finally, you will be prompted for any extra URLs -- file:, -ftp:, or http: -- that host a CPAN mirror. - -(1) Africa -(2) Asia -(3) Central America -(4) Europe -(5) North America -(6) Oceania -(7) South America -Select your continent (or several nearby continents) [] 5 - -(1) Bahamas -(2) Canada -(3) Mexico -(4) United States -Select your country (or several nearby countries) [] 4 - -(1) ftp://carroll.cac.psu.edu/pub/CPAN/ -(2) ftp://cpan-du.viaverio.com/pub/CPAN/ -(3) ftp://cpan-sj.viaverio.com/pub/CPAN/ -(4) ftp://cpan.calvin.edu/pub/CPAN -(5) ftp://cpan.cs.utah.edu/pub/CPAN/ -(6) ftp://cpan.cse.msu.edu/ -(7) ftp://cpan.erlbaum.net/CPAN/ -(8) ftp://cpan.glines.org/pub/CPAN/ -(9) ftp://cpan.hostrack.net/pub/CPAN -(10) ftp://cpan.llarian.net/pub/CPAN/ -(11) ftp://cpan.mirrors.redwire.net/pub/CPAN/ -(12) ftp://cpan.mirrors.tds.net/pub/CPAN -(13) ftp://cpan.netnitco.net/pub/mirrors/CPAN/ -(14) ftp://cpan.pair.com/pub/CPAN/ -(15) ftp://cpan.teleglobe.net/pub/CPAN -(16) ftp://cpan.uchicago.edu/pub/CPAN/ -40 more items, hit RETURN to show them -Select as many URLs as you like (by number), -put them on one line, separated by blanks, hyphenated ranges allowed - e.g. '1 4 5' or '7 1-4 8' [] 14 11 12 - -Enter another URL or RETURN to quit: [] -New set of picks: - ftp://cpan.pair.com/pub/CPAN/ - ftp://cpan.mirrors.redwire.net/pub/CPAN/ - ftp://cpan.mirrors.tds.net/pub/CPAN - - -Please remember to call 'o conf commit' to make the config permanent! - - -cpan shell -- CPAN exploration and modules installation (v1.8802) -ReadLine support enabled - - cpan[1]> o conf commit -commit: wrote '/home/moanui/jhoblitt/.cpan/CPAN/MyConfig.pm' -\end{verbatim} - -%$ --- Emacs needs this to balance the previous dollar sign - -Now we need to install the module that installs the other modules. - -\begin{verbatim} -cpan> install Module::Build -\end{verbatim} - -Exit out of cpan: - -\begin{verbatim} -cpan> exit -\end{verbatim} - -In order to use of the installed modules, we need to setup an -environment variable called \code{PERL5LIB} so that 'perl' can find -them. To do this, we need to know where under 'perl5' our modules -were actually installed. This will set variable with the version of -Perl that you are using. The easiest way to do this is just just look -in the root of the path where we did the install. - -\begin{verbatim} -> ls local/lib/perl5/ -5.8.8 site_perl -\end{verbatim} - -That means we're using perl 5.8.8 and \code{PERL5LIB} needs to be -setup as following: -\begin{verbatim} -export PERL5LIB=$HOME/local/lib/perl5/5.8.8:$HOME/local/lib/perl5/site_perl/5.8.8 -\end{verbatim} - -Now we should install the basic compliment of helper modules that -\code{CPAN.pm} needs to function fully. Go back into CPAN (\code{perl --MCPAN -e shell}) and: - -\begin{verbatim} -cpan> install Bundle::CPAN -\end{verbatim} - -You can quit out of the CPAN shell at this point with the `exit` -command or do the following few steps in another shell We're ready to -install the full set Perl module dependencies for IPP software. In -order to make this process a bit easier on the end user a "Bundle" -module has been created. In order to use it you need to create a -directory (if it doesn't already exist) called Bundle under your .cpan -directory. - -\begin{verbatim} -> mkdir -p .cpan/Bundle -\end{verbatim} - -The file \code{PS.pm} should copied into this directory: - -\begin{verbatim} -cp /path/to/PS.pm .cpan/Bundle/ -\end{verbatim} - -Back in the CPAN shell, 'force' the install of the PS Bundle. The -'force' keyword instructs the shell to ignore any tests failures. -This is necessary as some of the modules 'DBD::mysql'/etc. require a -properly working database setup in order for the tests to pass. You -will most likely be prompted for input by several the modules. It is -safe to answer with a carriage return to all questions. If it insists -on a path to \code{httpd}, hit \code{CTRL-C} and it will go on to the -next step. - -\begin{verbatim} -cpan> force install Bundle:PS -\end{verbatim} - -For further instructions on installing Perl modules from CPAN ''by -hand', see: -\begin{verbatim} -http://www.cs.ucsc.edu/~you/notes/perl-module-install.html -\end{verbatim} - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% - -\subsubsection{Modules and scripts} - -Pan-STARRS Perl modules and scripts are installed by executing the -following in the directory containing the source: -\begin{verbatim} -> perl Build.PL -> ./Build -> ./Build install --prefix=/path/to/install/ -\end{verbatim} - -Make sure you set your \code{PERL5LIB} environment variable to the -installation path (here, \code{/path/to/install/}). - -\section{Configuration} - -Correct use of the IPP depends on several configuration files. We -distinguish below between configuration files required for the image -processing and those for running the process scheduler, PanTasks. -Note, however, that the Perl scripts called by PanTasks to run the -processing do use the site and camera configuration files principally -used for the image processing. - -\subsection{Image Processing} - -Configuration information for the image processing is provided on four -levels: the site, camera, format and recipe configurations. Each uses -the ``MetaData Configuration'' (MDC) file format, which is briefly -described below; for a more detailed description, see the psLib SDRS -(PSDC-430-007). - -The configuration levels for the image processing components of the -IPP are: -\begin{itemize} -\item Options for the particular site installation of the - pipeline: the {\it site}; -\item Options specifying the instrument setup: the {\it camera}; -\item Options specifying the format of the FITS file: the {\it - format}; and -\item Options specifying the particular parameter choices that affect - the details of an analysis: the {\it recipe}. -\end{itemize} -Note that these are arranged in an hierarchical order, with the site -configuration being the most general, and the recipe configurations -the most specific. For example, not all sites will have to deal with -all cameras, and different cameras may require different recipes at -different times according to their particular quirks, analysis -experimentations, or their evolution. - -We have provided examples of each of these configurations in the -\code{config} component of the IPP, which should be a useful guide for -setting up your own. The Pan-STARRS IPP Configuration Guide -(PSDC-430-???) has all the detailed information. - -\subsubsection{Overview of MDC format} - -psLib defines a \code{psMetadata} structure which can carry labeled -data of arbitrary types. Originally designed for carrying the data in -FITS headers, the \code{psMetadata} have proved so generally useful -that we use them for our configurations (and a multitude of other -uses!). We have designed a human-readable text-based format --- the -``MetaData Configuration'' (MDC) format --- which we use for this end. - -Each simple entry in an MDC file must contain the name, type and -value. Each of these is on a single line, separated by whitespace, -and in that order. Comments may be placed at the end of the line (or -on a blank line), after a hash mark (\code{#}). Whitespace at the -beginning and end of strings (either the name, value or comment) are -stripped. - -The simple types follow the psLib types. Integers are specified by a -letter indicating if the integer is signed (\code{S}) or unsigned -(\code{U}) and a number indicating the dynamic range in bits (8, 16, -32 or 64); e.g., \code{U8} is commonly used for bit mask values, -\code{S32} is commonly used for ordinary integer values. Floating -point values are specified by the letter \code{F} and a number -indicating the precision in bits (32 or 64): \code{F32} (single -precision) or \code{F64} (double precision). Strings are specified by -\code{STR}. Times may be specified with the following types: -\code{UTC,UT1,TAI,TT}; values for the time are expected to be in -ISO8601 format (\code{YYYY-MM-DDTHH:MM:SS.sZ}). - -Names are traditionally all-caps, though there is no reason why they -must be; the names are case-sensitive. A name may not be repeated -unless it has previously been declared to be of type \code{MULTI} (no -value should be provided with this declaration): -\begin{verbatim} -COMMENT MULTI -COMMENT STR Having more than one COMMENT like this -COMMENT STR is permitted because of the MULTI. -\end{verbatim} - -A hierarchy can be made using the \code{METADATA} type, which signals -a new level: -\begin{verbatim} -JANITOR METADATA - NAME STR John Doe - PAY F32 1234.56 - ECCENTRICITY STR 9.87 -END -\end{verbatim} -Note that a \code{METADATA} block is closed by an \code{END}. No -identing need be done within a \code{METADATA} block, but it is useful -to be able to see the levels at a glance (just like in a C program). -\code{METADATA} blocks may be nested within \code{METADATA} blocks, -probably down as far as you have the patience to try. Note that -\code{MULTI} declarations only apply to the current level --- there is -no inheritance. - -The above format can be long if there are many \code{METADATA}s with -similar contents. For this reason, we provide the \code{TYPE} -declaration, which generates a \code{METADATA} with the contents each -of type \code{STR}: -\begin{verbatim} -TYPE EMPLOYEE NAME PAY ECCENTRICITY -\end{verbatim} -Now, the type \code{EMPLOYEE} may be used, with string values (NB: no -spaces allowed!) to specify multiple entries: -\begin{verbatim} -JANITOR EMPLOYEE JohnDoe 1234.56 9.87 -PROGRAMMER EMPLOYEE FooBar 2345.67 1.00 -\end{verbatim} -This is the same as the much longer block: -\begin{verbatim} -JANITOR METADATA - NAME STR JohnDoe - PAY STR 1234.56 - ECCENTRICITY STR 9.87 -END -PROGRAMMER METADATA - NAME STR FooBar - PAY STR 2345.67 - ECCENTRICITY STR 1.00 -END -\end{verbatim} -Like the \code{MULTI}, \code{TYPE} declarations only apply to the -current level. - - -\subsubsection{Setting up configuration files} - -You will generally want to link \code{~/.ipprc} to the site -configuration file (\code{ipprc.config} which gets installed in -\code{PREFIX/share/ippconfig/} where \code{PREFIX} is your -installation prefix). Then link \code{~/.ipp} to the \code{ippconfig} -directory to save hacking the \code{PATH} in the site configuration. - -\subsection{PanTasks} - -\subsubsection{Paths} - -Throughout PanTasks, a file may be referred to as: -\begin{itemize} -\item \code{/path/to/file.ext} --- not a URI, but it should work. -\item \code{file:///path/to/file.ext} --- the URI version of the above. -\item \code{path://PATH/file.ext} --- Uses the \code{DATAPATH} in the site configuration to set the path. -\end{itemize} - -\section{Subsystem specifics} - -\tbd{Need to document command-line arguments, perhaps even -algorithms.} - -\subsection{psLib} -\subsection{psModules} -\subsection{psphot} -\subsection{psastro} -\subsection{ppStats} -\subsection{ppImage} -\subsection{ppMerge} -\subsection{ppNorm} -\subsection{ippdb} -\subsection{ippTools} -\subsection{ippScripts} -\subsection{ippTasks and panTasks} -\subsection{ippMonitor} - \section{Trouble Shooting Build Issues}