 %%% $Id: manual.tex,v 1.12 2008-02-19 20:06:58 eugene Exp $
\documentclass[panstarrs,psreport,spec]{panstarrs}

% basic document variables
\title{Pan-STARRS Image Processing Pipeline}
\subtitle{User's Guide}
\shorttitle{PS IPP UG}
\author{Eugene A. Magnier, Paul A. Price, Joshua Hoblitt}
\audience{Pan-STARRS IPP Users}
\group{Pan-STARRS Image Processing Pipeline}
\project{Pan-STARRS}
\organization{Institute for Astronomy}
\version{DR}
\docnumber{PSDC-430-???}

% allow paragraphs to be listed in TOC for now
\setcounter{tocdepth}{3}

\begin{document}
\maketitle

% -- Revision History --
\RevisionsStart
% version     Date         Description
DR      & 2007-01-31 & First draft \\ \hline
\hline
\end{tabular}
\end{center}

\inserttbd

\inserttbr
\pagebreak

\tableofcontents
\pagebreak

\listoffigures
\pagebreak
\pagenumbering{arabic}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Overview}

The Pan-STARRS Image Processing Pipeline (IPP) is a software suite for
the reduction of astronomical images.  Although designed principally
for the Pan-STARRS project, it is highly configurable and extensible,
and hence highly applicable for other projects.

The IPP consists of a number of elements: programs which perform
specific image analysis steps; a parallel processing environment in
which the image processes steps are automatically sequenced and
tracked; a databasing system for associating and calibrating the
detections of astronomical objects.  These programs use a number of
internal and external libraries (see \S\ref{sec:dependencies}),
simplifying the process of adding additional elements as needed.

The core functionality of the IPP is implemented by NN principal
programs: \code{psphot} is used for the detection and analysis of
objects in astronomical images; \code{psastro} performs the
astrometric calibration of the images; \code{ppImage} is used for a
wide range of single image analysis step, include image detrending and
the generation of Q/A images and plots; \code{ppMerge} combines
collections of input detrend images into high-quality masters;
\code{pswarp} transforms images between different pixel projections
and coordinate systems; \code{ppStack} is used to combine multiple
science images, including outlier rejection which is sensitive to
varying image quality; \code{ppSub} performs image differencing
incorporating the variations in seeing with multiple optional
convolution kernels.

In addition to the image-level analysis, the IPP provides DVO, the
Desktop Virtual Observatory, a database system for tracking
astronomical objects and detections.  DVO includes tools for querying
and manipulating the contents of the object database. It also provides
a number of tools for performing the analysis of quantities at the
database level.  These include: \code{relphot}, which performs relative
photometry and calculates robust ensemble photometry for objects;
\code{relastro}, which calculates average astrometry quatities for
objects, including proper motion and parallax, as well as iterative
improvements to the astrometric calibrations of images injested by the
database; \code{uniphot}, which is used to calculate consistent
photometric calibrations and transformations.  

The IPP provides a system for bulk automation of all stages of the
image analysis process within a parallel processing environment.  The
parallelization scheme is very light-weight, and makes use of
distributed UNIX jobs operating on multiple machines within a cluster.
Process scheduling and distribution of the resulting jobs to the
parallel cluster is performed by \code{pantasks}.  A set of
\code{pantasks} scripts, \code{ippTasks}, is used to define the
processing stages.  

Data flow within the IPP is managed via interaction with a collection
of database tables, representing the steps of the analysis pipeline.
The IPP examines the state of these database tables to determine which
jobs should be performed next.  A set of simple text files
(\code{dbconfig}) defines the database scheme and are also used to
automatically generate C code used to query the database tables.
Higher level command-line programs (\code{ippTools}), built on these
APIs, are available to both the end user and to \code{pantasks)} to
examine the state of the pipeline database.  A set of Perl scripts
(\code{ippScripts}) provide the glue between the individual IPP
analysis programs and the parallel processing environment.  A
web-based tool, \code{ippMonitor} provides the user interface for
monitoring the pipeline and the current status of the data analysis
process.

\note{this is not needed here: move elsewhere}

\code{psLib} is the Pan-STARRS library, containing a range of
low-level structures (e.g., vectors, images, etc.) and functions
(e.g., write an image to FITS file), most of which are not specific to
astronomy.  \code{psModules} is built on top of psLib, and provides
higher-level capabilities that are centered around astronomical data
manipulation.

We have some Perl modules which are used to facilitate the data flow
in processing: \code{PS::IPP::Metadata::Config} reads ``metadata
configuration'' files; \code{PS::IPP::Metadata::Stats} interprets
statistics from \code{ppStats}; \code{PS::IPP::Metadata::List}
interprets output lists from the \code{ippTools}; and
\code{PS::IPP::Config} reads the configuration files.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Using the IPP}

\subsection{Conceptual Overview}

The operating goals of the IPP can be broken down into several major categories:
\begin{itemize}
\item reduction, analysis, and calibration of individual astronomical
  science images.
\item combinations (additions and subtractions) of groups of science
image, along with their analysis and calibration.
\item analysis of the ensemble properties of astronomical objects
detected from many images, including improved astrometric and
photometric calibrations.
\item construction of the necessary master detrend images and other
calibration information needed to perform the science analysis.
\item enabling further investigation into the data characteristics by
tracking sufficient metadata and providing tools for this analysis.
\end{itemize}
Furthermore, the IPP is designed to perform these tasks with a high
degree of automation and for large collections of data from multiple
instruments.

\begin{figure}
\psfig{file=userview.ps,width=6in,angle=0}
\caption{User's view of the IPP}
\label{fig:ipp-userview}
\end{figure}

A cartoon of the user's view on the IPP is seen in
Figure~\ref{fig:ipp-userview}.  Images are injected into the IPP,
either automatically by software interacting with the telescope
systems or manually by the user.  The images are processed by the IPP
through any number of analysis steps; the specifics of the analysis
will depend on the type of image, the overall configuration, and
additional requests the user may make.  The user may use the program
\code{pantasks} to control and monitor the ongoing processing, or view
summary result information in the ippMonitor web browser.  As science
images are processed, the information about the objects detected in
these images is passed to the DVO database.  The user may explore the
results or perform further science analysis by interacting with the
DVO database via the DVO Shell.  From this tool, the user may, for
example, create color-magnitude diagrams of regions in the sky, or
plot light curves of specific objects.  The DVO Shell makes it
possible to performing data extraction and data visualization on the
DVO database.

Below, we discuss in more detail how a user would use the IPP in
several different operating scenarios: to reduce the results from an
individual observing run; to perform automatic analysis of data from a
telescope with a blocked-out schedule, as for example the CFHT Megacam
Queue; to perform automatic analysis of data from a continuous survey
instrument such as the Pan-STARRS PS1 Telescope.  These difference
scenarios have slightly different conditions and underlying
assumptions, and make a good set of examples for the types of
decisions a user much make.

All of these scenarios have a few common elements.  The first goal of
the user is to explore the characteristics of the instrument in
question and to define initial master detrend images.  The intial
master detrend images may be used to test the validity of new detrend
images and thereby test the evolution of the instrument response.
Next, the user will make an initial analysis of science images.  This
initial analysis may be used to construct further master detrend or
calibration information (eg, flat-field correction images; reference
astrometric parameters; deeper astrometric reference catalog).
Finally, the user may begin large-scale automatic analysis of the
complete data collection or the real-time data stream.

\subsection{Projects and Databases}

The IPP has the concept of a 'project': a related group of data that
are analysed within the same context.  A project may consist of data
from many different camera, or the user may find it more convenient to
divide data even from the same camera into multiple independent
projects.  A single installation of the IPP may operate on many
separate projects at the same time.  

Within a single project, the user may also choose to define more than
one output DVO database.  For example, within the Pan-STARRS project,
the detections from the different survey modes, with different depths
and cadences, will initially be saved in separate DVO databases, and
only later joined for improved calibration and more in-depth analysis.

\subsection{User Interface Tools}

In this section, we introduce the user to the IPP user interface
tools.  These are the elements of the IPP which provide feedback to
the user or allow the user to initiate a new type of analysis or to
start and stop the automatic processing of the IPP.

\subsubsection{ippMonitor}

\begin{figure}
\psfig{file=ippMonitor.eps,width=6in,angle=0}
\caption{Example Screen Shot from the ippMonitor}
\label{fig:ippMonitor}
\end{figure}

The \code{ippMonitor} is a web-based tool that allows the end-user to
explore at a high level the results of the IPP analysis.  There is an
initial login page, after which the user is asked to choose a project.
Once a project is selected, the user may view pages which show summary
information of the images that have been processed at each of the
different stages.  There are top-level sections for the detrend
analysis, the single science image analysis, and the different image
combinations.  Further clicking takes the user to more detailed
information about the individual images: jpegs of the reduced image,
plots showing the psf modeling and the astrometric analysis, etc.

\note{to be added to ippMonitor: pages showing summary plots from DVO}

\subsubsection{DVO Shell}

The DVO database is the primary output destination for results from
the IPP.  The DVO shell is the basic user interface to the DVO
database.  Within the DVO shell, the user may extract measurments from
the database, create plots, or view the locations of objects and
images on projections of the sky.

A complete user's guide is available at DVO REF.  Here are a few
quick-start details:

Start the DVO shell by typing: \code{dvo}.  You will be given a
prompt, at which you may type commands.  This is an interactive shell,
with standard readline comand-line editing features.  The \code{help}
command gives further information about the commands.  Commands will
give additional information if they are given a '-help' option, or in
many cases if the command is typed without arguments.

\begin{figure}
\psfig{file=skycells.ps,width=4in,angle=0}
\psfig{file=tau-region.ps,width=2in,angle=0}
\caption{Example Plots from DVO}
\label{fig:dvo-sample}
\end{figure}

\begin{verbatim}
catdir skycells
region -n view1 0 0 80 ait
images

catdir taurus-project
region -n view2 64.8 26.1 4.0
images
avextract ra,dec where (g - i > 2.0)
cplot ra dec -c red
avextract ra,dec where (g - i < 0.5)
cplot ra dec -c blue
\end{verbatim}

Choose the DVO database with the command \code{catdir}.  \note{you
must specify the path to the top of the database directories}.  Define
a view port for plotting objects and images on a sky projection with:
\code{region RA DEC range [projection]}.  The projection may be
\code{ait} (aitoff projection), \code{sin} (sin projection), ....  

\subsubsection{Pantasks}

Pantasks is the program which schedules the analysis to be performed
and distributes the analysis jobs across the parallel processing
environemnt.  A more complete user's guide is available at REF. Here
we give a basic overview of starting up the IPP.  In this example, we
are using the stand-alone pantasks program.  It is also possible to
run pantasks in a client / server mode.  In this mode, the server runs
continuously in the background, and multiple users may interact with
the same pantasks server via the pantasks client tool.  Most of the
commands are the same in both contexts.  However, for security reasons
in the client / server mode, the clients are not able to issue certain
commands to the server.  Thus, there are some differences in the details.

Start \code{pantasks}.  Load the collection of IPP tasks with the
command: \code{module pantasks.pro}.  Add remote analysis hosts to the
parallel processing controller (pcontrol): \code{controller host add
NAME}, where NAME is the name of a machine set up to run the IPP.  Add
the IPP project(s) of interest: \code{add.database (project)}.  Start
the analysis: \code{run}.  Check the current status with
\code{status}.  If something goes wrong, stop the processing with
\code{stop} or \code{halt}.  The former will continue to accept the
results from the jobs already launched, while the latter halts all
processing. \note{halt does not current stop pcontrol}.  

\subsubsection{ippTools \& ippScripts}

The IPP uses a \code{mysql} database to keep track of the data to be
analysed and the results of analysis steps.  The IPP user interacts
with this database via a set of programs called \code{ippTools} and
\code{ippScripts}.  The difference between these is that the
\code{ippTools} are somewhat lower-level programs to interact directly
with the database, while the ippScripts perform a more complex
operation, sometimes wrapping multiple calls to commands within
\code{ippTools}.   

\begin{itemize}
\item \code{ipp_serial_inject.pl} : inject data into the IPP database.
\item \code{dettool -definebyquery} : define a new detrend run.  With
  this command, the user defines a set of criteria which define a
  collection of input images to be used to generate a detrend image.
  The parameters which define the output detrend image also
  specified.
\item \code{dettool -updatedetrun} : manually set the state of a
  detrend run.  
\item \note{extend this list...}
\end{itemize}

The ippTools accept a common set of command-line arguments which
modify their behavior:
\begin{itemize}
\item \code{-pretend} : show the expected result, but do not apply the
  results to the database
\item \code{-simple} return results with one line for each output
  component.  Without this flag, the results are returned with the
  versbose, but self-documenting, psMetadataConfig format.
\item \code{-dbname} specify the database to use
\item \note{extend this list...}
\end{itemize}

The collection of programs which are used to initiate different stages
of the IPP analysis is called \code{ippTools}.  This suite of programs 

\subsection{Operational Scenarios}

\subsubsection{Reducing an Observing Run}

For a single observing run, the user will typically have data from
several nights, both science and raw detrend images.  The following
steps illustrate the analysis with data from a sample observing run.
We assume the user has already installed the IPP software and the
external software (mysql, apache, php, etc), set up the configuration
for the camera, defined a project for this analysis, started pantasks,
and loaded the analysis tasks.  This example shows the steps to build
the detrend data and process the science data.

The first step is to inject all of the data into the IPP database with
the command \code{ipp_serial_inject.pl}.  It is possible to define
abstract data storage paths in the user's \code{.ipprc} file in the
\code{DATAPATH} section. Each line gives the relationship between an
abstract data location and the real-world UNIX path to that data.  The
script \code{ipp_serial_inject.pl} will interpret the location of the
data in terms of the DATAPATH hierarchy when identifying the data for
the database. When injecting the data, it is convenient to define a
working directory with the \code{--workdir} option to
\code{ipp_serial_inject.pl}.  The science processing steps will use
this location to store the output data products.

% complete command here

As the images are injected, they will be placed in the table of
\code{new} exposures.  If \code{pantasks} is running with the modules
loaded, the images will be processed for registration.  In this step,
a minor analysis of the image statistics and an examination of the
header information is performed, and the results used to provide
further information about the images in the database.  As images are
registered, they migrate from the \code{new} to the \code{raw}
tables.  These tables can be view in the \code{ippMonitor} under the
\code{Load & Setup} pages.

Once the images have been injected, the user will want to create, in
order, bias, dark, shutter, flat, and fringe frames.  As these are
created, they may be used to generate the next level of image.  It is
important not to define, eg, a flat run before the lower-precedence
elements are defined.  At the moment, it is not possible to define a
detrend analysis which blocks until a previous detrend analysis is
complete.  To start, define a bias based on a query to the database:

% example command hrere

\code{dettool -dbname name -definebyquery -det_type bias etc}

For a small observing run, it is probably possible to include all bias
images in the initial pass.  IPP will iteratively reject complete
input images which are outliers from the ensemble, as well as
individual pixels.  In the \code{ippMonitor}, rejected images will be
greyed out in the residual pages.  Examine the resulting residual
images.  If there are patterns to the residuals, it may be necessary
to define subgroups, perhaps by time or ccd temperature.  For example:

% example command here.

After a bias or set of biases are defined, follow the same process for
a dark frame.  It may be necessary to supply a non-linear dark current
correction (perhaps one for each amplifier or chip).  The script
\code{ipp_darkstats.pl} can be used to examine this trend.  

If a shutter correction is necessary, make sure to obtain data which
can be used to generate the correction: a series of flat-field images
with a range of exposure times, particularly near the short end of the
range.  If the shutter correction will be skipped, set the SHUTTER
entries in the camera's ppImage.config file to FALSE.

A flat-field image is next.  Initially, we must define a raw
flat-field image, from dome or twilight flat-field images.  Below, we
will discuss the generation of a flat-field correction image from
dithered science images.  Do not forget to specify the filter for each
flat-field image

% example command.

Up to now, we have been ignoring the bad pixel mask.  At this point,
the user may generate a bad pixel mask from the collection of
processed flat-field images.  Use the script \code{UNKNOWN} to build a
mask image.  Turn on the use of the mask by setting \code{MASK} to
true in the \code{ppImage.config} file.  It is probably a good idea to
now re-run the previous stages with the mask applied: this will result
in better statistics for the residual images.

If certain filters require a fringe image, the user may inform the IPP
of this fact by setting the FRINGE.FILTERS entry in the
\code{ppImage.config} file.  Use a set of night-time sky images to
generate a fringe frame. \tbd{How to specify more than one fringe
mode?}

Activate the chip-level science analysis tasks in pantasks: chip.on.
With this off, no data will be processed before the detrend images are
ready.  Once this is turned on, the science images will start to be
processed.

\tbd{Discuss blocks, re-queuing science images, analysis versions}.

\tbd{Illustrate how to specify the DVO output database}

\subsubsection{Pipeline Reduction : PI-Oriented Observatory}

An PI-oriented observatory has the opportunity to operate the analysis
system on a somewhat larger scale than the individual user.  At a
PI-oriented observatory, there will normally be a rotating suite of
instruments, each of which is mounted on the telescope for a
substantial number of nights.  Over the course of months and years,
the observatory will collect enough data from the instruments to
perform a much more detailed level of analysis of the instrument than
is possible in a single observing run by a single PI.  In such a
situation, the observatory may use the IPP to generate and track the
high-quality master detrend images, to apply the detrend images to the
data as it is being collected, and to monitor the instrumental zero
points and other calibration information.  The basic concepts of
running the IPP at a PI observatory are identical to that of a single
user.  There are, however, a few modifications which may be
interesting.

First, it is natural to automatically inject and register the data as
it is obtained at the telescope.  The telescope readout software could
be modified to call \code{ipp_serial_inject.pl} after each image is
obtained.  Alternatively, this step could take place downstream
wherever the data are archived.  In an observatory environment, more
so than in a single-user situation, it may be perfereable to use the
Nebulous infrastructure to manage the location and copies of the image
data.  \tbd{Define an inject mechanism that uses Nebulous}.

Second, it is interesting to consider on what timescale to test and
re-build the master detrend images.  In the single user case, the
possible input detrend images are fairly limited.  As illustrated
above, the single user may use the IPP to define master detrend images
and test the consistency of the inputs against these masters.  In an
observatory environment, it may be more appropriate to build the
masters only on regular intervals when there is a suspicion the
instrument may have changed.  The periods in which the instrument is
mounted define an obvious possible starting point for this period.
Alternatively, the detrend creation steps in the IPP can be defined to
run in verify mode on a nightly basis to test the current state of the
instrument.  For the PI-oriented observatory, the number of variables
may make it difficult to rely on these results in an automatic
fashion; user examination of the results with \code{ippMonitor} may be
needed to decide if a new master should be built.

\subsubsection{Pipeline Reduction : Survey Observatory}

A telescope used as a survey instrument has a somewhat different, more
limited range of circumstances, than a PI-oriented observatory.  There
is generally a pre-defined schedule, perhaps with only a single
instrument.  The instrument configuration is more controlled, and the
types of observations are more consistently defined.  In this case, an
automatic validation process can be more reliably used.  The IPP can
be set up to run the detrend validation stage at the beginning of
every night, to automatically generate a new master if any of the
input detrend images are out of spec, and to start automatic
processing of the night's data with the new master images when they
are ready. 

\section{IPP Components}

\tbd{Need to document command-line arguments, perhaps even
algorithms.}

\subsection{Analysis Programs}

\subsubsection{psphot}

The detection and characterization of astronomical objects within an
image is performed by the IPP component called \code{psphot}.  This
software may be used as a stand-alone program, or it may be called by
other IPP programs to operate on the images they generate.  A more
detailed guide is available at REF.  Here we summarize the basic
operation and command-line options.

The basic call to the stand-alone version of \code{psphot} is:
%
\code{psphot -file input.fits output [options]}
%
where \code{[options]}
represents options to the IPP configuration system as well as some
specific to \code{psphot}.  In the case of a ``split'' camera format,
where separate chips or amps are stored in separate files, the
``input.fits'' may be a UNIX file glob which will expand to the set of
file; note that the glob must be protected with double quotes for
psphot. Alternatively, the input file list may be written to a text
file and the \code{-file input.fits} entry replace with \code{-list
  file.txt}.  The output entry is used as the root of a number of
possible output files selected by the user.

Standard IPP-wide command-line options (document elsewhere):
\begin{itemize}
\item \code{-site site.mdc}
\item \code{-camera camera-name}
\item \code{-recipe NAME VALUE}
\item \code{-D KEY VALUE}
\item \code{-Df KEY VALUE}
\item \code{-Db KEY VALUE}
\item \code{-Di KEY VALUE}
\end{itemize}

Here are other possible command-line options specific to psphot:
\begin{itemize}
\item -version : report version info and exit
\item -mask mask.fits 
\item -weight weight.fits
\item -modeltest X Y : run the object model on the object at the given coordinates
\begin{itemize}
\item -model
\item -fitmode
\item -fitset
\end{itemize}
\item -photcode : manually specify the photcode for this image
\item -region : limit the analysis to a specific image area
\item -chip : limit the analysis to the given list of chips
\item -psf : use the specified psf model (do not build a psf model)
\item -src : perform photometry on the positions given in the list
\end{itemize}

There are many configuration options in the \code{psphot.config}
recipe file.  These are defined in the \code{psphot} manual.

\subsubsection{psastro}

\subsubsection{ppStats}

\subsubsection{ppImage}

\subsubsection{ppMerge}

\subsubsection{ppNorm}

\subsection{Pipeline Infrastructure}

\subsubsection{ippdb / dbconfig}

The IPP uses a mysql database to track the data to be processed,
the current state of the processing steps, and summary information
from each of the analysis steps.  The database schema is defined by
the set of MDC files in \code{dbconfig}.  These files are used to
generate a set of C-level APIs to access the database tables.  

\subsubsection{ippTools}

\subsubsection{ippScripts}

\subsubsection{ippTasks and panTasks}

\subsubsection{ippMonitor}

\subsection{Nebulous}

\subsection{DVO}

\subsubsection{DVO Shell}

\subsubsection{Adding and Removing Data}

\subsubsubsection{addstar}

\subsubsubsection{delstar}

\subsubsubsection{getstar}

\subsubsection{Database Level Calibrations}

\subsubsubsection{relphot}

\subsubsubsection{uniphot}

\subsubsubsection{relastro}

\subsubsection{Other Tools}

\subsubsubsection{sky cell tools}

\subsection{Software Architecture}

\subsubsection{psLib}

\subsubsection{psModules}

\subsubsection{Perl Modules}

\section{Configuration}

Correct use of the IPP depends on a collection of configuration files.
These files define information about the processing environment, such
as the location of reference data or the name of computers available
for analysis or storage.  They also define the operating parameters
for the different programs, and choices about which analysis steps to
perform.

Configuration information for the image processing is provided on four
levels: the site-specific information, camera-specific details,
information related to different formats of data from the same camera,
and recipe configurations for the different analysis programs or
steps.  The configuration information is stored in files using the
``psMetadataConfig'' (MDC) file format, which is briefly described
below (\S\ref{sec:MDC}); for a more detailed description, see the
psLib SDRS (PSDC-430-007).

The configuration levels for the image processing components of the
IPP are:
\begin{itemize}
\item Options for the particular installation of the
  IPP: the {\it site};
\item Options specifying the details about the instrument: the {\it
camera};
\item Options specifying the format of the FITS file: the {\it
  format}; and
\item Options specifying the particular parameter choices that affect
  the details of an analysis: the {\it recipe}.
\end{itemize}
Note that these are arranged in an hierarchical order, with the site
configuration being the most general, and the recipe configurations
the most specific.  For example, not all sites will have to deal with
all cameras, and different cameras may require different recipes at
different times according to their particular quirks, analysis
experimentations, or their evolution.

We have provided examples of each of these configurations in the
\code{config} component of the IPP, which should be a useful guide for
setting up your own.  The Pan-STARRS IPP Configuration Guide
(PSDC-430-???) has all the detailed information.

\subsubsection{Setting up configuration files}

When the IPP software is installed, it generates a copy of the
top-level {\it site} configuration file:
\code{PREFIX/share/ippconfig/ipprc.config}. Programs look for this
information in the user's home directory at \code{~/.ipprc}.  A new
user may want to link \code{~/.ipprc} to the installed copy
\code{PREFIX/share/ippconfig/ipprc.config}, in which case new
installations will be immediately available.  Alternatively, the user
may want to copy the file in order to make their own modifications, if
for example the user needs to modify the recipes for a specific
camera.

After the {\it site} configuration file is read by IPP programs, they
then attempt to read the {\it camera} configuration files defined in
the {\it site} file by the list of cameras.  These entries refer to
files for each camera.  The IPP configuration system searches for these
files (and others referred there in) in the PATH defined at the top of
the {\it site} file.  This value mimics the UNIX PATH variable, and
consists of a colon-separated list of locations.  The first entry
found from this list is used by the configuration system, allowing a
user to override, for example, the globally installed camera
configuration for some camera.  

\subsubsection{Overview of MDC format}
\label{sec:MDC}

psLib defines a \code{psMetadata} structure which can carry labeled
data of arbitrary types.  Originally designed for carrying the data in
FITS headers, the \code{psMetadata} have proved so generally useful
that we use them for our configurations (and a multitude of other
uses!).  We have designed a human-readable text-based format --- the
``MetaData Configuration'' (MDC) format --- which we use for this end.

Each simple entry in an MDC file must contain the name, type and
value.  Each of these is on a single line, separated by whitespace,
and in that order.  Comments may be placed at the end of the line (or
on a blank line), after a hash mark (\code{#}).  Whitespace at the
beginning and end of strings (either the name, value or comment) are
stripped.

The simple types follow the psLib types.  Integers are specified by a
letter indicating if the integer is signed (\code{S}) or unsigned
(\code{U}) and a number indicating the dynamic range in bits (8, 16,
32 or 64); e.g., \code{U8} is commonly used for bit mask values,
\code{S32} is commonly used for ordinary integer values.  Floating
point values are specified by the letter \code{F} and a number
indicating the precision in bits (32 or 64): \code{F32} (single
precision) or \code{F64} (double precision).  Strings are specified by
\code{STR}.  Times may be specified with the following types:
\code{UTC,UT1,TAI,TT}; values for the time are expected to be in
ISO8601 format (\code{YYYY-MM-DDTHH:MM:SS.sZ}).

Names are traditionally all-caps, though there is no reason why they
must be; the names are case-sensitive.  A name may not be repeated
unless it has previously been declared to be of type \code{MULTI} (no
value should be provided with this declaration):
\begin{verbatim}
COMMENT    MULTI
COMMENT    STR    Having more than one COMMENT like this
COMMENT    STR    is permitted because of the MULTI.
\end{verbatim}

A hierarchy can be made using the \code{METADATA} type, which signals
a new level:
\begin{verbatim}
JANITOR    METADATA
    NAME          STR    John Doe
    PAY           F32    1234.56
    ECCENTRICITY  STR    9.87
END
\end{verbatim}
Note that a \code{METADATA} block is closed by an \code{END}.  No
identing need be done within a \code{METADATA} block, but it is useful
to be able to see the levels at a glance (just like in a C program).
\code{METADATA} blocks may be nested within \code{METADATA} blocks,
probably down as far as you have the patience to try.  Note that
\code{MULTI} declarations only apply to the current level --- there is
no inheritance.

The above format can be long if there are many \code{METADATA}s with
similar contents.  For this reason, we provide the \code{TYPE}
declaration, which generates a \code{METADATA} with the contents each
of type \code{STR}:
\begin{verbatim}
TYPE          EMPLOYEE    NAME       PAY        ECCENTRICITY
\end{verbatim}
Now, the type \code{EMPLOYEE} may be used, with string values (NB: no
spaces allowed!) to specify multiple entries:
\begin{verbatim}
JANITOR       EMPLOYEE    JohnDoe    1234.56    9.87
PROGRAMMER    EMPLOYEE    FooBar     2345.67    1.00
\end{verbatim}
This is the same as the much longer block:
\begin{verbatim}
JANITOR       METADATA
    NAME            STR     JohnDoe
    PAY             STR     1234.56
    ECCENTRICITY    STR     9.87
END
PROGRAMMER    METADATA
    NAME            STR     FooBar
    PAY             STR     2345.67
    ECCENTRICITY    STR     1.00
END
\end{verbatim}
Like the \code{MULTI}, \code{TYPE} declarations only apply to the
current level.

\subsection{Filenames : UNIX Paths, Abstract Paths, Nebulous}

The IPP programs recognize three types of file names: a standard UNIX
path, an abstract path with a top-level location defined by the IPP
configuration system, and a Nebulous path, in which the file location
is completely abstracted by the Nebulous file management system:
\begin{itemize}
\item \code{file:///path/to/file.ext} : a UNIX path, always relative
  to the root of the file system.  It is valid to drop the
  \code{file:/} component of the name.  The IPP accepts an arbitrary
  number of slashes after \code{file:}.
\item \code{path://PATH/file.ext} : An abstract path. The value of
  PATH is iterpolated from the corresponding entry in the
  \code{DATAPATH} table given in the site configuration file.
\item \code{neb://full/nebulous/name} : A nebulous filename.  The IPP
  programs query Nebulous for the true path of (one copy of) the file.
\end{itemize}
If you wish to expand a file name in any of these forms to a standard
UNIX path, use the program \code{ipp_datapath.pl}.  

\section{Installation}

There presently exists a few different systems for automatically building the
the majority of IPP software.  The \code{psconfig} method of building is
targeted towards general end users.  Where as the current batch of
\code{jhbuild} configuration files require the user to have access to the IPP's
CVS tree, making it more of interest to active developers.

\subsection{psconfig}

The psconfig system allows the user to build and install the IPP
software suite into a location which is flexibly defined by the user.
The tools here also set up the user's environment variables
(\code{PATH}, \code{PERL5LIB}, \code{LIBRARY_PATH}, etc) to make use
of the installed software.  With the psconfig tools, it is easy to
switch between different installed versions or to recompile subsets of
the IPP tree.  It is also possible for a user to install the complete
IPP system without access to the root password.

\subsubsection{Preparation}

\paragraph{Choose the build and install directories}  You need to
decide where to install the IPP software, and also where to build the
source code.  By default, the psconfig system will install the IPP
software in \code{~/psconfig}.  The IPP code will be 

\paragraph{Unpack the IPP tarball.}  The IPP source code is
distributed as a single tarball for all of the IPP software
(ipp-M.NN.tgz, where M.NN is the version number, currently 2.1).  Use
the command \code{tar xvzf ipp-M.NN.tgz} or \code{gzcat ipp-M.NN.tgz |
tar xvz} on older platforms.  The tarball will unpack into a directory
named ipp-M.NN.

\paragraph{Unpack the external Perl modules.} If needed, the complete
collection of Perl modules used by the IPP is also distributed from
the IPP web site as a tarball, extperl.tgz.  Use the command \code{tar
xvzf extperl.tgz}, which will unpack the tarball into a directory
called \code{extperl}.  This must be at the same location as the top
level directory.  The program \code{pscheckperl} below will identify
any external Perl modules which are missing from your system.

\paragraph{Download the external C libraries.}  If needed, the
external libraries needed by the IPP are available from the IPP web
pages. Download these as needed to a directory called \code{extlibs},
again parallel to the \code{ipp-M.NN} directory.  The program
\code{pschecklibs} below will identify any external C libraries which
are missing from your system.

\paragraph{Set up your account to use the psconfig system.}  The
psconfig system provides a script which sets up the necessary UNIX
environment variables and aliases, enabling the compiler and the
programs to find your installed libraries and Perl modules.  The
details depend on your UNIX shell:

\subparagraph{csh users}

To use the psconfig system, place the following line in your ~/.cshrc
file:

\code{alias psconfig "source ../ipp-M.NN/psconfig/psconfig.csh"}

where ../ipp-M.NN is the location of the extracted ipp tarball.  If
you prefer, you may copy the file psconfig.csh to another location.
If, for example, you would like to remove the build directories after
building the IPP, you will need a persistent copy of psconfig.csh.

By default, the psconfig system places the installed IPP programs and
configuration files in a directory in your home directory:
\code{~/psconfig}.  To use a different location, place the following
line in \code{~/.psconfigrc} (otherwise not needed):

\code{set PSCONFDIR = INSTALL_PATH}

where \code{INSTALL_PATH} is the top-level directory for all installed files.

\subparagraph{bash users}

Add the following line to your \code{~/.bashrc} file:

\code{alias psconfig="source PATH/psconfig.bash"}

It is also necessary to edit the file psconfig.bash to set the
\code{PSCONFIG_DIR} variable to this directory as well.

\tbd{set this up for better discovery}

\paragraph {Set the installation version}

The IPP is a large and complex software system.  A major goal of the
IPP build system is to be user-friendly for those end users which do
not have root access on their machines.  Using the IPP build tools, it
is possible to install the complete system as a non-priviledged user.
The build system also makes it possible to maintain multiple
simultaneous installations with different versions of the
software. This latter feature is particularly important for developers
who need to be able to make tests and comparisons of different
versions.

With \code{psconfig}, you may have multiple, parallel installations of
the IPP.  These may be different versions of the software, or
installations for different computer architectures, or installations
with different compilation options.  For example, you may want an
installation with tracing and debug information turned on and a second
with all optimizations turned on.  With the psconfig build system,
each installation of the IPP software is placed in a directory
identified by an installation name and the computer architecture.  The
installation name is up to the person who compiles the IPP software,
and is an arbtrary word or string.  For example, you may choose the
install the optimized version under ipp-2.1-opt and the debug version
under ipp-2.1-debug.  The psconfig system will create a directory in
\code{~/psconfig} (or where ever \code{PSCONDIR} is set) called
ipp-2.1-opt.linux (if building on a linux 32bit system), or something
equivalent.

Before running or compiling the IPP, it is necessary to use
\code{psconfig} to set the installation name:
%
\code{psconfig (name)}
%
This command sets aliases and environment variables for the current
shell to point at the named IPP installation, for the current 
hardware.  For example:
%
\code{psconfig default}
%
will set the \code{PATH} to include
\code{~/psconfig/default.linux/bin} on a 32-bit linux system, and the
other paths to point at the corresponding installation directories.

Users who wish to automatically have access to an IPP installation
should add the psconfig line to their \code{~/.cshrc} or
\code{~/.bashrc} files.  

\subsubsection{Check External Dependencies}

The IPP build system is run from the directory
\code{../ipp-M.NN/psconfig}.  Within this directory are the build
scripts and scripts to prove the build environment.  Before building
the IPP suite, you should first check for the needed C libraries and
Perl scripts.  Start by \code{cd}-ing into \code{ipp-M.NN/psconfig}.  

\subsubsubsection{External C libraries}

The program \code{pschecklibs} in the \code{ipp-M.NN/psconfig}
directory will check for required system libraries and headers:
%
\code{pschecklibs}
%
It examines the system libraries, libraries defined by
\code{LIBRARY_PATH}, and the installation library defined by psconfig.
Any missing dependencies will be listed.  Tarballs for these libraries
may be found on the Pan-STARRS web site at:
%
\code{http://pan-starrs.ifa.hawaii.edu/project/IPP/software/ext}
%
These should be installed so they will be available in the user's
path.  This can be done with the \code{psconfig} tools by using
\code{psconfigure} to replace the standard \code{configure} command.
The \code{psconfigure} command applies the needed \code{--prefix}
options to place the libraries within the IPP installation tree.
There is also an equivalent for \code{autogen.sh}, \code{psautogen},
if needed.

\subsubsubsection{External Perl Modules}

The program \code{pscheckperl} in the \code{ipp-M.NN/psconfig}
directory will check for required Perl modules, and can be used to
install them in the appropriate user location in the psconfig system.
The command defaults to the latest perl installation table in the
\code{tagsets} directory.  If you have CVS access and choose to check
out an older IPP distribution, it is necessary to supply the
distribution name to \code{pscheckperl}.  Otherwise, run it without arguments:
%
\code{pscheckperl}
%
This will test for the perl modules specified for the latest ipp
release.  If any modules are missing, you should install the perl
module tarball (see above), or they can be download from the
Pan-STARRS web site:
%
\code{http://pan-starrs.ifa.hawaii.edu/project/IPP/software/extperl}

The tarballs should be placed in a directory \code{extperl} parallel
to the ipp directory (two levels up from this directory).  If the
tarballs are in the correct location, they can be built by supplying
the \code{-build} flag to \code{pscheckperl}:

\code{pscheckperl -build}

\subsubsection{Building IPP}

To build the full IPP tree using the psconfig system, run
\code{psbuild} in the \code{ipp-M.NN/psconfig} directory.  By default,
\code{psbuild} will use the latest IPP distribution table, in the
directory \code{tagsets}, and build the IPP components according to
the rules in that distribution file.  The distribution files specify
which versions (which CVS tags) of the different programs are to be
built, and in which order, for a given distribution of the IPP.  If
you have access to the IPP CVS tree and check out an old IPP
distribution, it is possible to specify the file for that
distribution, and build the software as it appeared for that older
distribution.  It is also possible to use the tools in this directory
to check out the older code and to build tarballs for the older
distributions.  However, for the typical end user building the IPP
from a distributed tarball, it is only necessary to run \code{psbuild}
without additional arguments:
%
\code{psbuild ipp-1.2}
%

There are a number of command-line options to \code{psbuild} which
control how the software is built or which components of the IPP to
build:
\begin{verbatim}
 -version (version) : specify alternate psconfig installation version
 -clean             : clean the source directories before building
 -rebuild           : run 'autogen' (C code)
 -optimize          : set flags for optimized code
 -only (module)     : only build the specified module
 -start (module)    : begin build at specified module
 -stop (module)     : stop build after specified module
\end{verbatim}

Summary of psconfig operations:
\begin{verbatim}
psdist -tag        : tag CVS tree
psdist -dist       : build tarball from tagged tree
psdist -dist -head : build tarball from head
psbuild            : build and install software in tree
pschecklibs        : check for needed external software
pscheckperl        : check for needed perl modules
pscheckperl -build : build and install external modules
\end{verbatim}

\subsection{jhbuild}

JH uses \code{jhbuild} even though the 'jh' in \code{jhbuild} doesn't
really refer to him.

\subsubsubsection{What is it?}

According to the introduction on the \code{jhbuild} website:

\begin{quote}
\code{jhbuild} is a program that can be used to pull a number of
modules from CVS and build them in the correct order. Unlike some
build scripts, \code{jhbuild} lets you specify what modules you want
built and it will then go and build those modules plus dependencies.

Although \code{jhbuild} was originally developed to build
\code{[WWW]Gnome}, it is now able to build a number of the modules in
\code{freedesktop.org} CVS. Extending it to handle new modules is
usually trivial (assuming the build infrastructure matches the other
modules it handles).
\end{quote}

In additional to retrieving source code from various SCM's (CVS, SVN,
arch, etc.), jhbuild has the ability to download tarballs via HTTP or
FTP.

\code{jhbuild} has been adopted as an official \code{freedesktop.org}
project. You can find more information on the project's homepage
(\code{http://www.freedesktop.org/Software/jhbuild}). Bugs can be
filed in the Gnome Bugzilla (\code{http://bugzilla.gnome.org}).

\subsubsubsection{Where to get it}

It was necessary to slightly modify \code{jhbuild} for use with IPP
software. Therefore, you must checkout the \code{jhbuild} module from
the Pan-STARRS CVS tree. Please see the Pan-STARRS CVS Guide for help
on setting up and using CVS. \code{jhbuild} will need to be able to
find it's own source tree even after installation so you should choose
a checkout path that can be permanent. Something along the lines of
\code{$HOME/src} is recommended.

\begin{verbatim}
cd
mkdir -p src
cd src
cvs co jhbuild
\end{verbatim}

After running CVS you should see something like this:

\begin{verbatim}
$ cvs co jhbuild
cvs checkout: Updating jhbuild
U jhbuild/.cvsignore
U jhbuild/COPYING
U jhbuild/ChangeLog
U jhbuild/HACKING
U jhbuild/Makefile
U jhbuild/README
U jhbuild/install-check.c
.
.
\end{verbatim}

\subsubsubsection{Installing jhbuild into your home directory}

\code{jhbuild} should be installed locally under your home
directory. This will require that you modify the \code{PATH}
environment variable so that you can run jhbuild after it has been
installed.

\begin{verbatim}
cd jhbuild
make
make install
\end{verbatim}

Which should look something like this:

\begin{verbatim}
$ make
gcc -Wall -O2 -o install-check install-check.c
Run "make install" to install.
$ make install
Creating /home/moanui/jhoblitt/bin/jhbuild
Creating /home/moanui/jhoblitt/.gnome2/vfolders/applications/jhbuild.desktop
install -m755 install-check /home/moanui/jhoblitt/bin/install-check
install -m755 config.guess /home/moanui/jhoblitt/bin/config.guess
\end{verbatim}

That will install the \code{jhbuild} executable under
\code{$HOME/bin}. You are responsible for including this path in your
\code{PATH} environment variable. It is highly recommended that you
add this to your \code{.bashrc} or equivalent shell login script.

For the \code{bash} shell, place this line in your \code{.bashrc}:
\begin{verbatim}
export PATH=${HOME}/bin:${PATH}
\end{verbatim}
For the \code{tcsh} shell, place this line in your \code{.tschrc}:
\begin{verbatim}
setenv PATH ${HOME}/bin:${PATH}
\end{verbatim}

\subsubsubsection{Configuring jhbuild}

\code{jhbuild} is configured via an rc file that lives at
\code{${HOME}/.jhbuildrc}. Please note that this rc file is executed
as Python code; be careful!

Example \code{.jhbuildrc}, suitable for cut and paste:

\begin{verbatim}
# what profile to build?
moduleset = 'http://pan-starrs.ifa.hawaii.edu/project/IPP/software/modulesets/ipp12.modules'

# modules to build by default
modules = [ 'pslib', 'psmodules' ]

# where should working copies go?
jhroot = os.environ['HOME'] + '/jhroot'

# where should tarballs be kept?
tarballdir = jhroot + '/src'

# in what prefix should things be installed? (must be writable)
target = os.popen('config.guess').read().rstrip()
prefix = jhroot + '/' + target
checkoutroot = prefix + '/build'

# extra arguments to pass to the autogen.sh script?
autogenargs = '--enable-maintainer-mode --disable-static'

# use an alternative install program that preserves the
# mtime on header files if they haven't changed.  Speeds
# up rebuilds.
os.environ['INSTALL'] = os.environ['HOME'] + '/bin/install-check'

# don't try to use /usr/ucb/cc on Solaris
import sys
if sys.platform == 'sunos5':
    os.environ['CC'] = 'gcc'
\end{verbatim}

\subsubsubsection{Running jhbuild}

\code{Jhbuild} can be executed as \code{jhbuild build
[modulename]}. Just \code{jhbuild} will build the packages specified
in the \code{modules} variable from your rc file.
\begin{verbatim}
jhbuild
\end{verbatim}
or
\begin{verbatim}
jhbuild build pslib
\end{verbatim}

Run \code{jhbuild} list to get a list of the packages \code{jhbuild}
knows how to build.

\begin{verbatim}
$ jhbuild list
cfitsio
gsl
fftw
libxml2
mysql
pslib
psmodules
\end{verbatim}

\code{jhbuild} supports many other commands. Please see \code{jhbuild
--help} for a complete list of options.

\begin{verbatim}
$ jhbuild --help
usage: jhbuild [ -f config ] command [ options ... ]
Build a set of CVS modules (such as GNOME).

Global options:
  -f, --file=CONFIG            use a non default configuration file
  -m, --moduleset=URI          use a non default module set
      --no-interact            do not prompt for input

Commands:
  gui                          build targets from a gui app
  update                       update from cvs
  updateone modules            update a fixed set of modules
  build [ opts... ] [modules]  update and compile (the default)
  buildone [ opts... ] modules build a single module
  tinderbox [ opts... ]        build non-interactively with logging
  run program [ args... ]      run a command in the build environment
  shell                        start a shell in the build environment
  sanitycheck                  check that required support tools exists
  bootstrap                    build required support tools
  list [ opts ... ] [modules]  list what modules would be built
  dot [ modules ]              output a dot file of dependencies suitable
                               for processing with graphviz
  info modules...              prints information about modules

Options valid for the build, buildone, tinderbox and update commands:
  -s, --skip=MODULES           treat the given modules as up to date
  -t, --start-at=MODULE        start building at the given module
  -D date_spec                 set a sticky date when checking out modules

Options valid for the build, buildone and tinderbox commands:
  -a, --autogen                always run autogen.sh
  -c, --clean                  run make clean before make
  -n, --no-network             skip cvs update

Options valid for the tinderbox command:
  -o, --output=DIR             directory to save build logs in

Options valid for the list command:
  -r, --show-revision          show which revision will be built
\end{verbatim}

\subsubsubsection{Dependancies}

\code{jhbuild} has a fairly minimal set of dependencies --- far less
than what may be required to actually compile and install any
packages. However, if your system can meet the base requirements,
\code{jhbuild} should be able to bootstrap your build environment.
\begin{itemize}
\item A working C compiler (eg. \code{gcc})
\item A working \code{libc} (eg. \code{glibc})
\item Perl 5 with the \code{XML::Parser} module (needed by
  \code{libtool})
\item Python 2.?
\item Either \code{wget} or \code{curl}
\item GNU \code{M4} 1.4
\item \code{tar}
\item \code{gzip}
\item \code{bzip2}
\end{itemize}

\subsubsubsection{Bootstrapping}

\code{jhbuild} has a limited ability to install some of the necessary
tools for maintaining software that configure its build environment
with the GNU autotools.

This step is probably required on OSX and Solaris. Your mileage will
vary per Linux distribution but you can probably skip this step if
your distribution is around RedHat 9 vintage or newer.

\begin{verbatim}
jhbuild bootstrap
\end{verbatim}

\code{jhbuild} will then will begin to build a series of packages.

\subsubsubsection{Using the jhbuild enviroment}

As you've already seen, \code{jhbuild} is capable of setting up an
independent build environment under the (configurable) directory of
your choice. In order to link non-\code{jhbuild} management software
against this build environment a number of your shell's environment
variable have to be modified. \code{jhbuild} is capable of doing this
for you. The syntax for this is \code{jhbuild shell}, which as the
syntax implies, spawns a new shell with the proper environment
variables.

This example demonstrates \code{jhbuild} setting up the dynamic
linkers default search path for you.

\begin{verbatim}
$ echo $LD_LIBRARY_PATH

$ jhbuild shell
$ echo $LD_LIBRARY_PATH
/home/moanui/jhoblitt/jhroot/i686-pc-linux-gnu/lib
\end{verbatim}

%$ --- Emacs needs this to balance out the dollar signs

A fair number of other variables are also adjusted for you. Enough so
that most (all?) \code{autoconf} configured software will be able to
find it's dependencies.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Aliases}

PAP puts the following in his \code{~/.tcshrc}:
\begin{verbatim}
setenv SWDIR $HOME/local/`$HOME/bin/config.guess`/
if (! -d $SWDIR) mkdir --parents $SWDIR
alias ./autogen.sh './autogen.sh --prefix=$SWDIR CFLAGS="-I$SWDIR/include/ -g" LDFLAGS=-L$SWDIR/lib/'
alias   autogen.sh './autogen.sh --prefix=$SWDIR CFLAGS="-I$SWDIR/include/ -g" LDFLAGS=-L$SWDIR/lib/'
alias ./configure  './configure  --prefix=$SWDIR CFLAGS="-I$SWDIR/include/ -g" LDFLAGS=-L$SWDIR/lib/'
alias   configure  './configure  --prefix=$SWDIR CFLAGS="-I$SWDIR/include/ -g" LDFLAGS=-L$SWDIR/lib/'
setenv PATH ${PATH}:$SWDIR/bin/
setenv LD_LIBRARY_PATH $SWDIR/lib/:$SWDIR/lib/mysql:$LD_LIBRARY_PATH
setenv MANPATH $SWDIR/man:$MANPATH
setenv PKG_CONFIG_PATH $SWDIR/lib/pkgconfig/:$PKG_CONFIG_PATH
\end{verbatim}

%$ --- Emacs needs this to balance out the dollar signs

Here, \code{config.guess} is the common GNU script for guessing the
build system triplet (e.g., \code{i686-pc-linux-gnu}).

There are a couple of notes:
\begin{itemize}
\item To compile a binary, simply do \code{./configure}, then
  \code{make && make install}.
\item \code{Ohana} doesn't like this setup, so you need to build it
  with: \code{\./configure --prefix=$SWDIR}
\item Perl modules can be installed: \code{./Build install
  --prefix=$SWDIR}.
\end{itemize}

\subsection{Manual Perl Module Installation}

Here we describe setting up the Perl dependencies followed by the
IPP components.

\subsubsection{Dependencies}
\label{sec:installing-perl-dependencies}

If you have access to the \code{root} account, installation as
\code{root} is much easier.  If not, you will have to go through
the more flaky installation as an unprivileged user.

\subsubsubsection{Installation as root}

Many of the Perl dependencies are available from the Comprehensive
Perl Archive Network (CPAN) at www.cpan.org.  If you have root access
on your target machines, they can be very simply retrieved, built and
installed (replacing \code{MODULE_NAME} for each module):

\begin{verbatim}
> su -
Password:
> cpan
[...]
cpan> install MODULE_NAME
[...]
cpan> quit
\end{verbatim}

Follow the prompts.  It's usually safe to accept the default (simply
hit enter) in response to most questions.

If you get into trouble, try: \code{force install MODULE_NAME}.

You can also try to use the \code{Bundle::PS} as described below if
you're feeling adventurous.

\subsubsubsection{Installation as unprivileged user}

To install modules from CPAN with the \code{CPAN.pm} interface, you
need to setup a CPAN configuration file in your home directory.  Then
\code{CPAN.pm} can walk you through setting up the most important
configuration values.  Unfortunately, there is some variation in the
behavior of the various versions of \code{CPAN.pm} that have shipped
with Perl.  Some (most) of these variants will not correctly create a
configuration files that allows a non-\code{root} user to install
modules outside of "system" paths.  In order to make sure that you get
a "correct" CPAN configuration file you need to ``prime'' it with a
few values.

First you need to create the directory in which the CPAN configuration file will live.
\begin{verbatim}
> mkdir -p .cpan/CPAN/
\end{verbatim}

Then we need to create a partial configuration file.  Note that this example
assumes that you want to install your perl modules under
\code{$HOME/local/lib/perl5}.

\begin{verbatim}
> echo "\$CPAN::Config = {" >> .cpan/CPAN/MyConfig.pm
> echo "  makepl_arg => q[PREFIX=$HOME/local/]," >> .cpan/CPAN/MyConfig.pm
> echo "  mbuildpl_arg => q[--install_base $HOME/local/]," >> .cpan/CPAN/MyConfig.pm
> echo "};" >> .cpan/CPAN/MyConfig.pm
> echo "1;" >> .cpan/CPAN/MyConfig.pm
> echo "__END__" >> .cpan/CPAN/MyConfig.pm
\end{verbatim}

%$ --- Emacs needs this to balance the previous dollar sign

Now you need to invoke \code{CPAN.pm} so it can walk you through
configuring the rest of the required values.  This is an example of
one possible configuration with \code{CPAN.pm} version 1.8802.
\textbf{Your version of CPAN.pm may present you with different
prompts.}  Use your common sense.  If in doubt, it is generally safe
to simply hit enter (and accept the default).

\begin{verbatim}
> perl -MCPAN -e shell
CPAN: File::HomeDir loaded ok
Sorry, we have to rerun the configuration dialog for CPAN.pm due to
the following indispensable but missing parameters:

build_cache, build_dir, cache_metadata, cpan_home, ftp_proxy, http_proxy,
index_expire, inhibit_startup_message, keep_source_where, make_arg,
make_install_arg, mbuild_arg, mbuild_install_arg, mbuild_install_build_command,
no_proxy, prerequisites_policy, scan_cache, urllist


The following questions are intended to help you with the
configuration. The CPAN module needs a directory of its own to cache
important index files and maybe keep a temporary mirror of CPAN files.
This may be a site-wide directory or a personal directory.



I see you already have a  directory
    /home/moanui/jhoblitt/.cpan
Shall we use it as the general CPAN build and cache directory?

CPAN build and cache directory? [/home/moanui/jhoblitt/.cpan]


Unless you are accessing the CPAN via the filesystem directly CPAN.pm
needs to keep the source files it downloads somewhere. Please supply a
directory where the downloaded files are to be kept. [/home/moanui/jhoblitt/.cpan/sources]
Directory where the build process takes place? [/home/moanui/jhoblitt/.cpan/build]


How big should the disk cache be for keeping the build directories
with all the intermediate files?

Cache size for build directory (in MB)? [100]


The CPAN indexes are usually rebuilt once or twice per hour, but the
typical CPAN mirror mirrors only once or twice per day. Depending on
the quality of your mirror and your desire to be on the bleeding edge,
you may want to set the following value to more or less than one day
(which is the default). It determines after how many days CPAN.pm
downloads new indexes.

Let the index expire after how many days? [1]


By default, each time the CPAN module is started, cache scanning is
performed to keep the cache size in sync. To prevent this, answer
'never'.

Perform cache scanning (atstart or never)? [atstart]


To considerably speed up the initial CPAN shell startup, it is
possible to use Storable to create a cache of metadata. If Storable
is not available, the normal index mechanism will be used.

Cache metadata (yes/no)? [yes]


The CPAN module can detect when a module which you are trying to build
depends on prerequisites. If this happens, it can build the
prerequisites for you automatically ('follow'), ask you for
confirmation ('ask'), or just ignore them ('ignore'). Please set your
policy to one of the three values.

Policy on building prerequisites (follow, ask or ignore)? [ask] follow


Every Makefile.PL is run by perl in a separate process. Likewise we
run 'make' and 'make install' in separate processes. If you have
any parameters (e.g. PREFIX, LIB, UNINST or the like) you want to
pass to the calls, please specify them here.

If you don't understand this question, just press ENTER.
Parameters for the 'make' command?
Typical frequently used setting:

    -j3              # dual processor system

Your choice:  []
Parameters for the 'make install' command?
Typical frequently used setting:

    UNINST=1         # to always uninstall potentially conflicting files

Your choice:  [] UNINST=1


The next questions deal with Module::Build support.

A Build.PL is run by perl in a separate process. Likewise we run
'./Build' and './Build install' in separate processes. If you have any
parameters you want to pass to the calls, please specify them here.

Parameters for the './Build' command?
Setting might be:

    --extra_linker_flags -L/usr/foo/lib  # non-standard library location

Your choice:  []
Do you want to use a different command for './Build install'?
Sudo users will probably prefer:

    su root -c ./Build
or
    sudo ./Build
or
    /path1/to/sudo -u admin_account ./Build

or some such. Your choice:  [./Build]
Parameters for the './Build install' command?
Typical frequently used setting:

    --uninst 1                           # uninstall conflicting files

Your choice:  [] --uninst 1


If you're accessing the net via proxies, you can specify them in the
CPAN configuration or via environment variables. The variable in
the $CPAN::Config takes precedence.

Your ftp_proxy? []
Your http_proxy? []
Your no_proxy? []
You have no /home/moanui/jhoblitt/.cpan/sources/MIRRORED.BY
  I'm trying to fetch one
CPAN: LWP::UserAgent loaded ok
Fetching with LWP:
  http://www.perl.org/CPAN/MIRRORED.BY


Now we need to know where your favorite CPAN sites are located. Push
a few sites onto the array (just in case the first on the array won't
work). If you are mirroring CPAN to your local workstation, specify a
file: URL.

First, pick a nearby continent and country by typing in the number(s)
in front of the item(s) you want to select. You can pick several of
each, separated by spaces. Then, you will be presented with a list of
URLs of CPAN mirrors in the countries you selected, along with
previously selected URLs. Select some of those URLs, or just keep the
old list. Finally, you will be prompted for any extra URLs -- file:,
ftp:, or http: -- that host a CPAN mirror.

(1) Africa
(2) Asia
(3) Central America
(4) Europe
(5) North America
(6) Oceania
(7) South America
Select your continent (or several nearby continents) [] 5

(1) Bahamas
(2) Canada
(3) Mexico
(4) United States
Select your country (or several nearby countries) [] 4

(1) ftp://carroll.cac.psu.edu/pub/CPAN/
(2) ftp://cpan-du.viaverio.com/pub/CPAN/
(3) ftp://cpan-sj.viaverio.com/pub/CPAN/
(4) ftp://cpan.calvin.edu/pub/CPAN
(5) ftp://cpan.cs.utah.edu/pub/CPAN/
(6) ftp://cpan.cse.msu.edu/
(7) ftp://cpan.erlbaum.net/CPAN/
(8) ftp://cpan.glines.org/pub/CPAN/
(9) ftp://cpan.hostrack.net/pub/CPAN
(10) ftp://cpan.llarian.net/pub/CPAN/
(11) ftp://cpan.mirrors.redwire.net/pub/CPAN/
(12) ftp://cpan.mirrors.tds.net/pub/CPAN
(13) ftp://cpan.netnitco.net/pub/mirrors/CPAN/
(14) ftp://cpan.pair.com/pub/CPAN/
(15) ftp://cpan.teleglobe.net/pub/CPAN
(16) ftp://cpan.uchicago.edu/pub/CPAN/
40 more items, hit RETURN to show them
Select as many URLs as you like (by number),
put them on one line, separated by blanks, hyphenated ranges allowed
 e.g. '1 4 5' or '7 1-4 8' [] 14 11 12

Enter another URL or RETURN to quit: []
New set of picks:
  ftp://cpan.pair.com/pub/CPAN/
  ftp://cpan.mirrors.redwire.net/pub/CPAN/
  ftp://cpan.mirrors.tds.net/pub/CPAN


Please remember to call 'o conf commit' to make the config permanent!


cpan shell -- CPAN exploration and modules installation (v1.8802)
ReadLine support enabled

 cpan[1]> o conf commit
commit: wrote '/home/moanui/jhoblitt/.cpan/CPAN/MyConfig.pm'
\end{verbatim}

%$ --- Emacs needs this to balance the previous dollar sign

Now we need to install the module that installs the other modules.

\begin{verbatim}
cpan> install Module::Build
\end{verbatim}

Exit out of cpan:

\begin{verbatim}
cpan> exit
\end{verbatim}

In order to use of the installed modules, we need to setup an
environment variable called \code{PERL5LIB} so that 'perl' can find
them.  To do this, we need to know where under 'perl5' our modules
were actually installed.  This will set variable with the version of
Perl that you are using.  The easiest way to do this is just just look
in the root of the path where we did the install.

\begin{verbatim}
> ls local/lib/perl5/
5.8.8  site_perl
\end{verbatim}

That means we're using perl 5.8.8 and \code{PERL5LIB} needs to be
setup as following:
\begin{verbatim}
export PERL5LIB=$HOME/local/lib/perl5/5.8.8:$HOME/local/lib/perl5/site_perl/5.8.8
\end{verbatim}

Now we should install the basic compliment of helper modules that
\code{CPAN.pm} needs to function fully.  Go back into CPAN (\code{perl
-MCPAN -e shell}) and:

\begin{verbatim}
cpan> install Bundle::CPAN
\end{verbatim}

You can quit out of the CPAN shell at this point with the `exit`
command or do the following few steps in another shell We're ready to
install the full set Perl module dependencies for IPP software.  In
order to make this process a bit easier on the end user a "Bundle"
module has been created.  In order to use it you need to create a
directory (if it doesn't already exist) called Bundle under your .cpan
directory.

\begin{verbatim}
> mkdir -p .cpan/Bundle
\end{verbatim}

The file \code{PS.pm} should copied into this directory:

\begin{verbatim}
cp /path/to/PS.pm .cpan/Bundle/
\end{verbatim}

Back in the CPAN shell, 'force' the install of the PS Bundle.  The
'force' keyword instructs the shell to ignore any tests failures.
This is necessary as some of the modules 'DBD::mysql'/etc. require a
properly working database setup in order for the tests to pass.  You
will most likely be prompted for input by several the modules.  It is
safe to answer with a carriage return to all questions.  If it insists
on a path to \code{httpd}, hit \code{CTRL-C} and it will go on to the
next step.

\begin{verbatim}
cpan> force install Bundle:PS
\end{verbatim}

For further instructions on installing Perl modules from CPAN ''by
hand', see:
\begin{verbatim}
http://www.cs.ucsc.edu/~you/notes/perl-module-install.html
\end{verbatim}

After the dependencies (\S\ref{sec:dependencies}) have been satisfied,
the IPP packages should be installed in the following order:
\begin{itemize}
\item \code{Ohana}
\item \code{psLib}
\item \code{psModules}
\item \code{psphot}
\item \code{psastro}
\item \code{ppStats}
\item \code{ppImage}
\item \code{ppMerge}
\item \code{ppNorm}
\item \code{pois}
\item \code{stac}
\item \code{pswarp}
\item \code{ppStac}
\item \code{ippdb}
\item \code{ippTools}
\item \code{PS-IPP-Metadata-Config}
\item \code{PS-IPP-Metadata}
\item \code{ippScripts}
\item \code{ippTasks}
\item \code{config}
\end{itemize}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsubsection{Modules and scripts}

Pan-STARRS Perl modules and scripts are installed by executing the
following in the directory containing the source:
\begin{verbatim}
> perl Build.PL
> ./Build
> ./Build install --prefix=/path/to/install/
\end{verbatim}

Make sure you set your \code{PERL5LIB} environment variable to the
installation path (here, \code{/path/to/install/}).

\section{System Requirements}

From the start, we have endeavoured to have the IPP to be flexible so
that it can be deployed on a range of systems.  With that in mind,
however, we have specifically targeted Linux systems, stressing POSIX
compliance.

\subsection{Hardware}

We have developed and tested on machines with different hardware
architectures.  Reports of successes and failures in deploying the IPP
on an untested hardware architecture are welcome.

We have had thorough testing on x86 and amd64 architectures, with
Gentoo and Red Hat Linux.

We expect that the binaries will compile under MacOS (if there are
problems please let us know).  We have had trouble compiling psLib
under Solaris.

\subsection{Dependencies}
\label{sec:dependencies}

We have dependencies on two levels: library dependencies (for the
binaries) and Perl module dependencies.

\subsubsubsection{Libraries}

\begin{figure}
\psfig{file=ipp.ps,height=8in,angle=0}
\caption{Dependency chart for the IPP binaries}
\label{fig:ipp-dependencies}
\end{figure}

The following external libraries are required to build \code{psLib}:
\begin{itemize}
\item \code{gsl} --- \code{http://www.gnu.org/software/gsl/}
\item \code{fftw} --- \code{http://www.fftw.org}
\item \code{mysql} --- \code{http://www.mysql.org}
\item \code{cfitsio} ---
  \code{http://heasarc.gsfc.nasa.gov/docs/software/fitsio/fitsio.html}
\item \code{libjpeg} --- \code{http://www.ijg.org}
\item \code{openssl} --- \code{http://www.openssl.org}
\item \code{doxygen} (optional: for generating documentation) ---
  \code{http://www.stack.nl/~dimitri/doxygen/}
\end{itemize}

In addition to the above, \code{libpng} (\code{http://www.libpng.org})
is required to build the \code{Ohana} package.


\subsubsubsection{Perl modules}

\begin{figure}
\psfig{file=perl.ps,height=8in,angle=0}
\caption{Dependency chart for the IPP Perl components}
\label{fig:perl-dependencies}
\end{figure}

The following external modules, available from CPAN (see
\S\ref{sec:installing-perl-dependencies}), are required to build the
Perl scripts:
\begin{itemize}
\item \code{Module::Build}
\item \code{ExtUtils::MakeMaker}
\item \code{Params::Validate}
\item \code{DateTime::TimeZone}
\item \code{DateTime::Locale}
\item \code{Time::Local}
\item \code{DateTime}
\item \code{MIME::Base64}
\item \code{IO::Compress::Base}
\item \code{Compress::Raw::Zlib}
\item \code{Class::Factory::Util}
\item \code{DateTime::Format::Strptime}
\item \code{Net::Domain::TLD}
\item \code{Sub::Uplevel}
\item \code{HTML::Tagset}
\item \code{Digest}
\item \code{IO::Compress::Zlib}
\item \code{version}
\item \code{Text::Balanced}
\item \code{DateTime::Format::Builder}
\item \code{ExtUtils::Manifest}
\item \code{URI}
\item \code{Data::Validate::Domain}
\item \code{Test::Exception}
\item \code{Tree::DAG_Node}
\item \code{Array::Compare}
\item \code{HTML::Parser}
\item \code{Digest::MD5}
\item \code{Net::FTP}
\item \code{Compress::Zlib}
\item \code{Locale::Maketext::Simple}
\item \code{Parse::RecDescent}
\item \code{Class::Accessor}
\item \code{DateTime::Format::ISO8601}
\item \code{CGI}
\item \code{Test::Cmd}
\item \code{Net::HTTPServer}
\item \code{Digest::MD5::File}
\item \code{File::Temp}
\item \code{Data::Validate::URI}
\item \code{Test::Warn}
\item \code{YAML}
\item \code{LWP}
\item \code{Module::Load}
\item \code{Params::Check}
\item \code{Template}
\item \code{Statistics::Descriptive}
\item \code{Storable}
\item \code{IO::String}
\item \code{Date::Parse}
\item \code{Digest::SHA1}
\item \code{DB_File}
\item \code{File::NFSLock}
\item \code{Heap}
\item \code{Module::Load::Conditional}
\item \code{IPC::Run}
\item \code{Cache}
\item \code{IPC::Cmd}
\end{itemize}

\subsection{Binaries}

Installation of the binaries is complicated by the fact that they may
be used on multiple architectures.  The three developers based at
Pan-STARRS HQ each use a different method for configuring the
environment and installing the binaries to deal with this problem.  We
describe each of these below.  Choose one that works for you!

\section{Trouble Shooting Build Issues}

\subsection{missing libX11.so}

RedHat RHEL 3 \& 4 (likely other RedHat variants as well) for \emph{amd64} seem
to often be installed without a \code{libX11.so} under
\code{/usr/X11R6/lib64/}.  This will cause 64bit builds trying to link against
libX11 (\code{-lX11}) to fail.

If you have root access you can fix this by adding the missing symlink
yourself.  Eg..

\begin{verbatim}
su -
cd /usr/X116/lib64
ln -s libX11.so.6.2 libX11.so
\end{verbatim}

or with \code{sudo}:

\begin{verbatim}
cd /usr/X116/lib64
sudo ln -s libX11.so.6.2 libX11.so
\end{verbatim}


For users without root access to thier system ,a symlink needs to be installed
under \code{\$prefix/lib}, where \code{\$prefix} is the path underwhich your
installing IPP software.  Note that \code{\$prefix/lib} will also need to be manually added the enviroment variable \code{LD_LIBRARY_PATH} if your not using jhbuild. E.g for ksh/bash users:

\begin{verbatim}
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/[foo]/[yourinstallprefix]/lib
\end{verbatim}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
                                                                                
\clearpage
\appendix
 
\bibliographystyle{plain}
\bibliography{panstarrs}
\end{document}
