\documentclass[panstarrs]{panstarrs}

% basic document variables
\title{Response to the IPP CDR Committee Report}
\subtitle{}
\shorttitle{IPP CDD Report Response}
\author{Eugene A. Magnier}
\audience{PMO}
\group{Pan-STARRS Algorithm Group}
\project{Pan-STARRS Image Processing Pipeline}
\organization{Institute for Astronomy}
\version{DR}
\docnumber{PSDC-440-004}

% allow paragraphs to be listed in TOC for now 
\setcounter{tocdepth}{3} 

\begin{document}
\maketitle

% -- Revision History --
%\RevisionsStart
% version     Date         Description
%00     & 2006.08.13 & Initial Release \\ \hline
%\RevisionsEnd

\tableofcontents
\pagebreak
\pagenumbering{arabic}

\section{Document Overview}

The IPP team appreciates the care, detail, and effort which went into
the IPP CDR Committee Report.  The guidance and suggestions provided
by this review are extremely valuable and will help us to refine the
IPP design and improve the overall reliability and usability.

This document is meant to address concerns raised by the CDR report
and offer a fuller explanation of certain issues.  Some of the
questions raised are addressed in separate documents (IPP-related
ICDs, Updates to the IPP SSDD, the IPP Commissioning Plan, and the IPP
Analysis Parameters Guide).  In particular, the CDR Review Committee
listed the following Action Items and requested a report on items 3-10
from the IPP team.

\begin{enumerate}
\item Work with other Pan-STARRS project members to complete a system
level requirements specification.  Not having such a document at this
late stage of PS1 development is a significant risk, and it is
incumbent upon all relevant parties to address this as soon as
possible.  (Note: Although recommending system level actions is
strictly outside the scope of the charge to this committee, it is
obvious that additional system level and operational requirements need
to be specified/clarified in the immediate future.)

\item Currently, no requirements on completeness and reliability have
been specified for the PS1 Design Reference Mission. The project
should develop internal goals for at least completeness to help IPP
development set limits on how hard to work on difficult objects.

\item Identify more explicitly the operational requirements of the IPP
(what is the IPP required to do as part of the complete PS1
system?). In particular, compare the functionality of certain IPP
components relative to the PSPS.  Then ensure that any functional
redundancy is intentionally present to increase system robustness, and
does not represent a needless duplication of effort.

\item Major version deliveries of the IPP should be defined in the
development schedule.  Each major version should be associated with
specific tasks and added functionality.  It is a good thing to have
major milestones not just to track progress, but also to help focus
the development work by the team.  Making these major deliveries can
also give a sense of accomplishment to the team, helping with morale.

\item All the external and internal interfaces to/within the IPP
should be specified in detail.  For the external interfaces, this
includes documenting the requirements in Interface Requirements
Specifications (IRSs) or as part of the PS1 System/Subsystem
Specification (SSS). The interface implementation should be presented
in Interface Control Documents (ICDs).  For future reference and
project reviews maintain a matrix showing the status of each, and
describe the ICD maintenance and update process.

\item Present the plan for finalizing the analysis algorithms.  The
plan should include a complete list of analysis algorithms and
functions, the status/completeness of each, what data is required to
complete those not yet fully implemented or designed, and (if known)
what choices are available to be traded.

\item Establish more clearly the data validation process within the
PS1 operations plan and budget.

\item Provide the IPP commissioning plan.

\item Provide additional information as to how the proposed hardware
configuration allows IPP to meet its requirements.

\item Explain more fully design philosophy and benefits of maintaining
CFHT compatibility.
\end{enumerate}

\section{Action Item 3: Operational Requirements of the IPP}

This question appears to be principally concerned with perceived
overlap between the IPP DVO system and the PSPS Object Data Manager
(ODM).  While there is superficial similarity between the systems, it
is our belief that the development of two complementary systems is not
only worthwhile, but extremely important for a variety of reasons.

The IPP operational requirements include the analysis of images to
produce the P2, P4$\Delta$, and P4$\Sigma$ detections, the Static Sky
images, the Astrometric and Photometric Reference catalog, and the
astrometric and photometric calibration of the data products (images
and detections).  These requirements are highly coupled: in order to
produce the Static Sky at the level required, the IPP must perform
astrometric and photometric calibrations of the data stream at a high
level of accuracy.  In order to perform these calibrations, the IPP
must construct an improved astrometric and photometric reference
catalog to supplant the existing references, at least within the PS1
pass-bands.  

The DVO system is designed to meet the twin requirements of performing
the astrometric and photometric calibrations and to perform the
construction of the astrometric and photometric reference catalog.
The IPP DVO system and the PSPS thus have overlapping capabilities in
terms of linking detections together into objects on the sky and in
terms of tracking the relationships between the detections, images,
and objects, along with the photometric and astrometric calibration
information.  There are several motivations for defining a separate
entity within the IPP specifically for this task:

\begin{itemize}
\item The PSPS system is foreseen as the definitive view on the object
  database problem, while the IPP requires the ability to manipulate
  and update the astrometric and photometric calibrations.  
\item The PSPS system requires a high-level of sophistication in the
  scientific queries which may be performed.  While critical, this
  sophistication will result in a much longer development time line for
  the PSPS.  
\item By performing the analysis of the astrometric and photometric
  calibration within the DVO system, and avoiding these requirements
  within the PSPS, the PSPS design may be focused on the database
  management issues of supplying the data to users.
\item The IPP requirements for object databasing are fairly limited in
  terms of the ability to perform detection/object matching.  This
  trade-off allows the IPP DVO system to operate at a much higher
  throughput than is possible for the PSPS, and perform more
  re-processing operations.
\end{itemize}

It is useful to ask if the PSPS could have been ready within the
necessary time frame, would the IPP use that instead of the DVO system
for support of photometric and astrometric calibrations.  The answer
would be 'yes' only if the PSPS requirements were modified to allow it
to perform the same operations as required by the IPP DVO.  Would it
be worthwhile to the project to merge the functionality of the two
subsystems after they have both matured?  My opinion, and I believe,
that of the project management, is that this choice would result in
development of a system which would be substantially over-engineered
for its role in either of the two subsystems, with concomitant
increase in the cost of development and support.  By dividing the
tasks served by these two systems in a logical way, the project as a
whole is better served.  

\section{Action Item 4: Major version deliveries of the IPP}

\subsection{IPP Release 1 (Aug 30, 2006) : Phase 1-3, Detrend Creation, Infrastructure, ISP Support}

\begin{itemize}
\item {\bf Release Contents} Analysis programs for Phases 1-3 and the
  Detrend Creation analysis, PanTasks (the controller/scheduler),
  PanTasks scripts for Phase 1-3 and Detrend Creation, the Metadata
  Database tools (ippTools) needed to track the process flow,
  Nebulous, and DVO.
\item {\bf Release Capabilities} Detrend image creation (bias, dark,
  flat, fringe), image detrending (excluding convolved guide kernels),
  single-image positive object detection and classification,
  astrometric calibration, object detection databasing, basic
  zero-point analysis.  Real-time ISP transparency measurements.
\end{itemize}

\subsection{IPP Release 2 (Oct 30, 2006) : Phase 4, static sky pixel definitions}

\begin{itemize}
\item {\bf Release Contents} Phase 4 analysis programs, Phase 4
  PanTasks scripts, Static Sky pixel layout, interface to Magic,
  psphot and ppImage upgrades.
\item {\bf Release Capabilities} Image warping, Image differencing,
  Image stacking, object modeling for difference images (loading PSF
  from an external source, fitting positive and negative sources),
  analysis of the OTA guide kernel, detrend images convolved with the
  guiding kernel. 
\end{itemize}

\subsection{IPP Release 3 (Dec 30, 2006) : Static Sky, AP Catalog tools, output interfaces}

\begin{itemize}
\item {\bf Release Contents} Updates to psphot; DVO crawler tools
  (uniphot, relphot, relastro); external interfaces.
\item {\bf Release Capabilities} Static sky version of the photometry
  analysis, with more-complete characterization of galaxy parameters
  and the simultaneous multi-filter photometry; tools to perform the
  astrometric and photometric calibration of the all-sky survey data;
  the interfaces to provide data to the MOPS, PSPS, and other science
  clients as defined.
\end{itemize}

\subsection{IPP Release 4 (Mar 30, 2007) : Code Freeze for ORR and the Survey Start}

\begin{itemize}
\item {\bf Release Contents} Updates to programs identified in the
  commissioning.
\item {\bf Release Capabilities} Improvements determined on the basis
  of the commissioning with GPC-1 (possibly new PSPhot PSF models,
  temperature dependent selects of input detrend data as needed, etc).
\end{itemize}

\section{Action Item 5: Internal \& External Interfaces}

We have worked with the other PS1 teams to define complete ICDs
between all of the interacting subsystems.  The IPP interfaces with 4
existing subsystems, and will interact with other as-yet-undefined
science clients.  We have completed the definitions of the ICDs for
the Camera-IPP (PSDC-940-003), OTIS-IPP (PSDC-940-004), IPP-MOPS
(PSDC-940-005), and IPP-PSPS (PSDC-940-006).  These are now part of
the PSDC document tree.

We have clarified the internal interfaces of the IPP in the SSDD.  We
have improved the discussion of the types of interfaces, created a
table listing the major interfaces, and identified the data format for
each of the interfaces.  The detailed contents of those interfaces are
specified in the software design description of the relevant software
components.  For example, the SDD for PSPhot defines the formats of
output object data tables, while the SDD for DVO/addstar references
this document.

\section{Action Item 6 : Plan for Finalizing the Analysis Algorithms}

\begin{table}[ht]
\begin{center}
\caption{IPP Analysis Algorithms\label{algorithms}}
\begin{tabular}{llll}
\hline
\hline
{\bf Analysis}	       	 & {\bf IPP program}      & {\bf Status}  & {\bf Data Needed} \\
\hline
Bias Creation 	       	 & \code{ppMerge} 	  & completed     & raw bias images (range of conditions) \\
Dark Creation 	       	 & \code{ppMerge} 	  & completed     & raw dark images (range of conditions) \\
Flat Creation 	       	 & \code{ppMerge} 	  & completed     & raw flat images (dome and sky flats) \\
Fringe Creation        	 & \code{ppMerge} 	  & 90\%          & raw fringe image (dome and night-sky images) \\
\hline   
Overscan Correction    	 & \code{ppImage} 	  & completed     & raw bias images (range of conditions) \\
OTA Convolution	       	 & \code{ppImage} 	  & partial       & raw science images \\
Non-Linear Correction  	 & \code{ppImage} 	  & partial       & correction functions (from camera group) \\
Shutter Correction     	 & \code{ppImage} 	  & missing       & correction functions (from camera group) \\
Domeflats or skyflats?   & \code{ppImage} 	  & completed     & photometry dither images \\
Domefringe or skyfringes? & \code{ppImage} 	  & completed     & night-time images (range of sky conditions) \\
\hline   
PSF model	       	 & \code{psphot}  	  & completed$^1$ & science images (range of seeing) \\
Sky model grid size    	 & \code{psphot}  	  & completed     & science images (range of seeing) \\
Other PSPhot parameters  & \code{psphot}  	  & completed     & science images (range of seeing) \\
\hline
Astrometry search radius & \code{psastro} 	  & completed     & telescope pointing measurements \\
Astrometry model order   & \code{psastro} 	  & completed     & science images (range of focus) \\
\hline
Interpolation kernel     & \code{stac}            & partial       & science images \\
Difference image kernel  & \code{poisub}          & partial       & science images \\
Static Sky cells         & \code{stac}            & partial       & simulations \\
Static Sky combination   & \code{stac}            & partial       & science images \\
\hline
\multicolumn{4}{l}{$^1$Only analytical models are implemented. If more
complex models are needed they will have to be implemented.}
\end{tabular}
\end{center}
\end{table}

The CDR Review Committee requests a plan for finalizing the analysis
algorithms within the IPP.  There are three stages to freezing the
analysis algorithms within the IPP:
\begin{itemize}
\item Algorithm conceptual design.  
\item Development of the software used to implement the algorithm.
\item Definition of parameters which modify the details of an
  algorithm.
\end{itemize}.

The IPP SSDD defines the conceptual details of nearly all of the
analysis algorithms in Sections 5, 6, and 7.  Of these descriptions,
only Sections 5.5.5, describing the stacking of the Static Sky, and
Section 7.2, defining the Static Sky photometry analysis, are
insufficient in detail for the analysis to be well understood.  The
discussion of the analysis used to generate the astrometric and
photometric reference catalog are included in Section 4 on the design
of DVO, and are somewhat sparse.  A new version of the SSDD with these
three sections updated will be posted by Aug 30.

Table~\ref{algorithms} lists the IPP algorithms, the relevant program
in the IPP tree, the development state of the program with respect to
the identified analysis, and the data needed to set the algorithm
parameters.  In a separate document (`How to Define IPP Analysis
Parameters'), we present a detailed discussion of the choices to be
made and guidelines on making those choices.

There are a handful of decisions which need to be made which have an
important impact on the way the analysis is performed.  Most of these
will have a major impact on the results, but have little or no impact
on the IPP {\em design}.  The one exception is the question of exactly
how the object analysis is performed on the Static Sky, as there are
several possible choices here.  The major analysis issues to address
are:

\begin{itemize}
\item {\bf Skyflats or Domeflats?}  We will need to choose between
  skyflats and domeflats. This choice will require a comparison of the
  photometric accuracy achieved by both methods.  Also important in
  making this decision are the questions of how efficiently raw flats
  may be obtained using both methods, how stable the two versions are,
  and what kind of operations load they represent.  These are
  questions beyond the scope of IPP and are largely the responsibility
  of OTIS.  It is also an open question at this time if the planned
  calibration screen will be available at the start of the Survey.
  This is an operational requirement/decision that does not affect the
  IPP design, though feedback from the IPP will be critical in making
  the decision.

\item {\bf Skyfringe or Domefringe?} This decision is similar to the
  question of domeflats vs skyflats.  Here the operational concerns
  for the domefringe images are more acute.  An important trade which
  must be made is how many raw fringe images are needed to generate
  the fringe images to be used by IPP.  It is also critical that the
  number of input fringe images required by IPP for any Phase 2
  analysis be minimized since it is impossible to afford the I/O load
  demanded by a large number of input fringe images.  A related
  question is that of how to sub-select the night-time fringe images
  for best effect, if sky fringe images are used.  Based on experience
  from CFHT/Megacam, it may be possible to use fringe images selected
  on the basis of the time of night, but this must be tested for
  Haleakala.  An alternative strategy, in which master fringe images
  are combined based on their consistency with the science image, has
  also been implemented for the IPP.  The decision between these
  options will be guided by further testing of Megacam images and real
  Haleakala data in a range of conditions.  It seems unlikely at this
  time that a spectral skyprobe will be available for the start of
  PS1, so we cannot rely on such a device to guide our choices.

\item {\bf Static Sky Cell definition} What is the layout of the
  Static Sky cells, and how does this depend on the different surveys?
  In particular, what are the projection centers and the size of
  individual files?  This question can be answered before we begin
  collecting data by testing several of the competing options.  The
  software is not tied to any specific choice of static sky cell
  arrangement.  The simulations required to perform this decision are
  being deferred until the IPP Release 1 can be completed.

\item {\bf How is the image difference PSF-matching kernel
  represented?}  The software currently implements the Alard-Lupton
  kernel (multiple Gaussians) and the POIS kernel (array of delta
  functions).  The software is easily modified to accommodate other
  equivalent kernels such as Airy or Bessel-function decompositions.
  It will be necessary to perform tests with real GPC-1 images to
  determine which of these kernels achieves the best results.

\item {\bf What interpolation kernel is used for image warping?}  The
  choice of interpolation kernel has two important impacts on the
  resulting data product.  First, there is a trade-off between the
  accuracy of the kernel and the processing time required for the
  analysis.  Second, there is the question of aliasing of
  high-frequency features.  The first trade-off can be easily
  performed with speed tests of the algorithms on the test hardware.
  The second part of this question requires examples of the range of
  PSFs and noise properties in real PS1 images.

\item {\bf How are the input images to the static sky combined?}  Once
  images are warped, there are several choices to be made in how the
  data from multiple images are combined into a single static sky
  image.  How do we select the input images as their PSFs change?  How
  are images with different PSFs weighted?  What smoothing kernel (if
  any) is applied to the input images before combination?  The 3$\pi$
  survey will necessarily require the combination of pixels from a
  range of field angles into the static sky.  These will have a range
  of image qualities.  We will need to set the cuts to trade-off
  between degradation of the final image quality versus degradation of
  the signal-to-noise in the final image.  In general, our guidance
  for the Static Sky is to maximize our ability to measure accurate
  morphology on the images (to meet the Weak Lensing requirements) and
  to provide the best reference for the image differencing algorithms.
  Astrometric and photometric accuracy of the static sky measurements
  is not as critical a driving requirement.

\item {\bf How are the Static Sky images processed?}.  This particular
  issue is critical to finalizing several aspects of the IPP design.
  First, is it possible to meet the requirements for the Static Sky
  (weak lensing accuracy and difference image accuracy) with a single
  combination?  Is it necessary to degrade all images to a single,
  common seeing (for a stable PSF across the image for difference
  image)?  Is it possible to measure the weak lensing parameters
  sufficiently accurately from the stacked image with knowledge of the
  PSF?  To what level of detail is the PSF model required?  Is it
  necessary to perform the weak lensing analysis (and other galaxy
  shape parameter measurements) from the individual images rather than
  from the Static Sky?

\end{itemize}

\section{Action Item 7: Data Validation Process}

Throughout the IPP, the analysis steps have been defined with data
validation as an integral step.  As the images are processed through
the detrend creation stage, the Phase 1-3, and Phase 4, the analysis
performed includes tests of the quality of the input images.  These
measurements are included in the stream of metadata produced by the
IPP (e.g., Tables 23, 25, 37 IPP SSDD).  In the discussion below, we
identify the data validation measurements performed for the different
steps.  The functional flow of these different analysis steps can be
seen in IPP SSDD Figures 22-30.  The use of the Q/A measurements is
not summarized very clearly within the text of version 00; this will
be updated in the new SSDD release.  The types of Q/A measurements are
also discussed below.

Within the IPP, the analysis stages use these measurements to mark
input images as accepted or rejected.  These assessments are passed
back to the OTIS subsystem, along with the image statistics measured
for the input images.  OTIS has the option of setting more stringent
filters on the input images and re-observing images on the basis of
the IPP feedback, even if the IPP accepted images which OTIS
re-observes.  There is also a system-wide plan in place to use
feedback from the IPP and from OTIS to guide the project's choices for
survey strategies and science goals.

\subsection{Detrend Images}

The input detrend images have their pixel count levels measured for
each chip.  Input images which have counts or fluxes outside of a
defined range will be flagged and excluded from any detrend analysis.
For example, the flat-field images should never use input images which
are saturated, nor should the dark image analysis use input images
with flux levels wildly outside of the nominal range.  Both conditions
are evidence that the data were incorrectly obtained.

In addition to raw pixel values, the input detrend images are
confronted with the resulting master detrend images.  In general, the
effect corrected by the master detrend image should adequately correct
each of the input raw detrend images.  The residual scatter of the
detrended raw images should be small.  As part of the detrend creation
process, input images with excessive residual scatter are
automatically rejected from the input stack.  These images are also
flagged in the database.  

The master detrend images are also evaluated by using the statistics
of the (surviving) input residual images.  The global statistics of
the residuals from the input set of images can be used to judge if the
master detrend images has sufficient signal-to-noise or consistency to
meet the requirements of the detrending process.

\subsection{Science Images : Single-Image phases}

As the science images are processed, a number of measurements are
performed to tests the quality of these input images.  I discuss here
the measurements which will be performed.

\begin{itemize}
\item median sky background : this can be used to trigger on images
  taken in very poor conditions (eg, too early, dome lights on, too
  close to the moon).
\item sky background statistics after detrending : this can be used to
  detect images taken under poor conditions (close to the moon, many
  bright stars).  In the early stages, until we have a model for the
  static sky surface brightness in each of the PS filters, these
  measurements will only provide a reference and cannot be used as a
  rigid identification of good/bad images. 
\item fringe residual statistics : after the fringe correction has
  been performed, a measurement of the variations correlated the
  fringe pattern can be used to measure the quality of the fringe
  correction.  Images which fall outside of the nominal range may be
  indicative of sky emission line conditions not expected in the
  initial master fringe sample, and may indicate a way towards further
  improvements to the fringe correction process.
\item image quality statistics : the PSF as a function of field
  position can be used to detect images taken in degraded focus or
  seeing conditions.  Images outside defined ranges can be excluded
  from further analysis.
\item Astrometric calibration \& scatter : failure of the astrometric
  analysis to converge, or elevated scatter of the astrometric
  solution are both evidence of images taken in poor conditions.
  These may include telescope mis-pointing, tracking failures, or
  focus failures.
\item Photometric calibration \& scatter : Excessive variations of the
  zero-point as a function of mosaic position, generally elevated
  scatter, and substantial photometric offsets are all evidence of
  problems with the observing conditions.  These may be the presence
  of clouds and/or haze, degradation of the optics, and/or extreme
  image-quality problems.
\end{itemize}


\subsection{Science Images : Image Stacking and Image Difference Phases}

During the image stacking (P4$\Sigma$) and image differencing
(P4$\Delta$) stages, data validation is based on measurements of
consistency of the input images with the output stack and on the
number of peaks detected in the difference image.  Images which have
passed the single-image quality tests should in general succeed on the
image stacking and difference stages.  However, large deviations can
be expected in the event of unusual features within the images.  For
example, uncorrected background variations from image to image will
result in large deviations between the components of an image stack,
and will also result in large numbers of difference image detections.
Similarly, bright stars with larger-than-expected halos or saturation
regions will result in excess difference image detections.

\section{Action Item 8 : IPP Commissioning Plan}

The IPP commissioning plan is supplied in the accompanying document,
IPP Commissioning Plan.

\section{Action Item 9 : Hardware Configuration and Requirements}

The proposed IPP hardware configuration is driven by three major
requirements: storage, processing power, I/O bandwidth.  We have
performed a detailed analysis of these issues, which we have presented
in Section 12 of the SSDD.  We concluded in our analysis that the
storage component of the requirements completely dominated our
hardware design considerations:  by specifying reasonable hardware to
meet the storage requirements, we were supplying more than enough
hardware to meet the processing and I/O requirements. 

That analysis was based largely on prototype tests of our processing
algorithms and has been made out-of-date by changes to the survey
plans.  We present here new numbers for the processing time line based
on the current baseline software on our existing baseline cluster
hardware.

% For PS1, there is a significant processing challenge in the first 6 -
% 9 months when only a fraction of the IPP storage hardware will be
% available.  This period is further complicated by the budgetary
% constraints placed on the IPP to limit the hardware purchase to a bare
% minimum.  An important area for clarification by the project is the
% processing requirement in the beginning of the project.  If the IPP is
% required to perform a complete Static Sky analysis on every image as
% it becomes available, then the total hardware required in the first
% 6-9 months for processing must be increased.  If it is only necessary
% to stack sets of, for example, 4 images as they are available, then
% the requirements are somewhat reduced.  A trade-off must be made by
% the project to choose between these options.

The data storage requirements are determined from the design reference
mission.  The current plan for the design reference mission has
increased somewhat the number of images obtained by the survey.  The
current plan consists of an All-Sky Survey re-visiting each point in
the sky 60 times over the 3 years (12 visits per filter).  There is
also a Medium Deep survey which is expected to generate roughly 75000
images per year.  Combining these two, we find that the total number
of raw image data is roughly 1.7PB (555,000 images).  In addition, we
have a requirement for Static Sky storage (assuming 0.2 arcsec pixels)
of roughly 300TB, and miscellaneous additional storage of nearly 100
TB.  Our hardware purchase plan has a minimum total storage of 2.4PB,
giving us a margin of about 10\%.  Our plan is to purchase the
hardware in 5 stages of 16 computers each, for a total system cost of
roughly \$1.25M (each purchase set is expected to cost roughly
\$250k).  Making reasonable estimates of disk capacity growth, we
expect these 80 machines to meet the storage capacity requirements.
Currently, we are able to buy 16.5TB per machine.  In the worst case
that the hard disks stopped increasing in storage capacity (or we were
required to buy all of the machines up front), we would require
roughly 145 machines, increasing the cost of the cluster to a total of
roughly \$2.2M.  We judge this to be a very low risk as hard disk
capacities continue to grow.

We have performed timing test of the current versions of the IPP tools
for the different stages of the analysis using MegaCam images.  Four
factors are critical in understanding if the cluster will be capable
of performing the processing needed: 1) processing power of the
cluster, 2) speed of the dominant analysis steps, 3) total number of
object for which a full non-linear model fit is performed, 4) total
number of images for which the warping and image differencing is
performed.

Current CPU development roadmaps project a slow rise in the clock
speeds of the CPU cores, but increasing numbers of cores per socket.
By the end of this year, quad-core processors are expected.  If we
stagger the purchase of the computers as planned (5 sets of 16
machines), and make reasonable estimates for the number of cores
available, we expect the final cluster configuration to have between
400 and 800 cores.  We will use 600 as an estimate.  Note that it is
possible to supplement the processing power of the cluster by buying
1U boxes with processors but minimal storage.  Each of these boxes
cost roughly 15\% of a storage node and add an equal number of
processor cores.  Such an option can be taken at any time to
supplement the raw processing power, though it is not needed in our
current development plan.

There are two potentially dominant analysis steps in the process: the
warping and PSF-matching for a single image (ie, the Phase 4 steps)
and the non-linear model fitting performed on the brighter sources.
The processing time for the warping and PSF-matching is essentially
constant for each image.  On our demonstration hardware, this step
takes roughly 16 seconds for a Megacam chip (single core), equivalent
to roughly 38 seconds on a full GPC-1 chip.  Most other steps of the
analysis require a constant amount of time per image, and contribute
only a few seconds relative to the 38 seconds.  We use 50 seconds per
chip per core to judge the total processing power for the portion
which scales by the number of images.

A useful statistic to judge the capability of the processing system is
the time required to re-process all images at the end of the survey.
Given the total number of images above (555,000), the per-image
analysis portion of the processing would require a total of $\sim 1.8
\times 10^9$ CPU core-seconds, or about 35 days on the 600 cores.

The non-linear fitting speed is a bit more uncertain: the speed will
depend on the number of parameters in the model and the number of
pixels needed for most objects.  We have run a variety of example
Megacam images and measured the speed per object for a several
possible combinations of models and object footprints.  We find that
the speed ranges from 2 - 10 milliseconds per object.  This number can
be confronted with the total number of objects in images for which the
non-linear fitting is performed.  This number depends on both our
choice of a threshold for the non-linear fitting and the actual sky
density of sources at that depth.  Our current proposal is to perform
the non-linear fitting for all objects above a S/N limit of 20.  The
total number of sources is dominated by the sources in the Galactic
Plane.  As a rough estimate of the total number of sources above our
threshold, we have examined the 2MASS catalog in the Galactic Plane
anti-center region.  We have used the 2MASS and USNO-B colors to
predict the Pan-STARRS magnitudes of stars, then extrapolated the
source counts to our magnitudes limits.  We find 50,000 objects per
square degree above our threshold in this region.  If we make the
conservative assumption that every image required the non-linear
fitting for this density of objects, and that each object requies
10ms, this analysis would require a total of $\sim 2.0 \times 10^9$
CPU core-seconds, or about 39 days on the 600 cores.

In conclusion, given the assumptions above, the processing power of
the proposed hardware will be sufficient to allow for a complete
re-processing of all survey data within less than three months after
the survey is completed.  We recognize, however, that there are still
some large potential uncertainties in this analysis, particularly in
the number of sources for which the analysis is performed.  To get a
better understanding of the sky density of sources, we are extending
our analysis of the 2MASS and USNO-B source densities across the full
sky.  However, given the fact that the processing power of the cluster
can be extended significantly without a large fractional cost, and the
fact that the re-processing time above is well below our requirement,
we do not believe there is a large risk in the hardware capabilities
of the IPP design.

\section{Action Item 10 : CFHT compatibility}

The concern that the IPP is driven by CFHT compatibility issues is
understandable, but easily addressed.  The IPP design is not locked
into any specific design choices of the CFHT Elixir system.  The IPP
uses one major subsystem, DVO, which is also used by the Elixir
system.  It also is using code shared by certain Elixir systems in the
IPP scheduler/controller system (PanTasks).

The IPP is using the DVO system developed at CFHT for the object
databasing.  This is a natural decision because the DVO system 1)
existed, 2) was extensively tested, and 3) met most of the basic
requirements of the IPP for object databasing.  Some effort has been
needed to make DVO completely suitable for its role within the IPP.
Regardless of what object databasing system was chosen, however, a
certain level of effort would have been required.  In the case of DVO,
we were clear what effort was required, and it was judged to be
reasonable.

Of that effort, only the ability to support older table formats was
required to maintain compatibility with the existing CFHT DVO
databases.  In fact, this is a feature which we would have added even
if we did not want to maintain compatibility with CFHT's DVO
installation.  We have found in our experience with the Elixir system
that having a rigidly defined schema hindered the usability and
extensibility of the DVO system.  The fixed tables made it difficult
to add new elements to the database, and required multiple versions to
support previously defined tables.  The new design allows us to be
more flexible about changes without fear that this will break database
instances which already exist.  One of the best ways we have found to
test the DVO object databasing system is to engage students in science
projects using DVO.  These projects explore the user interface and
highlight problems and areas for possible improvement.  Such projects
would not be possible if the users feared that their DVO instances
would be unusable in the future because of lack of backwards
compatibility.

The PanTasks system used the existing Opihi command-line interface
system which is also used by elements of DVO among other programs.
The choice to use this existing framework made the implementation of
the PanTasks system quick and efficient.

The IPP owes a substantial intellectual heritage to the CFHT Elixir
pipeline.  What the IPP gains by maintaining personal relationships
with the CFHT Elixir team is access to an unparalleled set of test
data and ongoing experience with a live running system facing the same
concerns the IPP will soon face.  For example, only by virtue of this
relationship are we able to test the IPP flat-field correction scheme
of a large set of real images obtained by the CFHT engineering staff
over several years.  We also gain by discussions with our Elixir
colleagues about details of the analysis and possible sources of errors
observed in the CFHT dataset.  The only cost to the IPP is in
preventing excessive forking of the DVO databasing system, something
we would attempt to avoid in any case.

\end{document}

-----------------------------------------------------

* things we need to do within IPP

  - clarify the rules of MAGIC
    - do we run a version MAGIC on P2 images?
    - do we use the imclean algorithm to remove detection lines?
  - better list of all data products with details
  - better list of all metadata database tables
    - identify the Q/A elements
  - explain authority chain:
    - image headers have the primary authority
    - non-image tables from the summit have the primary authority
    - metadata database table elements derived from 

  - completeness
    - static sky images + weight maps specify if a region has been
  observed at all
    - DVO image query specifies which images (generally) overlapped an
  area.
    - queries via IPP mask tool specify if a location could have been
  observed (PSF * mask).
    - queries via IPP object tool could be used to force an object
  measurement on a given location (PSF or aperture).
    - IPP is willing to support up to 10 / min / machine (~200 - 800
  per minute) during the daytime.  one request may specify many
  objects per image (up to 100?)

  - clarify DVO / MySQL relationship.  
    - DVO needs to use the FITS tables internally for speed
    - DVO will export the FITS tables to MySQL for external use

  - better definition / discussion of the Phase 4 and Static Sky
    components.

  - commissioning plan
  - deliverables and schedule


---

hardware test points:

ppImage (no psphot) on Megacam : 40.650u 31.489s 2:46.41 43.3%   0+0k 0+0io 30pf+0w

  * using alala I/O with 50 MB/sec read (and write?)
  * total I/O was 4*708 MB (read) + 2778MB (write) = 112 sec

  * processing time = 40 sec (actual) | 54 sec (clock)
  * Npix = 353Mpix
  * speed = 0.15 sec / Mpix

  ** for 1 OTA : 3.5 sec processing time + 2-3 sec I/O (200 MB/sec)
  ** for 1 ISP : 0.7 sec processing time + I/O

psphot : 45 sec

