\documentclass[panstarrs]{panstarrs}
\newcommand\note[1]{{\bf #1}}
\newcommand\tbd[1]{{\bf [#1]}}

\title{Proposal for the Pilot Project Engine / Architecture}
\shorttitle{IPP.c Architecture Proposal}
\project{Pan-STARRS Pilot Project}
\author{Eugene Magnier}
\group{}
\organization{}
\docnumber{PSDC-400-004-00}

\fancyhead{}
\fancyhead[L]{\theshorttitle}
\fancyhead[C]{}
\fancyhead[R]{\thedocnumber}
\fancyfoot[L]{\theproject}
\fancyfoot[R]{\today}

\begin{document}

%% title page
\pagenumbering{roman} \thispagestyle{empty} \maketitle
\pagebreak 

%% revisions page
\pagestyle{fancy}
{ \Large \bf Revision History}
\begin{center}
\begin{tabular}{|p{1.25in}|p{1in}|p{4in}|}
\hline
{\bf Revision Number} & {\bf Release Date} & {\bf Description} \\
\hline
00 & 2004/01/08 & Proposal \\
\hline
\end{tabular}
\end{center}
\pagebreak

%% contents page
\tableofcontents
\pagebreak

%% figures page
\listoffigures
\pagebreak \pagenumbering{arabic}

%%%%% start of main text

The IPP Engine initially produced by MHPCC, implemented in IPP.c along
with supporting code in the engine/dataAPI and the module/moduleAPI
directories, provides a method to connect the low-level functional
modules together on the basis of a simple scripting language.  This
document attempts to evaluate this engine on the basis of the needs,
expectations, and biases of the IfA team, and to propose modifications
which will adapt the architecture in ways more in line with our
desires.  This document first discusses the current implementation,
second outlines our (IfA) concerns and thoughts about this
implementations, and finally makes a proposal for future
implementation.

\milsection{Engine Description}

The Pilot Project implements low-level modules and the engine, which
connects the modules together.  A module performs a specific,
well-defined task, such as reading in an image from disk to a data
structure, correcting for a bias, or applying a flat-field.  The
engine allows these modules to be connected together in nearly
arbitrary ways.  

For every module and for the engine itself, there are a set of
functions which may be unique to that component.  These include: {\tt
malloc, free, warning handler, addDescriptor, addDataItem,
deleteDataItem}, and {\tt getDataItem}.  

A module is invoked in the config file like this:
\begin{verbatim}
moduleName    par1=value1 par2=value2 par3=value3;
\end{verbatim}

There is no data typing at the script level; type assignement is
performed in the module and in the engine, and depends on the usage of
parameter names.

Two IPP data structures initially carry the information about the
modules: {\tt char *moduleNames} and {\tt DataItemList *params}.  The
list 'params' contains the par=value entries in a linked list of data
structures with name and value pairs elements.  When the parameter
list is parsed, the types are identified (int/float/string) and cast
to those value types in the dataItem structure.  An additional data
type is available to the module programmer, cell.  A cell data type
may be created in the script language by a call to a module which
returns a cell-type, such as readFITS.

There is also the {\tt global} parameter data structure which acts
like an environment variable stack: the complete list is always passed
to the modules.

There is an IPP variable stack 'dataSpace' which contains data
currently in use by IPP.  Modules share data through the dataSpace
stack, either leaving output data on the stack or pulling required
data from the stack, though in the current implementation, the engine
layer (IPP.c) mediates the actual interaction with dataSpace.

\milsubsection{Basic Steps of IPP.c}

The top-level of the IPP program performs the following operations:

\begin{itemize}

\item {\bf Define engine functions (initEngine) } This sets the values
    of the abstracted engine functions discussed above.

\item {\bf Parse configuration file (ParseConfigurationFile) } This
    reads the config file and sets up the moduleName/params
    structures.

\item {\bf Load the identified modules (a loop in main) } This calls
    {\tt loadModule} for each module identified in the config file,
    which in turn creates a Module structure for the module, loads the
    library file associated with the named module, assigns the
    parameter list to the module structure, and calls the module's
    init function.  The init function makes the association between
    module parameters and module data descriptors.  Note that not all
    parameters are associated with a data descriptor; the module
    programmer may choose this association on whatever basis.

\item {\bf Execute each module (a loop in main)}  The execution of the
  module includes the following steps:

  \begin{itemize}
    \item {\bf Determine the module type} This simply loops over the
      module data descriptors and identifies the data type, but gives
      an error if a module defines multiple data types.

    \item {\bf Identify the required data in the dataSpace}  This
      section will automatically create a MAJOR\_FRAME data item in the
      dataSpace if required. 

    \item {\bf Call the module} In this step, callModule iterates
      through data type levels to match the given data entry with the
      given module data level.

    \item {\bf free temporary data}
  \end{itemize}
\end{itemize}

\milsection{IfA Comments and Concerns}

The existing IPP.c engine is quite complex, and performs a significant
fraction of the data manipulation.  Much of the complexity seems to
stem from the concerns about data I/O demands introduced by the large
data volume and the need to control aspects of the parallelization
scheme in great detail.  Aside from the point that parallization is
outside the scope of the Pilot Project, we do not share these I/O
throughput concerns, as discussed below, and are instead concerned
that the level of complexity involved in the current design will
result in a high level of development and debugging effort which is
not justified.  Some of our specific concerns follow.

\milsubsection{Excessive Abstraction (engineFcns)}

The use of unique abstracted functions for each module for such basic
functions as {\tt malloc} and {\tt free} and for the list management
APIs {\tt addDescriptor, addDataItem, deleteDataItem}, and {\tt
getDataItem} seems extreme and unneccesary.  

It is very unclear why this level of flexibility and abstraction is
helpful or useful.  The only one of the engineFcns entries which is
obvious to associate with the module is the warning handler.  Clearly,
it is important for the system, IPP.c, to define its unique value of
malloc and free, but why should this be allowed to vary for each
module?  Also, why should the methods to interact with the I/O data
structures be allowed to vary from module to module.  This
implementations seems to add a layer of complexity that is not
justified.  These may have been introduced for reasons of
parallelization, but we believe this is a red herring.

\milsubsection{Obfuscated Data Handling}

There is a surprisingly complex series of steps necessary for a module
to expose data to other modules, via the IPP.c dataSpace structure.

Consider, for example, the {\tt debias} module, which needs to perform
its operation on a given image.  It is specified in the configuration
file as:
%
\begin{verbatim}
debias image="myImage" overscans=32
\end{verbatim}
%
In this syntax, the term {\tt image=''myImage''} identifies that the
debias image parameter should be associated with the data space item
{\tt myImage}.  In order to provide debias with access to {\tt
myImage}, the following steps take place:

First, the parameters are parsed at the {\tt initModule} stage, and
the name associated with the parameter {\tt image} is identified.
This is then pushed on the module descriptor list with its associated
data type (CELL) and direction (INPUT/OUTPUT).  

Next, when the module is ready to be called, the engine searches the
module descriptor list for an INPUT or INPUT/OUTPUT entry and finds
the corresponding data item from the dataSpace, and pushes it on the
module data space.  (Note that if an entry is output only, the engine
assumes it to require a MAJOR\_FRAME and creates the data space entry.
This is an odd choice: why not have the modules responsible for
creating their own data space entries as needed?)

Finally, back in debias, the module searches the parameter list again
to identify the image parameter name, then searches the module data
space for the matching entry.  

This seems far too complex.  Why not simply have a set of APIs to
access the data space directly?  All of these steps could be reduced
to two steps:

First, search the parameter list for the desired parameter/name
match.  Second, request the data value from the engine data space:

\begin{verbatim}
varName = psDiGetDataItem(*parameters,IMAGE_PARAMETER_NAME);
if (varName) {
  imageItem = psGetDataItemfromDataSpace((char*)varName->value);
}
\end{verbatim} 

There would be no need to expose the dataSpace variable structure to
the modules or to the basic engine code.  This would isolate all data
space access to the modules.  Alternatively, the dataSpace variable
could be passed to the modules directly, rather than via the
temporary module data space.  

\milsubsection{Unnecessary Iteration in IPP.c / execute module}

The function {\tt callModule} has a clever layout to make the
appropriate association between the data type and the module operation
level.  However, it seems to be far too complex.  We do not see the
need for this level of complexity in the engine; we would strongly
prefer that a module have a specified data level of operation (ie, OTA
or CELL) and the iteration over higher levels be performed explicity
either at the script level or at the level of the system which invokes
the engine, while iteration at a lower level be performed within the
module itself.  We are not convinced that the arguments about
parallelizing justify the level of complexity in this implementation.
In addition, the dataDescriptor construction is apparently introduced
to allow the iteration which takes place in the IPP.c / execute module
state.  If the engine-level iteration were dropped, then the entire
dataDescriptor construct could be dropped from the engine.

\milsection{Proposals for the Pilot Project Engine}

We would like to capitalize on the effort expended so far on the
module and engine development.  There is some concern within the IfA
that the current design differs far too much from our expectations to
be of any use.  However, we would like propose here that we try to
make use of the development so far in a way that will bring the ideas
of the two communities into closer agreement.  Therefore, we have
several specific proposals for the IPP.c and for the IPP architectural
developement generally.

The basic architectural requirements boil down to three concepts:
First, there must be a scripting language which will allow a
high-level description of how to link low-level modules together in a
flexible way.  Second, there must be a way of associating a specific
script with a given data item (ie, an input image), or a list of data
items.  Third, there must be a system to organize the execution of
these operations in parallel for many data items.

The existing engine, IPP.c, performs the first of these tasks.
Although we have specific concerns with the implementation and perhaps
the grammar of the scripting language, the general structure defines a
language which allows the modules to be linked together.  

There is a strong concern at the IfA about investing more than a very
minor effort to create a linking language from scratch, for either the
Pilot Project or for the IPP in general.  We feel that such an effort
is likely to waste a substantial amount of time and to have
significant risk of bugs and errors.  The IfA would prefer to have the
real functionality of the modules directly accessible.  

Although the existing module/engine structure makes it rather
difficult, it is easy to envision modules which are flexible enough to
allows us to implement test analysis runs for each of these three
philosophical approaches.  With some minor modification, the existing
IPP structures would make such tests easy and feasible as well.  We
would like to make the following proposals for the Pilot Project
engine / module architecture.

\begin{enumerate}

\item {\bf ignore parallel optimization} Until it is demonstrated
  otherwise, we feel that the addition of large amounts of complexity
  to the engine only for the goal of enabling fancy parallel
  processing optimizations is a distraction and a waste of time.  We
  believe that all of the parallelization can be performed on very
  large data chunks (ie, OTAs), and that these operations are
  essentially independent.  Therefore, no inter-module communication
  is necessary, and each phase (in the sense defined by the complete
  IPP design) can be executed as a single command.  Therefore, only
  scripting sufficient to link modules together in a linear fashion
  for a single data instance is necessary.  If it is ultimately
  necessary to add module-level intercommunication to ease the
  handling of the parallel processing problems, this can be done
  within the modules or with blocking in the scripting language at
  that time.  We believe this stance is strongly supported by the
  switch tests and data I/O rates as examined by Josh Hobblit (see
  section~\ref{datarate}).

\item {\bf provide direct module APIs without engine wrapping}.  The
  current module APIs are completely flat and uniform.  While this has
  some theoretical appeal, it requires the use of the engine
  data API to interact with the modules.  This essentially excludes
  the use of simple C or SWIG'ed Perl code.  In our proposal, each
  module would have an API (or multiple APIs as needed) which
  correspond to their function.  For example, the debias module should
  have the debias functionality accessible via a {\tt divideByFlat (OTA
  *image, OTA *flat);} function call.  Note that this proposal does
  not imply dropping the existing {\tt invokeModule ();} functions
  associated with each module; rather, the invokeModule function
  should wrap the lower-level call.

\item {\bf simplify the data-space API} If the engine is only
  responsible for scripting, there is no need for such a complex set
  of interactions to handle data between the modules and the engine.
  The definition of a simple set of engine data-space APIs would allow
  the modules to gain access to the data structures known by other
  modules without so many layers of complexity.

\item {\bf simplify module API} If the modules interact via a simple
  data space API layer, then the module API need only consist of the
  module parameters.  The modules should make their own decisions
  about data typing issues and should simply return errors if the
  script programmers attempt to pass the wrong data types to a module.

\item {\bf remove engine-level iteration / implement module iteration}
  The modules should have a defined level or set of levels of
  operation.  We would like to see the modules all operating at the
  OTA level, since that is the proposed parallelization quantum.  For
  example, the debias module should receive the OTA structure and
  performe the iteration if necessary.  This is already implemented by
  readFITS.c 

\item {\bf freeze grammar} We would like to freeze the IPP.c grammar
  for the purpose of the Pilot Project.  Although the current grammar
  is very simple, and minor changes could make the grammar much more
  sophisticated, what has been delivered will suffice for the Pilot
  Project.  Since this is potentially an area where far more
  development time may be spent than is really necessary, we would
  like to avoid extending the range of the grammar at this time.

\item {\bf implement SWIG interface files}  If the modules have been
  defined with direct access to the low-level functionality, then it
  becomes trivial to write SWIG interface files.  Each module should
  have an associated header file (eg, debias.c gets a debias.h) and an
  associated interface file (eg, debias.i) which is used to generate
  the SWIG wrapper file (eg, debias\_wrap.c).  

\item {\bf implement sample Perl script using SWIG}.  With the defined
  SWIG interface files, it is simple to generate wrapper files for the
  functions specific to one of the languages supported by SWIG.  The
  modules can then be linked together in a script within that language
  by using calls to those modules.  We propose that, in addition to
  the basic engine script, an equivalent basic Perl script be defined
  to perform the Pilot Project pipeline steps.
\end{enumerate}

With these suggestions, we feel the IPP.c engine becomes a simple,
easily understood tool for linking the modules together, without
spending vast amounts of effort on a completely general scripting
language.  In addition, the modules are sufficiently well-defined with
the data well-encapsulated that they can be easily used within a
variety of environments.  The demonstration of a Perl implementation
will show the use of at least one high-level language as a scripting
language.

\milsubsection{Rough Example Code}

I have made a quick and dirty modification to the existing IPP modules
and engine to implement the changes we propose above, as an example /
demonstration.  The code can be found on our website at:

{\tt http://panstarrs.ifa.hawaii.edu/project/people/magnier/IPPx} 

(there is also a tar-gzipped copy in the {\tt magnier} directory).
The example is based on the original release of the Pilot Project
Phase I code.

I stripped out the dataDescrition components and all of the data-level
iteration section in the callModule function.  I also wrapped the
module core operations in APIs which can be more conveniently swigged.
For the readFITS, writeFITS, and debias modules, I have defined Cell
and OTA level operations.  Rather than implement the module level
iterations for debias, I have limited the engine calls to the
Cell-level module APIs for debias, readFITS, and writeFITS.  By
limiting the data to Cells (ignoring the OTA level data), I have been
able to compile and run these three modules successfully together.
The text configuration file is in src/engine/example.  The other two
modules would require a bit more work to be sure the interaction is at
the right data level.

There were a few places where the header and library dependencies were
confusing to me, so I did a small amount of reorganization just to get
things clear for me.  This should not be taken as an organization
suggestion, just something that helped me understand the difference
between the module, data, and engine API pieces.  It is probably not
ideal in terms of file organization.  I also had to alter the
Makefiles somewhat mostly because of the header dependencies.

I have done some of the work on SWIG examples.  There are *.i files
for debias, fits, and divideByFlat.  I have successfully run SWIG for
TCL and compiled the results for the debias module.  The
debias/Makefile contains the relevant steps.  I have not attempted to
write a TCL script using the resulting module, but I have successfully
run TCL with SWIG for sample code.  There are also apparently
configuration problems with Perl on my laptop and the IfA poi cluster,
so I have not demonstrated the Perl implementation at this time for
IPPx.  SWIG / Perl interactions have been successfully demonstrated
using Perl compiled without multithread support.

\milsection{Data Throughput Calculations}
\label{datarate}

The following data throughput discussion is cut from an email by Josh
Hoblitt.

We can do some quick 'back of the envelope' estimates to figure out
the worst case bandwidth needs.  Let's assume that control messages
and db traffic is negligible compared to the image data.  Let's also
assume that there isn't even trivial data locality optimization {\em
and} we have to reload all calibration data for every exposure.  That
gives us:

\begin{verbatim}
 4 fully populated focal planes

     2bytes * ( 4096^2pixels * 1.125overclocks ) * 240otas = 72477573120b

 slew/settle/integrate/read cadence of 30s

     72477573120b / 30s = 2415919104b/s

 exposure data stored as float

     4bytes * 4096^2pixels * 240otas = 128849018880b

     128849018880b / 30s = 4294967296b/s

 best/working stacked stored as float

     4bytes * 4096^2pixels * 60otas = 32212254720

     32212254720b / 30s = 1073741824b/s

 debias, dark, flat, 2x fringe, and 2x sky calibration frames

     7 * 4294967296b/s = 30064771072b/s

 Which gives us a grand total of:

 [phase 2]
      2415919104b/s  summit -> disk (exposure)
 +    2415919104b/s  non-local disk -> memory (exposure)
 +   30064771072b/s  non-local disk -> memory (calibration)
 +    4294967296b/s  memory -> non-local disk (reduced)
 ------------------
  39,191,576,576b/s  phase 2 total

 [phase 4]
      4294967296b/s  non-local disk -> memory (reduced)
 +    1073741824b/s  non-local disk -> memory (best)
 +    1073741824b/s  non-local disk -> memory (working)
 +    1073741824b/s  memory -> non-local disk (diff)
 +    1073741824b/s  memory -> non-local disk (working)
 ------------------
   8,589,934,592b/s  phase 4 total

 [total]
     39191576576b/s  phase 2 total
 +    8589934592b/s  phase 4 total
 ------------------
  47,781,511,168b/s  total is ~48Gb/s
\end{verbatim}

 48Gb/s is 1/15th of the bandwidth available in a 65xx/SUP720.  Given
 these numbers I don't think that engineering effort to optimize for a
 hierarchical network topology is justifiable.

 To address your concerns about the backplane numbers being accurate,
 I can tell you that I have a significant amount of experience with
 enterprise class switches from Cisco, Foundry, and HP.  I've always
 found the switching fabric specs to be accurate from all three of
 these manufactures.  In addition, as soon as I have equipment
 available for use by Pan-Starrs I will generate my own bandwidth
 tests.  However it should be noted that as demonstrated above that
 even our worst case bandwidth needs are rather modest compared to
 COTS equipment that is available today.

\end{document}
