Index: /trunk/doc/pilot/engine/engine_notes.txt
===================================================================
--- /trunk/doc/pilot/engine/engine_notes.txt	(revision 4151)
+++ /trunk/doc/pilot/engine/engine_notes.txt	(revision 4151)
@@ -0,0 +1,77 @@
+
+Evaluation of the MHPCC Engine (IPP.c)
+
+- for every module & for the engine itself, there are a set of
+  functions which may be unique to that component.  These include:
+  malloc, free, warning handler, addDescriptor, addDataItem,
+  deleteDataItem, and getDataItem.  
+
+  It is very unclear why this level of flexibility and abstraction is
+  helpful or useful.  The only one of these which is obvious to
+  encapsulate with the module is the warning handler.  Clearly, it is
+  important to the system, IPP.c, to define its unique value of malloc
+  and free, but why should this be allowed to vary for each module?
+  Also, why should the methods to interact with the I/O data
+  structures be allowed to vary from module to module.  This
+  implementations seems to add a layer of complexity that is not
+  justified.  
+
+- a module is invoked in the config file like this:
+
+moduleName    par1=value1 par2=value2 par3=value3;
+
+  there is no data typing at the script level; typing is implicit in
+  the module's usage of a parameter name.
+
+- two IPP data structures carry the information about the modules:
+  char *moduleNames and DataItemList *params.  The list 'params'
+  contains the par=value entries is a set of data structures with name
+  and value pairs.  When the parameter list is parsed, the types are
+  identified (int/float/string) and cast to those value types in the
+  dataItem structure.  An additional data type is available to the
+  module programmer, cell.  A cell data type may be created in the
+  script language by a call to a module which returns a cell-type,
+  such as readFITS.
+
+- there is also the global parameter data structure which acts like an
+  environment variable stack: these are always passed to the modules. 
+
+- there is an IPP variable stack 'dataSpace' which contains data
+  currently in use by IPP.  Modules share data through the dataSpace
+  stack, either leaving output data on the stack or pulling required
+  data from the stack.
+
+- basic steps of IPP.c:
+
+  - define engine functions (initEngine) 
+    this sets the values of the abstracted engine functions discussed
+    above.
+  - parse config file (ParseConfigurationFile)
+    this reads the config file and sets up the moduleName/params
+    structures.
+  - load the identified modules (a loop in main)
+    this calls 'loadModule' for each module identified in the config
+    file, which in turn creates a Module structure for the module,
+    loads the library file associated with the named module, assigns
+    the parameter list to the module structure, and calls the module's
+    init function.  The init function makes the association between
+    module parameters and module data descriptors.  Note that not all
+    parameters are associated with a data descriptor; the module
+    programmer may choose this association on whatever basis.
+  - execute each module (a loop in main)
+
+    * determine the module type: this simply loops over the module
+      data descriptors and identifies the data type, but gives an
+      error if a module defines multiple data types.
+
+    * identify the required data in the dataSpace.  this section will
+      automatically create a MAJOR_FRAME data item in the dataSpace if
+      required.  ** a strange choice here **
+
+    * call the module 
+      callModule iterates through data type levels to match the given
+      data entry with the given module data level. 
+
+    * free temporary data
+
+    
Index: /trunk/doc/pilot/engine/pilot-engine-prop.tex
===================================================================
--- /trunk/doc/pilot/engine/pilot-engine-prop.tex	(revision 4151)
+++ /trunk/doc/pilot/engine/pilot-engine-prop.tex	(revision 4151)
@@ -0,0 +1,408 @@
+\documentclass[panstarrs]{panstarrs}
+\newcommand\note[1]{{\bf #1}}
+\newcommand\tbd[1]{{\bf [#1]}}
+
+\title{Proposal for the Pilot Project Engine / Architecture}
+\author{Eugene Magnier}
+\group{}
+\organization{}
+\docnumber{PSDC-xxx-xx-01}
+
+\fancyhead{}
+\fancyhead[L]{IPP.c Architecture Proposal }
+\fancyhead[R]{}
+\fancyhead[C]{Revision 01 -- Draft}
+\fancyfoot[L]{}
+\fancyfoot[R]{\today}
+
+\begin{document}
+
+\pagenumbering{roman} \thispagestyle{empty} \maketitle
+
+\pagebreak \pagestyle{fancy}
+{ \Large \bf Revision History}
+\begin{center}
+\begin{tabular}{|p{1.25in}|p{1in}|p{4in}|}
+\hline
+{\bf Revision Number} & {\bf Release Date} & {\bf Description} \\
+\hline
+01 & 2003/12/29 & First draft \\
+02 & 2004/01/06 & Update based on IfA comments \\
+\hline
+\end{tabular}
+\end{center}
+
+\pagebreak
+\tableofcontents
+
+\pagebreak
+\listoffigures
+
+\pagebreak \pagenumbering{arabic}
+
+The IPP Engine initially produced by MHPCC, implemented in IPP.c along
+with supporting code in the engine/dataAPI and the module/moduleAPI
+directories, provides a method to connect the low-level functional
+modules together on the basis of a simple scripting language.  This
+document attempts to evaluate this engine on the basis of the needs,
+expectations, and biases of the IfA team, and to propose modifications
+which will adapt the architecture in ways more in line with our
+desires.  This document first discusses the current implementation,
+second outlines our (IfA) concerns and thoughts about this
+implementations, and finally makes a proposal for future
+implementation.
+
+\milsection{Engine Description}
+
+The Pilot Project implements low-level modules and the engine, which
+connects the modules together.  A module performs a specific,
+well-defined task, such as reading in an image from disk to a data
+structure, correcting for a bias, or applying a flat-field.  The
+engine allows these modules to be connected together in nearly
+arbitrary ways.  
+
+For every module and for the engine itself, there are a set of
+functions which may be unique to that component.  These include: {\tt
+malloc, free, warning handler, addDescriptor, addDataItem,
+deleteDataItem}, and {\tt getDataItem}.  
+
+A module is invoked in the config file like this:
+\begin{verbatim}
+moduleName    par1=value1 par2=value2 par3=value3;
+\end{verbatim}
+
+There is no data typing at the script level; type assignement is
+performed in the module and in the engine, and depends on the usage of
+parameter names.
+
+Two IPP data structures initially carry the information about the
+modules: {\tt char *moduleNames} and {\tt DataItemList *params}.  The
+list 'params' contains the par=value entries in a linked list of data
+structures with name and value pairs elements.  When the parameter
+list is parsed, the types are identified (int/float/string) and cast
+to those value types in the dataItem structure.  An additional data
+type is available to the module programmar, cell.  A cell data type
+may be created in the script language by a call to a module which
+returns a cell-type, such as readFITS.
+
+There is also the {\tt global} parameter data structure which acts
+like an environment variable stack: the complete list is always passed
+to the modules.
+
+There is an IPP variable stack 'dataSpace' which contains data
+currently in use by IPP.  Modules share data through the dataSpace
+stack, either leaving output data on the stack or pulling required
+data from the stack, though in the current implementation, the engine
+layer (IPP.c) mediates the actual interaction with dataSpace.
+
+\milsubsection{Basic Steps of IPP.c}
+
+The top-level of the IPP program performs the following operations:
+
+\begin{itemize}
+
+\item {\bf Define engine functions (initEngine) } This sets the values
+    of the abstracted engine functions discussed above.
+
+\item {\bf Parse configuration file (ParseConfigurationFile) } This
+    reads the config file and sets up the moduleName/params
+    structures.
+
+\item {\bf Load the identified modules (a loop in main) } This calls
+    {\tt loadModule} for each module identified in the config file,
+    which in turn creates a Module structure for the module, loads the
+    library file associated with the named module, assigns the
+    parameter list to the module structure, and calls the module's
+    init function.  The init function makes the association between
+    module parameters and module data descriptors.  Note that not all
+    parameters are associated with a data descriptor; the module
+    programmar may choose this association on whatever basis.
+
+\item {\bf Execute each module (a loop in main)}  The execution of the
+  module includes the following steps:
+
+  \begin{itemize}
+    \item {\bf Determine the module type} This simply loops over the
+      module data descriptors and identifies the data type, but gives
+      an error if a module defines multiple data types.
+
+    \item {\bf Identify the required data in the dataSpace}  This
+      section will automatically create a MAJOR\_FRAME data item in the
+      dataSpace if required. 
+
+    \item {\bf Call the module} In this step, callModule iterates
+      through data type levels to match the given data entry with the
+      given module data level.
+
+    \item {\bf free temporary data}
+  \end{itemize}
+\end{itemize}
+
+\milsection{IfA Comments and Concerns}
+
+The existing IPP.c engine is quite complex, and performs a significant
+fraction of the data manipulation.  Much of the complexity seems to
+stem from the concerns about data I/O demands introduced by the large
+data volume and the need to control aspects of the parallelization
+scheme in great detail.  We do not share these concerns, as discussed
+below, and are instead concerned that the level of complexity involved
+in the current design will result in a high level of development and
+debugging effort which is not justified.  Some of our specific
+concerns follow.
+
+\milsubsection{Excessive Abstraction (engineFcns)}
+
+The use of unique abstracted functions for each module for such basic
+functions as {\tt malloc} and {\tt free} and for the list management
+APIs {\tt addDescriptor, addDataItem, deleteDataItem}, and {\tt
+getDataItem} seems extreme and unneccesary.  
+
+It is very unclear why this level of flexibility and abstraction is
+helpful or useful.  The only one of the engineFcns entries which is
+obvious to associate with the module is the warning handler.  Clearly,
+it is important for the system, IPP.c, to define its unique value of
+malloc and free, but why should this be allowed to vary for each
+module?  Also, why should the methods to interact with the I/O data
+structures be allowed to vary from module to module.  This
+implementations seems to add a layer of complexity that is not
+justified.  These may have been introduced for reasons of
+parallelization, but we believe this is a red herring.
+
+\milsubsection{Obfuscated Data Handling}
+
+There is a surprisingly complex series of steps necessary for a module
+to expose data to other modules, via the IPP.c dataSpace structure.
+
+Consider, for example, the {\tt debias} module, which needs to perform
+its operation on a given image.  It is specified in the configuration
+file as:
+%
+\begin{verbatim}
+debias image=''image'' overscans=32
+\end{verbatim}
+%
+In this syntax, the term {\tt image=''image''} identifies that the
+debias image parameter should be associated with the data space item
+``image''.  In order to provide debias with access to ``image'', the
+following steps take place:
+
+First, the parameters are parsed at the {\tt initModule} stage, and
+the name associated with the parameter {\tt image} is identified.
+This is then pushed on the module descriptor list with its associated
+data type (CELL) and direction (INPUT/OUTPUT).  
+
+Next, when the module is ready to be called, the engine searches the
+module descriptor list for an INPUT or INPUT/OUTPUT entry and finds
+the corresponding data item from the dataSpace, and pushes it on the
+module data space.  (Note that if an entry is output only, the engine
+assumes it to require a MAJOR\_FRAME and creates the data space entry.
+This is an odd choice: why not have the modules responsible for
+creating their own data space entries as needed?)
+
+Finally, back in debias, the module searches the parameter list again
+to identify the image parameter name, then searches the module data
+space for the matching entry.  
+
+This seems far too complex.  Why not simply have a set of APIs to
+access the data space directly?  All of these steps could be reduced
+to two steps:
+
+First, search the parameter list for the desired parameter/name
+match.  Second, request the data value from the engine data space:
+
+\begin{verbatim}
+varName = psDiGetDataItem(*parameters,IMAGE_PARAMETER_NAME);
+if (varName) {
+  imageItem = psGetDataItemfromDataSpace((char*)varName->value);
+}
+\end{verbatim} 
+
+There would be no need to expose the dataSpace variable structure to
+the modules or to the basic engine code.  This would isolate all data
+space access to the modules.  Alternatively, the dataSpace variable
+could be passed to the modules directly, rather than via the
+temporary module data space.  
+
+\milsubsection{Unnecessary Iteration in IPP.c / execute module}
+
+The function {\tt callModule} has a clever layout to make the
+appropriate association between the data type and the module operation
+level.  However, it seems to be far too complex.  We do not see the
+need for this level of complexity in the engine; we would strongly
+prefer that a module have a specified data level of operation (ie, OTA
+or CELL) and the iteration over higher levels be performed explicity
+either at the script level or at the level of the system which invokes
+the engine, while iteration at a lower level be performed within the
+module itself.  We are not convinced that the arguments about
+parallelizing justify the level of complexity in this implementation.
+In addition, the dataDescriptor construction is apparently introduced
+to allow the iteration which takes place in the IPP.c / execute module
+state.  If the engine-level iteration were dropped, then the entire
+dataDescriptor construct could be dropped from the engine.
+
+\milsection{Proposals for the Pilot Project Engine}
+
+We would like to capitalize on the effort expended so far on the
+module and engine development.  There is some concern within the IfA
+that the current design differs far too much from our expectations to
+be of any use.  However, we would like propose here that we try to
+make use of the development so far in a way that will bring the ideas
+of the two communities into closer agreement.  Therefore, we have
+several specific proposals for the IPP.c and for the IPP architectural
+developement generally.
+
+The basic architectural requirements boil down to three concepts:
+First, there must be a scripting language which will allow a
+high-level description of how to link low-level modules together in a
+flexible way.  Second, there must be a way of associating a specific
+script with a given data item (ie, an input image), or a list of data
+items.  Third, there must be a system to organize the execution of
+these operations in parallel for many data items.
+
+The existing engine, IPP.c, performs the first of these tasks.
+Although we have specific concerns with the implementation and perhaps
+the grammar of the scripting language, the general structure defines a
+language which allows the modules to be linked together.  
+
+There is a strong concern at the IfA about investing more than a very
+minor effort to create a linking language from scratch, for either the
+Pilot Project or for the IPP in general.  We feel that such an effort
+is likely to waste a substantial amount of time and to have
+significant risk of bugs and errors.  The IfA would prefer to have the
+real functionality of the modules directly accessible.  
+
+Although the existing module/engine structure makes it rather
+difficult, it is easy to envision modules which are flexible enough to
+allows us to implement test analysis runs for each of these three
+philosophical approaches.  With some minor modification, the existing
+IPP structures would make such tests easy and feasible as well.  We
+would like to make the following proposals for the Pilot Project
+engine / module architecture.
+
+\begin{enumerate}
+
+\item {\bf ignore parallel optimization} Until it is demonstrated
+  otherwise, we feel that the addition of large amounts of complexity
+  to the engine only for the goal of enabling fancy parallel
+  processing optimizations is a distraction and a waste of time.  We
+  believe that all of the parallelization can be performed on very
+  large data chunks (ie, OTAs), and that these operations are
+  essentially independent.  Therefore, no inter-module communication
+  is necessary, and each phase (in the sense defined by the complete
+  IPP design) can be executed as a single command.  Therefore, only
+  scripting sufficient to link modules together in a linear fashion
+  for a single data instance is necessary.  If it is ultimately
+  necessary to add module-level intercommunication to ease the
+  handling of the parallel processing problems, this can be done
+  within the modules or with blocking in the scripting language at
+  that time.  We believe this stance is strongly supported by the
+  switch tests and data I/O rates as examined by Josh Hobblit
+  \tbd{REF}.
+
+\item {\bf provide direct module APIs without engine wrapping}.  The
+  current module APIs are completely flat and uniform.  While this has
+  some theoretical appeal, it requires the use of the engine
+  data API to interact with the modules.  This essentially excludes
+  the use of simple C or SWIG'ed Perl code.  In our proposal, each
+  module would have an API (or multiple APIs as needed) which
+  correspond to their function.  For example, the debias module should
+  have the debias functionality accessible via a {\tt divideByFlat (OTA
+  *image, OTA *flat);} function call.  Note that this proposal does
+  not imply dropping the existing {\tt invokeModule ();} functions
+  associated with each module; rather, the invokeModule function
+  should wrap the lower-level call.
+
+\item {\bf simplify the data-space API} If the engine is only
+  responsible for scripting, there is no need for such a complex set
+  of interactions to handle data between the modules and the engine.
+  The definition of a simple set of engine data-space APIs would allow
+  the modules to gain access to the data structures known by other
+  modules without so many layers of complexity.
+
+\item {\bf simplify module API} If the modules interact via a simple
+  data space API layer, then the module API need only consist of the
+  module parameters.  The modules should make their own decisions
+  about data typing issues and should simply return errors if the
+  script programmars attempt to pass the wrong data types to a module.
+
+\item {\bf remove engine-level iteration / implement module iteration}
+  The modules should have a defined level or set of levels of
+  operation.  We would like to see the modules all operating at the
+  OTA level, since that is the proposed parallelization quantum.  For
+  example, the debias module should receive the OTA structure and
+  performe the iteration if necessary.  This is already implemented by
+  readFITS.c 
+
+\item {\bf freeze grammar} We would like to freeze the IPP.c grammar
+  for the purpose of the Pilot Project.  Although the current grammar
+  is very simple, and minor changes could make the grammar much more
+  sophisticated, what has been delivered will suffice for the Pilot
+  Project.  Since this is potentially an area where far more
+  development time may be spent than is really necessary, we would
+  like to avoid extending the range of the grammar at this time.
+
+\item {\bf implement SWIG interface files}  If the modules have been
+  defined with direct access to the low-level functionality, then it
+  becomes trivial to write SWIG interface files.  Each module should
+  have an associated header file (eg, debias.c gets a debias.h) and an
+  associated interface file (eg, debias.i) which is used to generate
+  the SWIG wrapper file (eg, debias\_wrap.c).  
+
+\item {\bf implement sample Perl script using SWIG}.  With the defined
+  SWIG interface files, it is simple to generate wrapper files for the
+  functions specific to one of the languages supported by SWIG.  The
+  modules can then be linked together in a script within that language
+  by using calls to those modules.  We propose that, in addition to
+  the basic engine script, an equivalent basic Perl script be defined
+  to perform the Pilot Project pipeline steps.
+\end{enumerate}
+
+With these suggestions, we feel the IPP.c engine becomes a simple,
+easily understood tool for linking the modules together, without
+spending vast amounts of effort on a completely general scripting
+language.  In addition, the modules are sufficiently well-defined with
+the data well-encapsulated that they can be easily used within a
+variety of environments.  The demonstration of a Perl implementation
+will show the use of at least one high-level language as a scripting
+language.
+
+\misubsection{Rough Example Code}
+
+I have made a quick and dirty modification to the existing IPP modules
+and engine to implement the changes we propose above, as an example /
+demonstration.  The code can be found on our website at {\tt
+http://panstarrs.ifa.hawaii.edu/project/people/magnier/IPPx} (there is
+also a tar-gzipped copy in the magnier directory).  The example is
+based on the original release of the Pilot Project Phase I code.
+
+I stripped out the dataDescrition components and all of the data-level
+iteration section in the callModule function.  I also wrapped the
+module core operations in APIs which can be more conveniently swigged.
+For the readFITS, writeFITS, and debias modules, I have defined Cell
+and OTA level operations.  Rather than implement the module level
+iterations for debias, I have limited the engine calls to the
+Cell-level module APIs for debias, readFITS, and writeFITS.  By
+limiting the data to Cells (ignoring the OTA level data), I have been
+able to compile and run these three modules successfully together.
+The text configuration file is in src/engine/example.  The other two
+modules would require a bit more work to be sure the interaction is at
+the right data level.
+
+There were a few places were the header and library dependencies were
+confusing to me, so I did a small amount of reorganization just to get
+things clear for me.  This should not be taken as an organization
+suggestion, just something that helped me understand the difference
+between the module, data, and engine API pieces.  It is probably not
+ideal in terms of file organization.  I also had to hack the Makefiles
+somewhat mostly because of the header dependencies.
+
+I have done some of the work on SWIG examples.  There are *.i files
+for debias, fits, and divideByFlat.  I have successfully run SWIG for
+TCL and compiled the results for the debias module.  The
+debias/Makefile contains the relevant steps.  I have not attempted to
+write a TCL script using the resulting module.  There are also
+apparently configuration problems with Perl on my laptop and the IfA
+poi cluster, so I have not demonstrated the Perl implementation at
+this time.  
+
+\end{document}