Index: /trunk/doc/pilot/engine/PSDC-400-004-00.tex =================================================================== --- /trunk/doc/pilot/engine/PSDC-400-004-00.tex (revision 407) +++ /trunk/doc/pilot/engine/PSDC-400-004-00.tex (revision 407) @@ -0,0 +1,499 @@ +\documentclass[panstarrs]{panstarrs} +\newcommand\note[1]{{\bf #1}} +\newcommand\tbd[1]{{\bf [#1]}} + +\title{Proposal for the Pilot Project Engine / Architecture} +\shorttitle{IPP.c Architecture Proposal} +\project{Pan-STARRS Pilot Project} +\author{Eugene Magnier} +\group{} +\organization{} +\docnumber{PSDC-400-004-00} + +\fancyhead{} +\fancyhead[L]{\theshorttitle} +\fancyhead[C]{} +\fancyhead[R]{\thedocnumber} +\fancyfoot[L]{\theproject} +\fancyfoot[R]{\today} + +\begin{document} + +%% title page +\pagenumbering{roman} \thispagestyle{empty} \maketitle +\pagebreak + +%% revisions page +\pagestyle{fancy} +{ \Large \bf Revision History} +\begin{center} +\begin{tabular}{|p{1.25in}|p{1in}|p{4in}|} +\hline +{\bf Revision Number} & {\bf Release Date} & {\bf Description} \\ +\hline +00 & 2004/01/08 & Proposal \\ +\hline +\end{tabular} +\end{center} +\pagebreak + +%% contents page +\tableofcontents +\pagebreak + +%% figures page +\listoffigures +\pagebreak \pagenumbering{arabic} + +%%%%% start of main text + +The IPP Engine initially produced by MHPCC, implemented in IPP.c along +with supporting code in the engine/dataAPI and the module/moduleAPI +directories, provides a method to connect the low-level functional +modules together on the basis of a simple scripting language. This +document attempts to evaluate this engine on the basis of the needs, +expectations, and biases of the IfA team, and to propose modifications +which will adapt the architecture in ways more in line with our +desires. This document first discusses the current implementation, +second outlines our (IfA) concerns and thoughts about this +implementations, and finally makes a proposal for future +implementation. + +\milsection{Engine Description} + +The Pilot Project implements low-level modules and the engine, which +connects the modules together. A module performs a specific, +well-defined task, such as reading in an image from disk to a data +structure, correcting for a bias, or applying a flat-field. The +engine allows these modules to be connected together in nearly +arbitrary ways. + +For every module and for the engine itself, there are a set of +functions which may be unique to that component. These include: {\tt +malloc, free, warning handler, addDescriptor, addDataItem, +deleteDataItem}, and {\tt getDataItem}. + +A module is invoked in the config file like this: +\begin{verbatim} +moduleName par1=value1 par2=value2 par3=value3; +\end{verbatim} + +There is no data typing at the script level; type assignement is +performed in the module and in the engine, and depends on the usage of +parameter names. + +Two IPP data structures initially carry the information about the +modules: {\tt char *moduleNames} and {\tt DataItemList *params}. The +list 'params' contains the par=value entries in a linked list of data +structures with name and value pairs elements. When the parameter +list is parsed, the types are identified (int/float/string) and cast +to those value types in the dataItem structure. An additional data +type is available to the module programmer, cell. A cell data type +may be created in the script language by a call to a module which +returns a cell-type, such as readFITS. + +There is also the {\tt global} parameter data structure which acts +like an environment variable stack: the complete list is always passed +to the modules. + +There is an IPP variable stack 'dataSpace' which contains data +currently in use by IPP. Modules share data through the dataSpace +stack, either leaving output data on the stack or pulling required +data from the stack, though in the current implementation, the engine +layer (IPP.c) mediates the actual interaction with dataSpace. + +\milsubsection{Basic Steps of IPP.c} + +The top-level of the IPP program performs the following operations: + +\begin{itemize} + +\item {\bf Define engine functions (initEngine) } This sets the values + of the abstracted engine functions discussed above. + +\item {\bf Parse configuration file (ParseConfigurationFile) } This + reads the config file and sets up the moduleName/params + structures. + +\item {\bf Load the identified modules (a loop in main) } This calls + {\tt loadModule} for each module identified in the config file, + which in turn creates a Module structure for the module, loads the + library file associated with the named module, assigns the + parameter list to the module structure, and calls the module's + init function. The init function makes the association between + module parameters and module data descriptors. Note that not all + parameters are associated with a data descriptor; the module + programmer may choose this association on whatever basis. + +\item {\bf Execute each module (a loop in main)} The execution of the + module includes the following steps: + + \begin{itemize} + \item {\bf Determine the module type} This simply loops over the + module data descriptors and identifies the data type, but gives + an error if a module defines multiple data types. + + \item {\bf Identify the required data in the dataSpace} This + section will automatically create a MAJOR\_FRAME data item in the + dataSpace if required. + + \item {\bf Call the module} In this step, callModule iterates + through data type levels to match the given data entry with the + given module data level. + + \item {\bf free temporary data} + \end{itemize} +\end{itemize} + +\milsection{IfA Comments and Concerns} + +The existing IPP.c engine is quite complex, and performs a significant +fraction of the data manipulation. Much of the complexity seems to +stem from the concerns about data I/O demands introduced by the large +data volume and the need to control aspects of the parallelization +scheme in great detail. Aside from the point that parallization is +outside the scope of the Pilot Project, we do not share these I/O +throughput concerns, as discussed below, and are instead concerned +that the level of complexity involved in the current design will +result in a high level of development and debugging effort which is +not justified. Some of our specific concerns follow. + +\milsubsection{Excessive Abstraction (engineFcns)} + +The use of unique abstracted functions for each module for such basic +functions as {\tt malloc} and {\tt free} and for the list management +APIs {\tt addDescriptor, addDataItem, deleteDataItem}, and {\tt +getDataItem} seems extreme and unneccesary. + +It is very unclear why this level of flexibility and abstraction is +helpful or useful. The only one of the engineFcns entries which is +obvious to associate with the module is the warning handler. Clearly, +it is important for the system, IPP.c, to define its unique value of +malloc and free, but why should this be allowed to vary for each +module? Also, why should the methods to interact with the I/O data +structures be allowed to vary from module to module. This +implementations seems to add a layer of complexity that is not +justified. These may have been introduced for reasons of +parallelization, but we believe this is a red herring. + +\milsubsection{Obfuscated Data Handling} + +There is a surprisingly complex series of steps necessary for a module +to expose data to other modules, via the IPP.c dataSpace structure. + +Consider, for example, the {\tt debias} module, which needs to perform +its operation on a given image. It is specified in the configuration +file as: +% +\begin{verbatim} +debias image="myImage" overscans=32 +\end{verbatim} +% +In this syntax, the term {\tt image=''myImage''} identifies that the +debias image parameter should be associated with the data space item +{\tt myImage}. In order to provide debias with access to {\tt +myImage}, the following steps take place: + +First, the parameters are parsed at the {\tt initModule} stage, and +the name associated with the parameter {\tt image} is identified. +This is then pushed on the module descriptor list with its associated +data type (CELL) and direction (INPUT/OUTPUT). + +Next, when the module is ready to be called, the engine searches the +module descriptor list for an INPUT or INPUT/OUTPUT entry and finds +the corresponding data item from the dataSpace, and pushes it on the +module data space. (Note that if an entry is output only, the engine +assumes it to require a MAJOR\_FRAME and creates the data space entry. +This is an odd choice: why not have the modules responsible for +creating their own data space entries as needed?) + +Finally, back in debias, the module searches the parameter list again +to identify the image parameter name, then searches the module data +space for the matching entry. + +This seems far too complex. Why not simply have a set of APIs to +access the data space directly? All of these steps could be reduced +to two steps: + +First, search the parameter list for the desired parameter/name +match. Second, request the data value from the engine data space: + +\begin{verbatim} +varName = psDiGetDataItem(*parameters,IMAGE_PARAMETER_NAME); +if (varName) { + imageItem = psGetDataItemfromDataSpace((char*)varName->value); +} +\end{verbatim} + +There would be no need to expose the dataSpace variable structure to +the modules or to the basic engine code. This would isolate all data +space access to the modules. Alternatively, the dataSpace variable +could be passed to the modules directly, rather than via the +temporary module data space. + +\milsubsection{Unnecessary Iteration in IPP.c / execute module} + +The function {\tt callModule} has a clever layout to make the +appropriate association between the data type and the module operation +level. However, it seems to be far too complex. We do not see the +need for this level of complexity in the engine; we would strongly +prefer that a module have a specified data level of operation (ie, OTA +or CELL) and the iteration over higher levels be performed explicity +either at the script level or at the level of the system which invokes +the engine, while iteration at a lower level be performed within the +module itself. We are not convinced that the arguments about +parallelizing justify the level of complexity in this implementation. +In addition, the dataDescriptor construction is apparently introduced +to allow the iteration which takes place in the IPP.c / execute module +state. If the engine-level iteration were dropped, then the entire +dataDescriptor construct could be dropped from the engine. + +\milsection{Proposals for the Pilot Project Engine} + +We would like to capitalize on the effort expended so far on the +module and engine development. There is some concern within the IfA +that the current design differs far too much from our expectations to +be of any use. However, we would like propose here that we try to +make use of the development so far in a way that will bring the ideas +of the two communities into closer agreement. Therefore, we have +several specific proposals for the IPP.c and for the IPP architectural +developement generally. + +The basic architectural requirements boil down to three concepts: +First, there must be a scripting language which will allow a +high-level description of how to link low-level modules together in a +flexible way. Second, there must be a way of associating a specific +script with a given data item (ie, an input image), or a list of data +items. Third, there must be a system to organize the execution of +these operations in parallel for many data items. + +The existing engine, IPP.c, performs the first of these tasks. +Although we have specific concerns with the implementation and perhaps +the grammar of the scripting language, the general structure defines a +language which allows the modules to be linked together. + +There is a strong concern at the IfA about investing more than a very +minor effort to create a linking language from scratch, for either the +Pilot Project or for the IPP in general. We feel that such an effort +is likely to waste a substantial amount of time and to have +significant risk of bugs and errors. The IfA would prefer to have the +real functionality of the modules directly accessible. + +Although the existing module/engine structure makes it rather +difficult, it is easy to envision modules which are flexible enough to +allows us to implement test analysis runs for each of these three +philosophical approaches. With some minor modification, the existing +IPP structures would make such tests easy and feasible as well. We +would like to make the following proposals for the Pilot Project +engine / module architecture. + +\begin{enumerate} + +\item {\bf ignore parallel optimization} Until it is demonstrated + otherwise, we feel that the addition of large amounts of complexity + to the engine only for the goal of enabling fancy parallel + processing optimizations is a distraction and a waste of time. We + believe that all of the parallelization can be performed on very + large data chunks (ie, OTAs), and that these operations are + essentially independent. Therefore, no inter-module communication + is necessary, and each phase (in the sense defined by the complete + IPP design) can be executed as a single command. Therefore, only + scripting sufficient to link modules together in a linear fashion + for a single data instance is necessary. If it is ultimately + necessary to add module-level intercommunication to ease the + handling of the parallel processing problems, this can be done + within the modules or with blocking in the scripting language at + that time. We believe this stance is strongly supported by the + switch tests and data I/O rates as examined by Josh Hobblit (see + section~\ref{datarate}). + +\item {\bf provide direct module APIs without engine wrapping}. The + current module APIs are completely flat and uniform. While this has + some theoretical appeal, it requires the use of the engine + data API to interact with the modules. This essentially excludes + the use of simple C or SWIG'ed Perl code. In our proposal, each + module would have an API (or multiple APIs as needed) which + correspond to their function. For example, the debias module should + have the debias functionality accessible via a {\tt divideByFlat (OTA + *image, OTA *flat);} function call. Note that this proposal does + not imply dropping the existing {\tt invokeModule ();} functions + associated with each module; rather, the invokeModule function + should wrap the lower-level call. + +\item {\bf simplify the data-space API} If the engine is only + responsible for scripting, there is no need for such a complex set + of interactions to handle data between the modules and the engine. + The definition of a simple set of engine data-space APIs would allow + the modules to gain access to the data structures known by other + modules without so many layers of complexity. + +\item {\bf simplify module API} If the modules interact via a simple + data space API layer, then the module API need only consist of the + module parameters. The modules should make their own decisions + about data typing issues and should simply return errors if the + script programmers attempt to pass the wrong data types to a module. + +\item {\bf remove engine-level iteration / implement module iteration} + The modules should have a defined level or set of levels of + operation. We would like to see the modules all operating at the + OTA level, since that is the proposed parallelization quantum. For + example, the debias module should receive the OTA structure and + performe the iteration if necessary. This is already implemented by + readFITS.c + +\item {\bf freeze grammar} We would like to freeze the IPP.c grammar + for the purpose of the Pilot Project. Although the current grammar + is very simple, and minor changes could make the grammar much more + sophisticated, what has been delivered will suffice for the Pilot + Project. Since this is potentially an area where far more + development time may be spent than is really necessary, we would + like to avoid extending the range of the grammar at this time. + +\item {\bf implement SWIG interface files} If the modules have been + defined with direct access to the low-level functionality, then it + becomes trivial to write SWIG interface files. Each module should + have an associated header file (eg, debias.c gets a debias.h) and an + associated interface file (eg, debias.i) which is used to generate + the SWIG wrapper file (eg, debias\_wrap.c). + +\item {\bf implement sample Perl script using SWIG}. With the defined + SWIG interface files, it is simple to generate wrapper files for the + functions specific to one of the languages supported by SWIG. The + modules can then be linked together in a script within that language + by using calls to those modules. We propose that, in addition to + the basic engine script, an equivalent basic Perl script be defined + to perform the Pilot Project pipeline steps. +\end{enumerate} + +With these suggestions, we feel the IPP.c engine becomes a simple, +easily understood tool for linking the modules together, without +spending vast amounts of effort on a completely general scripting +language. In addition, the modules are sufficiently well-defined with +the data well-encapsulated that they can be easily used within a +variety of environments. The demonstration of a Perl implementation +will show the use of at least one high-level language as a scripting +language. + +\milsubsection{Rough Example Code} + +I have made a quick and dirty modification to the existing IPP modules +and engine to implement the changes we propose above, as an example / +demonstration. The code can be found on our website at: + +{\tt http://panstarrs.ifa.hawaii.edu/project/people/magnier/IPPx} + +(there is also a tar-gzipped copy in the {\tt magnier} directory). +The example is based on the original release of the Pilot Project +Phase I code. + +I stripped out the dataDescrition components and all of the data-level +iteration section in the callModule function. I also wrapped the +module core operations in APIs which can be more conveniently swigged. +For the readFITS, writeFITS, and debias modules, I have defined Cell +and OTA level operations. Rather than implement the module level +iterations for debias, I have limited the engine calls to the +Cell-level module APIs for debias, readFITS, and writeFITS. By +limiting the data to Cells (ignoring the OTA level data), I have been +able to compile and run these three modules successfully together. +The text configuration file is in src/engine/example. The other two +modules would require a bit more work to be sure the interaction is at +the right data level. + +There were a few places where the header and library dependencies were +confusing to me, so I did a small amount of reorganization just to get +things clear for me. This should not be taken as an organization +suggestion, just something that helped me understand the difference +between the module, data, and engine API pieces. It is probably not +ideal in terms of file organization. I also had to alter the +Makefiles somewhat mostly because of the header dependencies. + +I have done some of the work on SWIG examples. There are *.i files +for debias, fits, and divideByFlat. I have successfully run SWIG for +TCL and compiled the results for the debias module. The +debias/Makefile contains the relevant steps. I have not attempted to +write a TCL script using the resulting module, but I have successfully +run TCL with SWIG for sample code. There are also apparently +configuration problems with Perl on my laptop and the IfA poi cluster, +so I have not demonstrated the Perl implementation at this time for +IPPx. SWIG / Perl interactions have been successfully demonstrated +using Perl compiled without multithread support. + +\milsection{Data Throughput Calculations} +\label{datarate} + +The following data throughput discussion is cut from an email by Josh +Hoblitt. + +We can do some quick 'back of the envelope' estimates to figure out +the worst case bandwidth needs. Let's assume that control messages +and db traffic is negligible compared to the image data. Let's also +assume that there isn't even trivial data locality optimization {\em +and} we have to reload all calibration data for every exposure. That +gives us: + +\begin{verbatim} + 4 fully populated focal planes + + 2bytes * ( 4096^2pixels * 1.125overclocks ) * 240otas = 72477573120b + + slew/settle/integrate/read cadence of 30s + + 72477573120b / 30s = 2415919104b/s + + exposure data stored as float + + 4bytes * 4096^2pixels * 240otas = 128849018880b + + 128849018880b / 30s = 4294967296b/s + + best/working stacked stored as float + + 4bytes * 4096^2pixels * 60otas = 32212254720 + + 32212254720b / 30s = 1073741824b/s + + debias, dark, flat, 2x fringe, and 2x sky calibration frames + + 7 * 4294967296b/s = 30064771072b/s + + Which gives us a grand total of: + + [phase 2] + 2415919104b/s summit -> disk (exposure) + + 2415919104b/s non-local disk -> memory (exposure) + + 30064771072b/s non-local disk -> memory (calibration) + + 4294967296b/s memory -> non-local disk (reduced) + ------------------ + 39,191,576,576b/s phase 2 total + + [phase 4] + 4294967296b/s non-local disk -> memory (reduced) + + 1073741824b/s non-local disk -> memory (best) + + 1073741824b/s non-local disk -> memory (working) + + 1073741824b/s memory -> non-local disk (diff) + + 1073741824b/s memory -> non-local disk (working) + ------------------ + 8,589,934,592b/s phase 4 total + + [total] + 39191576576b/s phase 2 total + + 8589934592b/s phase 4 total + ------------------ + 47,781,511,168b/s total is ~48Gb/s +\end{verbatim} + + 48Gb/s is 1/15th of the bandwidth available in a 65xx/SUP720. Given + these numbers I don't think that engineering effort to optimize for a + hierarchical network topology is justifiable. + + To address your concerns about the backplane numbers being accurate, + I can tell you that I have a significant amount of experience with + enterprise class switches from Cisco, Foundry, and HP. I've always + found the switching fabric specs to be accurate from all three of + these manufactures. In addition, as soon as I have equipment + available for use by Pan-Starrs I will generate my own bandwidth + tests. However it should be noted that as demonstrated above that + even our worst case bandwidth needs are rather modest compared to + COTS equipment that is available today. + +\end{document}