Index: /trunk/Ohana/doc/www/html/DVO/.cvsignore =================================================================== --- /trunk/Ohana/doc/www/html/DVO/.cvsignore (revision 4727) +++ /trunk/Ohana/doc/www/html/DVO/.cvsignore (revision 4727) @@ -0,0 +1,2 @@ +*.html +index.idx Index: /trunk/Ohana/doc/www/html/Elixir-System/index.htm =================================================================== --- /trunk/Ohana/doc/www/html/Elixir-System/index.htm (revision 4727) +++ /trunk/Ohana/doc/www/html/Elixir-System/index.htm (revision 4727) @@ -0,0 +1,15 @@ + + + + +This section describes the Ohana programs used to tie the Elixir +System together. These include + + Index: /trunk/Ohana/doc/www/html/Elixir-System/sequence.idx =================================================================== --- /trunk/Ohana/doc/www/html/Elixir-System/sequence.idx (revision 4727) +++ /trunk/Ohana/doc/www/html/Elixir-System/sequence.idx (revision 4727) @@ -0,0 +1,6 @@ +elixir +mkdetrend +mkfringe +nightd +postrun +gconfig Index: /trunk/Ohana/doc/www/html/Elixir-Tools/.cvsignore =================================================================== --- /trunk/Ohana/doc/www/html/Elixir-Tools/.cvsignore (revision 4727) +++ /trunk/Ohana/doc/www/html/Elixir-Tools/.cvsignore (revision 4727) @@ -0,0 +1,2 @@ +*.html +index.idx Index: /trunk/Ohana/doc/www/html/IPP-subsystems/.cvsignore =================================================================== --- /trunk/Ohana/doc/www/html/IPP-subsystems/.cvsignore (revision 4727) +++ /trunk/Ohana/doc/www/html/IPP-subsystems/.cvsignore (revision 4727) @@ -0,0 +1,2 @@ +*.html +index.idx Index: /trunk/Ohana/doc/www/html/IPP-subsystems/index.htm =================================================================== --- /trunk/Ohana/doc/www/html/IPP-subsystems/index.htm (revision 4727) +++ /trunk/Ohana/doc/www/html/IPP-subsystems/index.htm (revision 4727) @@ -0,0 +1,17 @@ + + + +The Pan-STARRS IPP is using several programs from the Ohana collection +for portions of the IPP Infrastructure. These subsystems include: + + + +In addition to these program, the IPP will use DVO for object +photometry manipulation. IPP requires the upgraded capability which +will be provided by DVO after Fall 2005, including enhanced +throughput. DVO is discussed in its own section. + Index: /trunk/Ohana/doc/www/html/IPP-subsystems/pclient/.cvsignore =================================================================== --- /trunk/Ohana/doc/www/html/IPP-subsystems/pclient/.cvsignore (revision 4727) +++ /trunk/Ohana/doc/www/html/IPP-subsystems/pclient/.cvsignore (revision 4727) @@ -0,0 +1,2 @@ +*.html +index.idx Index: /trunk/Ohana/doc/www/html/IPP-subsystems/pclient/index.htm =================================================================== --- /trunk/Ohana/doc/www/html/IPP-subsystems/pclient/index.htm (revision 4727) +++ /trunk/Ohana/doc/www/html/IPP-subsystems/pclient/index.htm (revision 4727) @@ -0,0 +1,5 @@ + + + + +pclient is the remote process monitor Index: /trunk/Ohana/doc/www/html/IPP-subsystems/pcontrol/.cvsignore =================================================================== --- /trunk/Ohana/doc/www/html/IPP-subsystems/pcontrol/.cvsignore (revision 4727) +++ /trunk/Ohana/doc/www/html/IPP-subsystems/pcontrol/.cvsignore (revision 4727) @@ -0,0 +1,2 @@ +*.html +index.idx Index: /trunk/Ohana/doc/www/html/IPP-subsystems/pcontrol/index.htm =================================================================== --- /trunk/Ohana/doc/www/html/IPP-subsystems/pcontrol/index.htm (revision 4727) +++ /trunk/Ohana/doc/www/html/IPP-subsystems/pcontrol/index.htm (revision 4727) @@ -0,0 +1,240 @@ + + + + +pcontrol is the IPP parallel process controller. + +

Overview

+ +

+The IPP uses a group of computers to store and process images and to +manipulate collections of detections. These computers perform any of +a large number of analysis stages or other processing tasks without +significant interprocess communication. It is necessary to have a +mechanism which initiates computing tasks on the different computers, +which monitors the tasks as they are executed, which handles the +output and the errors from these tasks, and which reacts to the +failure of any of the computing nodes. The system responsible for the +tasks in the IPP is pcontrol. + +

+pcontrol interacts with the collection of computers under its +management and with other subsystems in the IPP. The IPP Controller +receives a variety of inputs from other subsystems, described below, +and initiates actions such as adding a new process to the queue of +pending tasks. pcontrol also provides information to other +subsystems on demand about its processing history and current state. +Each physical computer may have multiple processors; since +pcontrol is managing processing tasks, it treats each +processor independently. It is up to the system configuration if each +computer needs to reserve one of its CPUs to manage background tasks +or if pcontrol should attempt to send one task per CPU and +let the operating system handle the I/O load. + +

hosts

+ +The Controller maintains a table of available processing computers +(hosts) and tracks their status. Hosts managed by +pcontrol are allowed to be in one of several states, and +pcontrol must interact with it in an appropriate way for each +of those states. A Node may be {\tt alive}, {\tt dead} or {\tt off}. +If the Node is {\tt alive}, it responds to commands from the IPP +Controller and may be used for tasks subject to other constraints. If +it is {\tt dead}, the Node is not responsive and must not be used for +executing tasks. pcontrol must identify Nodes which have +died (not responding) and occasionally test them to see if they are +{\tt alive} again. Nodes which are {\tt off} are not available for +tasks and must not be tested. Nodes may be set to the {\tt off} or +{\tt dead} states by external subsystems; it is the responsibility of +pcontrol to return a Node to the {\tt alive} state if +possible. + +pcontrol must honor requests (normally from the users) to +change the mode of any computing node on demand between {\tt off} and +{\tt dead}. This would normally be done after a Node has been +rebooted and is released to pcontrol for its use. It +must also be able to change the list of allowed tasks as requested by +external commands. + +Two example scenarios illustrate the transition between these states, +and the basic concept of operations for pcontrol. First, +imagine a computer crashes. At this point pcontrol should +detect that the Node is no longer responsive and mark it as {\tt +dead}. It should occasionally try to re-establish communication with +the Node, potentially with longer and longer delays between attempts. +A human could be notified if the Node seems to remain {\tt dead} for a +very long time. In another scenario, a person needs to work on a +Node. They notify pcontrol that the machine is {\tt off}, +perhaps with a prior notification that the machine should be prepared +to go off. When work on the machine is complete, it should be placed +in the {\tt dead} state. Only when the person is done working and +testing the machine, and tells pcontrol that the machine is +now {\tt dead} can pcontrol attempt to re-start +communications and re-new processing operations on that Node. + +

pclient

+ +When the Controller starts, it attempts to launch a Node Agent on each +of the available processing Nodes. Nodes which are not responsive are +marked as {\tt dead} so they may be re-tried. A Node Agent runs on +each of the individual nodes to execute the tasks as directed by the +Controller. The Node Agents communicate with the Controller via a +socket connection. + +A Node Agent (which is only running on a Node in the {\tt alive} +state) may be in one of four modes: {\tt idle}, {\tt busy}, {\tt +done}, {\tt crash}. A Node Agent which is {\tt busy} currently has a +task assigned to it which is executing. The pcontrol may only +assign one task to a Node at a time. A Node Agent which is in the +{\tt idle} state may have a task assigned to it. When the Node Agent +detects that a tasks has finished, it changes to either the {\tt done} +or {\tt crash} states depending on the outcome of the process +execution. The pcontrol must also respect a list of task +restrictions which may require specific tasks to run on specific CPUs +or exclude specific tasks from specific CPUs. + +A task being executed by the Node is run in the UNIX user space as a +forked process. The Node Agent must monitor the standard error and +standard output of the executing task and save them in separate +buffers. If the process exits or dies, the Node Agent must detect +this result and change state appropriately. The Node Agent must +respond to various commands from the Controller, as follows: + +\paragraph{Report status} + +The Node Agent returns its state ({\tt idle}, {\tt busy}, {\tt done}, +{\tt crash}) and the exit status of the current processing task, if +available. The reported exit state, if the process has completed +without crashing, is the UNIX exit state reported by the task: 0--256 +with 0 indicating a successful completion. + +\paragraph{Report stdout} + +Send and flush the current stdout buffer. The Node Agent will return +the complete contents of the stdout buffer via a buffered write and +flush the buffer when it is finished. The Node Agent will not accept +more data on the stdout buffer from the current processing task until +the send is complete and the buffer is flushed. The daemon must +accept all of the buffer output. + +\paragraph{Report stderr} + +Identical to `report stdout', but for stderr. + +\paragraph{Kill task } + +The Node Agent should send a kill signal (\code{KILL} or \code{TERM}) +to the current processing task. When the processing task has exited, +the Node Agent should set its state to {\tt crash}. + +\paragraph{Clear task} + +The Node Agent should set its state {\tt idle}. If a processing stage +is currently running, it should be killed (\code{KILL} or \code{TERM}) +before the task is cleared. + +\paragraph{Start processing stage} + +The Node Agent forks a specified command. The command should be a +standard UNIX command without command line redirection or +backgrounding. The task is run with the same user ID as the Node +Agent, which is also the same user ID as the Controller. + +\subsubsection{Tasks} + +The pcontrol accepts tasks from other IPP subsystems. The task +requests include the specific command to be executed and are in the +form of a UNIX command which could be performed on any of the +computing nodes. Any input or output data in the commands must be a +valid resource regardless of the node on which the task is executed. +Input and output data resources must be unique where necessary to +avoid conflicts. It is the responsibility of the task to wait for +network lags (ie, NFS delays). The pcontrol gives each task a +unique identifier, which is returned to the requesting entity. The +requestor may then use that ID to obtain status information on that +task or to send control signals to the specific task. + +Task requests may specify a desired node for the task execution. The +pcontrol attempts to honor the request if the node is {\tt +alive}, but will execute it on another node if the requested one is +{\tt dead} or {\tt off}. Even if a node is {\tt alive}, the IPP +Controller will choose another node if the specified task is not +allowed on the requested node. In all other cases, the pcontrol +waits until the currently executing processes, and processes with +higher priority, are completed before executing the specified task on +the requested node. + +Task requests may specify an urgency level. The pcontrol +determines the priority of the task on the basis of both the urgency +and the age of the request. An executing task must be completed on a +CPU before any new task is started on that CPU, regardless of +priority. The urgency levels range from 0 to 2. Tasks with an +urgency of 1 are scheduled whenever they reach the top of the stack. +Tasks with an urgency of 2 are sent immediately to the top of the +stack. Tasks assigned a priority of 0 are maintained in the queue and +never executed. + +It may be useful for the Controller to distinguish between tasks +dominated by I/O and tasks dominated by data processing. It is +possible that one of each of these types of tasks may be sent to the +same node without significantly impacting the system performance. +Alternatively, it may be necessary to limit a single machine with 2 +CPUs to only one of each of these types of tasks (i.e., one processor +will be working on I/O while the other is working on processing). +Such details will be studied by the IfA IPP Team. + +The pcontrol monitors the output streams from the executing +tasks and the exit status of the tasks. Each task is associated with +a log file, to which all output is written. The status, including the +exit status, of each task is maintained by the pcontrol so that +other subsystems may determine if specific tasks have started or +completed. + +\subsubsection{Controller Interfaces} + +The pcontrol must accept commands from other IPP subsystems. +These commands include those which govern the processing of specified +tasks, those which govern the behavior of specific computing nodes, +and those which request information from the pcontrol. The IPP +Controller must be able to halt the execution of a specified task, +delete an unexecuted task from the task list, change the priority of +tasks, and change the requested nodes for tasks. The pcontrol +must also be able to stop the current execution of a task and push it +to the end of the queue and also change its priority. + +The pcontrol must respond to informational requests regarding the +collection of machines and their states as well as the collection of +tasks and their states. The pcontrol must monitor the execution +times of the different tasks and provide summary statistics. Finally, +the pcontrol must respond to three top-level commands: {\tt finish}, +{\tt stop} and {\tt abort}. When {\tt finish} is requested, no more +new tasks are accepted on the stack of task, and when all tasks in the +stack have completed, the pcontrol must exit. When {\tt stop} is +requested, the currently executing tasks must be completed at which +point the pcontrol must exit, but tasks remaining in the stack which +have not been started are flushed. When {\tt abort} is issued, the +pcontrol immediately kills all executing tasks and exits. + +The pcontrol and the IPP Image Server have related needs for +information from the combined storage-and-processing nodes regarding +which nodes are available. It is not yet clear if this information is +best stored in a single location (either pcontrol or IPP Image +Server), which provides the information to other systems on demand, or +if both systems should maintain the information. Also, it may be +necessary to distinguish nodes which are available for processing from +those that are available to serve data as part of the IPP Image +Server. + +The Controller maintains three tables of processing jobs: pending +stages, active stages, and completed stages. The pending stages are +those which have not yet been performed. The active stages are those +currently being performed on one of the remote nodes. The completed +stages are those which have finished, either successfully or with an +error state. The Controller daemon monitors the collection of remote +clients and sends them new pending stages when they become free. + +The pcontrol provides a mechanism for users (either other +programs or humans) to interact with it. The user interface provides +commands to check the current processing job queues, the tables of +successful and failed jobs, to stop or delete jobs, etc. + Index: /trunk/Ohana/doc/www/html/IPP-subsystems/psched/.cvsignore =================================================================== --- /trunk/Ohana/doc/www/html/IPP-subsystems/psched/.cvsignore (revision 4727) +++ /trunk/Ohana/doc/www/html/IPP-subsystems/psched/.cvsignore (revision 4727) @@ -0,0 +1,2 @@ +*.html +index.idx Index: /trunk/Ohana/doc/www/html/IPP-subsystems/psched/index.htm =================================================================== --- /trunk/Ohana/doc/www/html/IPP-subsystems/psched/index.htm (revision 4727) +++ /trunk/Ohana/doc/www/html/IPP-subsystems/psched/index.htm (revision 4727) @@ -0,0 +1,161 @@ + +This article describes the concept, design, and operation of +psched, the Pan-STARRS IPP task scheduler. + +

Basic Concept

+ +

+ The purpose of psched is to manage the automatic construction + and execution of inter-related (often repetative) operations. + psched uses a set of rules to define UNIX commands, and + their corresponding command-line arguments, to be performed on some + regular, repeated basis. The utility of psched is that it + can easily define an analysis system which is completely + state-based, as opposed to an event-driven system. + +

+ Consider, for example, a telescope which obtains a collection of + images over the course of a night. Every minute or two, it takes an + image and writes the image to some disk. An event-driven analysis + system would involve having the telescope initiate a process at the + end of the exposure. This process would perform an analysis, write + some output, then send trigger another process. This type of + operation works very well for a simple set up with reliable + hardware. Such a system becomes more difficult to maintain when + hardware failures occur or when multiple systems need to interact + with each other. When failures occur, the triggering information + (the events) is easily lost, thus some mechanisms are needed to + detect these failures and either re-send the trigger or send an + alternative failure-mode trigger. Or, if two systems need to + interact, one or the other system must block for results from the + first. Stopping and restarting such an analysis system is very + delicate since the appropriate triggers must be set up some how, eg + by noticing which images have not succeeded and restarting them at + the appropriate stage. All of these types of methods of handling + complexity and failures are essentially state-based rules. + psched allows the easy definition of a totally state-based + analysis system. + +

+ In a state-based system, some mechanism examines the state of the + system and decides which actions to perform based on the current + state. In the illustration above, the mechanism could examine the + images available (either by examining the disk or by examining the + state of a data table) and decide to perform an operation based on + what images are available. This makes it very easy to handle + complexity and errors. If an analysis fails, the state either is + not successfully updated or the error state is recorded, both + situations being easy to detect and easy to handle. Restarting the + system simply involves starting the state-monitoring mechanism. + Combining results from multiple input sources simply involves + watching for the multiple inputs to be available. psched + provides a mechanism to define state monitors, and to define the + actions which are performed when those states occur. + psched action consist of initiating UNIX commands, where + the arguments of those commands may depend on the results of the + state tests. + +

Tasks vs Jobs

+ +

+ The primary function of psched is to repeatedly perform + tasks, and execute jobs on the basis of those tasks. + A task consists of a set of rules which describe system state tests + to perform on a regular time scale. Based on the results of those + state tests, the task will then choose whether or not to construct a + job. The task also defines actions to perform upon the completion + of a job, based upon the output and exit status of the job. A task + thus defines the repeat period. It may optionally define valid or + invalid time ranges (eg, Mon-Fri or 10:00-17:00, etc). The task may + also specify that the job be run locally (ie, in the background on + the same computer as psched) or remotely by the parallel process + controller (pcontrol). A job may even be restricted to a + specific computer managed by pcontrol. + + An example of a simple tasks is given below. + +

+  task datalist
+    command ls /data/foo
+    periods -exec 5.0
+    periods -timeout 50.0
+    periods -poll 1.0
+
+    task.exit 0
+      queueprint stdout
+      queuedelete stdout
+    end
+ 
+    task.exit 1
+      queuepush failure "task failed"
+    end
+  end
+
+ +

+ This task does not perform any system state tests; it is simply + constructs a new job every 5.0 seconds. The job in this case is + always the same: ls /data/foo . When the job finished, + if the job exit status is 0 (normal UNIX success status), the + resulting output is printed to the screen. If the job returns an + exit status of 1 (a failure), the failure queue receives a single + entry. Although they are not defined in this case, it is also + possible to specify the action to be taken if the job crashes (does + not exit normally) or if it times out (runs beyond the specified + timeout period). + + A slightly more complex task which performs a state test and + constructs a command based on that test is shown below + +

+  task datalist
+    periods -exec 5.0
+    periods -timeout 50.0
+    periods -poll 1.0
+
+    task.exec 
+      $file = `next.file`
+      if ($file == "none")
+        break
+      end
+      command cp /data/foo/$file /data/bar
+    end
+
+    task.exit 0
+      queueprint stdout
+      queuedelete stdout
+      queuepush copied $file
+    end
+ 
+    task.exit 1
+      queuepush failure $file
+    end
+  end
+
+ + The task.exec macro is executed by psched every 5.0 + seconds. This macro executes a (hypothetical user-defined) UNIX + command (next.file) which examines the system state, return + either a filename or the word "none". If the result of this test is + "none", the task does nothing: no job is constructed. Otherwise, a + job is constructed using the name of the file returned by the state + test. Successful jobs have the filename added to the 'copied' + queue, while failed jobs add the filename to the 'failure' queue. + +

Parallel vs Local Job Processing

+ +

Task Restrictions

+ +

Inter-Task and Inter-Job Communications

+ +

psched Design

+ +

The Opihi Shell

+ +

Task List

+ +

Job List

+ +

pcontrol Interface

+ +

Index: /trunk/Ohana/doc/www/html/Opihi-Programs/.cvsignore =================================================================== --- /trunk/Ohana/doc/www/html/Opihi-Programs/.cvsignore (revision 4727) +++ /trunk/Ohana/doc/www/html/Opihi-Programs/.cvsignore (revision 4727) @@ -0,0 +1,2 @@ +*.html +index.idx Index: /trunk/Ohana/doc/www/html/Opihi-Programs/index.htm =================================================================== --- /trunk/Ohana/doc/www/html/Opihi-Programs/index.htm (revision 4727) +++ /trunk/Ohana/doc/www/html/Opihi-Programs/index.htm (revision 4727) @@ -0,0 +1,128 @@ + + + + +

Introduction

+ +

+Opihi is a program to manipulate and display 1-D (vector) and 2-D +(matrix) data. Opihi has a simple command-line interaction that +resembles the UNIX tcsh, but with many additional useful features. +Opihi consists of several, vaguely independant portions: the +command-line interpreter language (Opihi), the image display tool +(Kii), the graphing tool (Kapa), and the vector and matrix +manipulation commands. + +

User Interface -- Opihi

+ +

+Opihi is a generic command-line interpreter which has been used as the +front-end for Opihi, as well as other programs in the Elixir system. +The interpreter has a variety of features of shell-scripting +languages: if statements, for loops, macros with command-line +arguments, and so forth. The shell also allows for variables, +arithmetic on variables, input from source files, and a variety of +other useful tools. I have used the Opihi shell language for several +other user interface front ends, in addition to the Opihi data analysis +tool: a CCD controller interface and a variable star database in the +LONEOS project. The programming structure of the Opihi front-end +makes it very easy to add commands to the package; more about this +later. + +

+The command-line interaction is based on the readline libraries and +behaves like tcsh. Arrows can be used for editing. There is both +command and file completion with the TAB key. Multiple commands can +generally be placed on one line with semi-colons as separators. + +

+Scalar variables in Opihi are proceeded with a dollar sign ($). A +variable may be created and the value assigned by a line which looks +like: + +

+$var = (expression) +

+ +where (expression) is some math expression. The math expression may +consist of the standard math operators (+,-,*,/) as well as any +already-defined variables and the functions log(), ln(), sqrt(), +exp(), ten() ($10^x$), sin(), cos(), etc. Also, if there is a pair of +curly brackets {} anywhere on a command line, whatever is inside is +assumed to be a math expression and evaluated as well. This later +feature allows functions of variables to be passed as arguments to +Opihi functions. Variables can be numeric or character strings. If +the shell does not understand the syntax of the line as a math +expression, it is assumed to be a string. + +

+Some of the programming functions are for loops and if (if else) +statements. There are no delimiting characters; ends of program +blocks are defined by the word 'end'. Syntax is rather simplistic at +the moment. For example: + +

+for i 1 10 0.1
+ echo $i
+end
+
+macro test
+ echo "this is a macro"
+ echo "number of arguments $0"
+ echo "first argument $1"
+ if ($1 = 10) 
+   echo first argument is 10
+ end
+end
+
+ +

+Some of the other useful programming features are the ability to +run files as scripts (input (filename)) and the option of +performing unix system calls (exec (command)). + +

1D Data and the Graphing Window

+ +

+Opihi has a variety of commands to manipulate 1-D data (vectors). +Vectors can be loaded from a data file or created with uniform +spacing. New vectors can be defined as the arithmetical combinations +of other vectors. For example, we could create a sine wave with the +following two lines: + +

+create x 1 20 0.01
+set y = sin(x)
+
+ +We can plot this pair of vectors on the Kapa graphing window: + +
+limit x y
+plot x y
+box
+
+ +The Kapa window has a variety of style options to change the plotting +type (line, histogram, point), point type, line color, weight and +style, and the errorbar style (if errorbars are plotted). + +

2D Data and the Image Display Window

+ +

+Opihi has many commands to manipulate 2-D data (images). Images can be +loaded from FITS files and mathematical operations applied to them. +Various other operations, such as rebinning, shifting, rotating, and +so forth are also available. Images can be displayed using the Kii +image window. In an 8-bit visual, the image window has a dynamic +colormap to allow the user to change the relationship between pixel +value and the displayed color. There are three colormaps which can be +selected by pressing the middle three buttons on the bottom row. The +PS button produces a PostScript file from the image. The user can +change the magnification and the position of the image with the mouse +(left - recenter at cursor; middle - zoom out; right - zoom in). +There are four color overlays which the user can draw objects of +different shapes (circles, boxes, lines) or draw a contour. The four +buttons labeled R, G, B, Y turn on or off the display of the red, +green, blue, and yellow overlays. +