+The program #pclient/# is used to support the remote jobs which are
+run on the remote hosts by #pcontrol/#. The concept of pclient is to
+act as a buffer between the job running on the remote host and
+pcontrol. The pcontrol design uses (by default) ssh connections
+initiated by pcontrol to the remote hosts. These connections execute
+the remote program of pclient. The use of a remote login process lets
+the UNIX system take care of the user authentication issues. In this
+case, the recommended practice is to set up ssh to allow the
+connection to the remote host without additional authentication using
+the appropriate authorized keys (see this
+article on ssh issues).
+
+
+It is convenient to keep a continuous connection to the remote hosts.
+This avoids incurring the overhead of authentication for each command
+which is executed, while keeping a high-quality user authentication
+process in place.
+
+
+pclient acts as a buffer between pcontrol and the remote background
+process, allowing the continuous connection to remain viable without
+samping pcontrol with output from the jobs.
+
+
+pclient has a very limited command set, as follows:
+
@@ -18,223 +18,229 @@
tasks in the IPP is pcontrol.
-
-pcontrol interacts with the collection of computers under its
-management and with other subsystems in the IPP. The IPP Controller
-receives a variety of inputs from other subsystems, described below,
-and initiates actions such as adding a new process to the queue of
-pending tasks. pcontrol also provides information to other
-subsystems on demand about its processing history and current state.
-Each physical computer may have multiple processors; since
-pcontrol is managing processing tasks, it treats each
-processor independently. It is up to the system configuration if each
-computer needs to reserve one of its CPUs to manage background tasks
-or if pcontrol should attempt to send one task per CPU and
-let the operating system handle the I/O load.
-
-
+pcontrol maintains a table of available processing computers
(hosts) and tracks their status. Hosts managed by
-pcontrol are allowed to be in one of several states, and
-pcontrol must interact with it in an appropriate way for each
-of those states. A Node may be {\tt alive}, {\tt dead} or {\tt off}.
-If the Node is {\tt alive}, it responds to commands from the IPP
-Controller and may be used for tasks subject to other constraints. If
-it is {\tt dead}, the Node is not responsive and must not be used for
-executing tasks. pcontrol must identify Nodes which have
-died (not responding) and occasionally test them to see if they are
-{\tt alive} again. Nodes which are {\tt off} are not available for
-tasks and must not be tested. Nodes may be set to the {\tt off} or
-{\tt dead} states by external subsystems; it is the responsibility of
-pcontrol to return a Node to the {\tt alive} state if
-possible.
-
-pcontrol must honor requests (normally from the users) to
-change the mode of any computing node on demand between {\tt off} and
-{\tt dead}. This would normally be done after a Node has been
-rebooted and is released to pcontrol for its use. It
-must also be able to change the list of allowed tasks as requested by
-external commands.
-
-Two example scenarios illustrate the transition between these states,
-and the basic concept of operations for pcontrol. First,
-imagine a computer crashes. At this point pcontrol should
-detect that the Node is no longer responsive and mark it as {\tt
-dead}. It should occasionally try to re-establish communication with
-the Node, potentially with longer and longer delays between attempts.
-A human could be notified if the Node seems to remain {\tt dead} for a
-very long time. In another scenario, a person needs to work on a
-Node. They notify pcontrol that the machine is {\tt off},
-perhaps with a prior notification that the machine should be prepared
-to go off. When work on the machine is complete, it should be placed
-in the {\tt dead} state. Only when the person is done working and
-testing the machine, and tells pcontrol that the machine is
-now {\tt dead} can pcontrol attempt to re-start
-communications and re-new processing operations on that Node.
-
-
+If the host is off, it is known to pcontrol, but pcontrol
+does not have an active connection to the machine. Hosts which are
+off are not available for jobs, and pcontrol does not attempt
+to initiate a connection to them.
+
+
+When pcontrol is told to consider a machine on, the machine is moved
+from the off state to the down state. Pcontrol
+attempts to initiate a connection to the host. Connections are made
+by running a remote client on the host, using the specified connection
+method. The connection method may be ssh, rsh, or
+an equivalent remote shell connection. The choice is specified by the
+COMMAND Opihi variable. The remote connection starts a dedicated
+remote client which must accept the pcontrol client commands and
+respond appropriately. The provided remote client is called
+pclient, though in principal other equivalent programs could
+be used by setting the Opihi variable SHELL (this feature more
+generally allows a user to specify a path to the remote client, if it
+is not in the user's path). A pcontrol user may force a host to
+transition to the off state with the command host off
+(hostname). ( Note that this command will set only one of
+the connections to the named host to off. If multiple
+connections to a machine have been defined, multiple off
+commands must be sent).
+
+
+If the remote connection is successful, the connected host is moved by
+pcontrol from the down state to the idle state. If
+the connection is unsuccessful, pcontrol will try again after a
+certain period of time. If the connection continues to be
+unsuccessful, the retry period is doubled for each successiver
+connection attempt. If the user wants to force pcontrol to retry the
+connection to a machine (if, for example, the timeout is now very
+long, but the user knows the machine's ethernet cable has been
+re-inserted...), this can be achieved with the command host retry
+(hostname). A host which is down is in the limbo state
+between off and idle.
+
+
+Once pcontrol has made a successful connection to the host, the host
+is in the idle state. At this point, it is ready to accept
+jobs from pcontrol for execution. Pcontrol repeatedly queries the
+hosts to check that they are still alive. If a host is discovered to
+be unresponsive, and particularly if the remote pipe connection has
+closed, then the machine is moved back to the down state.
+
+
+Hosts which are idle may accept a job from pcontrol. A job simply
+consists of a bare UNIX command, without redirection of standard input
+or standard output. The host will initiate the job, and pcontrol will
+place the host into the busy state. The remote client, pclient,
+runs the job in the background and will continue to accept input from
+pcontrol. pcontrol will continue to check the status of the host, and
+now also the status of the specific job. As before, if the connection
+breaks, pcontrol will migrate the host to the down state. Any job
+already initiated on a host which goes down will be returned for later
+processing, so the job will not be lost.
+
+
+When the job exits, pclient tells pcontrol that the job is completed,
+and specifies the exit status. At this point, pcontrol will move the
+host from busy to done state. It will stay in this
+state until pcontrol can determine the ending conditions and reset the
+remote client. pcontrol requests the standard error and standard
+output from the job from pclient. pcontrol stores this data with its
+information about the completed job, and send a reset command to the
+remote client. Once these cleanup tasks are successfully completed,
+pcontrol will move the host to the idle state, ready for
+further jobs.
+
+
+Each physical computer may have multiple processors.
+pcontrol treats each processor independently. It is up to
+the system configuration if each computer needs to reserve one of its
+CPUs to manage background tasks or if pcontrol should attempt
+to send one task per CPU and let the operating system handle the I/O
+load. some of this behavior will probably be eventually more
+intelligent. For example, the commands which turn a host on or off
+should be able to do the same operation to all host connections for
+the same machine name.
+
+
+A machine may be completely removed from pcontrol's host tables with
+the command host delete (hostname).
+
+
+The pcontrol accepts new jobs with the command job
+..., in which the ellipsis represents the command and arguments
+of a valid UNIX command. The commands are run under sh, and
+are executed in the user's home directory. (If it is desired, we
+can easily add a command to tell pclient to perform cd).
+Users should be wary of the conditions under which the remote jobs are
+run. If the nodes in question all cross-mount the same home
+directories, multiple jobs which interact with the same named file may
+produce unexpected results. The controller cannot enforce good
+behavior on the part of the remote jobs; it is the responsibility of
+the user to ensure that conflicts do not arise by, eg, always using
+unique output file names.
+
+
+Other issues may arise from the fact that pcontrol may be choosing any
+of the hosts to run the job. Typical failures arise if the user does
+not realize that specific jobs do not behave the same on all machines,
+or if a necessary resource (eg, some input data file) is only
+available or accessible from some of the hosts. It is the
+responsibility of the task to wait for network lags (ie, NFS delays).
+
+
+pcontrol gives each task a unique internal identifier (Job
+ID) equivalent to the process ID used in UNIX. When a job is
+submitted to pcontrol, the command echoes back the Job ID. This ID
+may be used by other pcontrol commands to obtain information about or
+interact with the job.
+
+
+A job may specify a specific host for the task execution. The host
+specified for a job may be required, or desired. In the
+first case, pcontrol, will only run the job on the specified host,
+waiting until it is available before attempting the job. In the
+second case, pcontrol will attempt to send the job to the specified
+host, but if the host is unavailable (how long? what
+conditions?), pcontrol will allow the job to be sent to an
+alternative host. pcontrol attempts to honor the requests
+for required and desired hosts, giving priority first to required-host
+jobs, then to the desired-host jobs, and finally to all other jobs.
+To specify a host for a job, the following commands are used:
+
+
+Job priority / urgency levels are not implemented at this time.
+
+
+I/O vs CPU tasks are not currently distinguished by pcontrol
+
+
+pcontrol stores the stdout and stderr for each completed job.
+To retrieve these data from these streams, the user issues the
+commands stdout (JobID) and stderr (JobID). The
+result is a single line specifying the number of bytes to expect,
+followed by a dump of the buffers, followed by the prompt. It is the
+user's responsibility to relieve pcontrol of this data load by
+deleting jobs once they are no longer needed. Job deletion is
+performed with the command delete (JobID).
+
+
+Jobs are moved between the following states by pcontrol:
+
+It is possible to check the status of a single host or job with the
+user command check.
+
+
+pcontrol continuously examines the stack of jobs, adjusting their
+state as needed and extracting their output when it is ready. These
+checks are performed in the background, with pcontrol ready to accept
+further commands from the user in the foreground. These checks are
+performed after every keystroke, and also after an inactivity timeout.
+The interrupt interval defaults to 1 second, but may be adjusted with
+the pulse command, which takes as an argument, the number of
+microseconds for the timeout.
+
+
+the pcontrol system status may be examined with the command
+status. This provides a dump of the job stacks and the host
+stacks.
+
+
+It is possible to list the jobs currently in a specific stack,
+corresponding to the list of jobs with a given state. This is done
+with the command jobstack (stackname). The valid stack names are
+pending, busy, exit, crash, and done. The result is a list of all
+jobs on the specified stack. This is useful to determine quickly
+which jobs have exited or crashed.
+
+
+A specific job may be killed with the command kill (JobID).
+This command is only valid for a job in the busy state. Any
+job in the pending, exit, or crash state
+may be deleted with the delete (JobID) command. This is
+necessary to free the memory associated with the job and its output
+streams.
+
+
+The command verbose (mode) turns the verbosity of the
+pcontrol operations on or off.
+
+
The pcontrol and the IPP Image Server have related needs for
information from the combined storage-and-processing nodes regarding
which nodes are available. It is not yet clear if this information is
-best stored in a single location (either pcontrol or IPP Image
-Server), which provides the information to other systems on demand, or
-if both systems should maintain the information. Also, it may be
-necessary to distinguish nodes which are available for processing from
-those that are available to serve data as part of the IPP Image
-Server.
-
-The Controller maintains three tables of processing jobs: pending
-stages, active stages, and completed stages. The pending stages are
-those which have not yet been performed. The active stages are those
-currently being performed on one of the remote nodes. The completed
-stages are those which have finished, either successfully or with an
-error state. The Controller daemon monitors the collection of remote
-clients and sends them new pending stages when they become free.
-
-The pcontrol provides a mechanism for users (either other
-programs or humans) to interact with it. The user interface provides
-commands to check the current processing job queues, the tables of
-successful and failed jobs, to stop or delete jobs, etc.
-
+best stored in a single location (either pcontrol or IPP
+Image Server), which provides the information to other systems on
+demand, or if both systems should maintain the information. Also, it
+may be necessary to distinguish nodes which are available for
+processing from those that are available to serve data as part of the
+IPP Image Server.
+
Index: /trunk/Ohana/doc/www/html/IPP-subsystems/psched/index.htm
===================================================================
--- /trunk/Ohana/doc/www/html/IPP-subsystems/psched/index.htm (revision 4731)
+++ /trunk/Ohana/doc/www/html/IPP-subsystems/psched/index.htm (revision 4732)
@@ -1,77 +1,77 @@
-This article describes the concept, design, and operation of
-psched, the Pan-STARRS IPP task scheduler.
-
-
- The purpose of psched is to manage the automatic construction
- and execution of inter-related (often repetative) operations.
- psched uses a set of rules to define UNIX commands, and
- their corresponding command-line arguments, to be performed on some
- regular, repeated basis. The utility of psched is that it
- can easily define an analysis system which is completely
- state-based, as opposed to an event-driven system.
-
-
- Consider, for example, a telescope which obtains a collection of
- images over the course of a night. Every minute or two, it takes an
- image and writes the image to some disk. An event-driven analysis
- system would involve having the telescope initiate a process at the
- end of the exposure. This process would perform an analysis, write
- some output, then send trigger another process. This type of
- operation works very well for a simple set up with reliable
- hardware. Such a system becomes more difficult to maintain when
- hardware failures occur or when multiple systems need to interact
- with each other. When failures occur, the triggering information
- (the events) is easily lost, thus some mechanisms are needed to
- detect these failures and either re-send the trigger or send an
- alternative failure-mode trigger. Or, if two systems need to
- interact, one or the other system must block for results from the
- first. Stopping and restarting such an analysis system is very
- delicate since the appropriate triggers must be set up some how, eg
- by noticing which images have not succeeded and restarting them at
- the appropriate stage. All of these types of methods of handling
- complexity and failures are essentially state-based rules.
- psched allows the easy definition of a totally state-based
- analysis system.
-
-
- In a state-based system, some mechanism examines the state of the
- system and decides which actions to perform based on the current
- state. In the illustration above, the mechanism could examine the
- images available (either by examining the disk or by examining the
- state of a data table) and decide to perform an operation based on
- what images are available. This makes it very easy to handle
- complexity and errors. If an analysis fails, the state either is
- not successfully updated or the error state is recorded, both
- situations being easy to detect and easy to handle. Restarting the
- system simply involves starting the state-monitoring mechanism.
- Combining results from multiple input sources simply involves
- watching for the multiple inputs to be available. psched
- provides a mechanism to define state monitors, and to define the
- actions which are performed when those states occur.
- psched action consist of initiating UNIX commands, where
- the arguments of those commands may depend on the results of the
- state tests.
-
-
- The primary function of psched is to repeatedly perform
- tasks, and execute jobs on the basis of those tasks.
- A task consists of a set of rules which describe system state tests
- to perform on a regular time scale. Based on the results of those
- state tests, the task will then choose whether or not to construct a
- job. The task also defines actions to perform upon the completion
- of a job, based upon the output and exit status of the job. A task
- thus defines the repeat period. It may optionally define valid or
- invalid time ranges (eg, Mon-Fri or 10:00-17:00, etc). The task may
- also specify that the job be run locally (ie, in the background on
- the same computer as psched) or remotely by the parallel process
- controller (pcontrol). A job may even be restricted to a
- specific computer managed by pcontrol.
-
- An example of a simple tasks is given below.
+This article describes psched, the Pan-STARRS IPP task
+scheduler.
+
+
+The purpose of psched is to manage the automatic construction
+and execution of inter-related (often repetative) operations.
+Psched uses a set of rules to define UNIX commands, and
+their corresponding command-line arguments, to be performed on some
+regular, repeated basis. The utility of psched is that it
+can easily define an analysis system which is completely
+state-based, as opposed to an event-driven system.
+
+
+Consider, for example, a telescope which obtains a collection of
+images over the course of a night. Every minute or two, it takes an
+image and writes the image to some disk. An event-driven analysis
+system would involve having the telescope initiate a process at the
+end of the exposure. This process would perform an analysis, write
+some output, then send trigger another process. This type of
+operation works very well for a simple set up with reliable
+hardware. Such a system becomes more difficult to maintain when
+hardware failures occur or when multiple systems need to interact
+with each other. When failures occur, the triggering information
+(the events) is easily lost, thus some mechanisms are needed to
+detect these failures and either re-send the trigger or send an
+alternative failure-mode trigger. Or, if two systems need to
+interact, one or the other system must block for results from the
+first. Stopping and restarting such an analysis system is very
+delicate since the appropriate triggers must be set up some how, eg
+by noticing which images have not succeeded and restarting them at
+the appropriate stage. All of these types of methods of handling
+complexity and failures are essentially state-based rules.
+Psched allows the easy definition of a totally state-based
+analysis system.
+
+
+In a state-based system, some mechanism examines the state of the
+system and decides which actions to perform based on the current
+state. In the illustration above, the mechanism could examine the
+images available (either by examining the disk or by examining the
+state of a data table) and decide to perform an operation based on
+what images are available. This makes it very easy to handle
+complexity and errors. If an analysis fails, the state either is
+not successfully updated or the error state is recorded, both
+situations being easy to detect and easy to handle. Restarting the
+system simply involves starting the state-monitoring mechanism.
+Combining results from multiple input sources simply involves
+watching for the multiple inputs to be available. Psched
+provides a mechanism to define state monitors, and to define the
+actions which are performed when those states occur.
+Psched action consist of initiating UNIX commands, where
+the arguments of those commands may depend on the results of the
+state tests.
+
+
+The primary function of psched is to repeatedly perform
+tasks, and execute jobs on the basis of those tasks.
+A task consists of a set of rules which describe system state tests
+to perform on a regular time scale. Based on the results of those
+state tests, the task will then choose whether or not to construct a
+job. The task also defines actions to perform upon the completion
+of a job, based upon the output and exit status of the job. A task
+thus defines the repeat period. It may optionally define valid or
+invalid time ranges (eg, Mon-Fri or 10:00-17:00, etc). The task may
+also specify that the job be run locally (ie, in the background on
+the same computer as psched) or remotely by the parallel process
+controller (pcontrol). A job may even be restricted to a
+specific computer managed by pcontrol.
+
+An example of a simple tasks is given below.
- This task does not perform any system state tests; it is simply
- constructs a new job every 5.0 seconds. The job in this case is
- always the same: ls /data/foo . When the job finished,
- if the job exit status is 0 (normal UNIX success status), the
- resulting output is printed to the screen. If the job returns an
- exit status of 1 (a failure), the failure queue receives a single
- entry. Although they are not defined in this case, it is also
- possible to specify the action to be taken if the job crashes (does
- not exit normally) or if it times out (runs beyond the specified
- timeout period).
-
- A slightly more complex task which performs a state test and
- constructs a command based on that test is shown below
+
+This task does not perform any system state tests; it is simply
+constructs a new job every 5.0 seconds. The job in this case is
+always the same: ls /data/foo . When the job finished,
+if the job exit status is 0 (normal UNIX success status), the
+resulting output is printed to the screen. If the job returns an
+exit status of 1 (a failure), the failure queue receives a single
+entry. Although they are not defined in this case, it is also
+possible to specify the action to be taken if the job crashes (does
+not exit normally) or if it times out (runs beyond the specified
+timeout period).
+
+A slightly more complex task which performs a state test and
+constructs a command based on that test is shown below
+It is possible to interact directly with the parallel processor to
+examine the current status, halt the parallel processor, etc.
+Commands to the parallel processor are defined under the
+controller command. The following controller commands are
+available:
+
+Commands
+
+
+
+
Index: /trunk/Ohana/doc/www/html/IPP-subsystems/pcontrol/index.htm
===================================================================
--- /trunk/Ohana/doc/www/html/IPP-subsystems/pcontrol/index.htm (revision 4731)
+++ /trunk/Ohana/doc/www/html/IPP-subsystems/pcontrol/index.htm (revision 4732)
@@ -1,9 +1,9 @@
-
-
+
+
pcontrol is the IPP parallel process controller.
-Overview
+Overview
hosts
-
-The Controller maintains a table of available processing computers
+Host States
+
+pclient
-
-When the Controller starts, it attempts to launch a Node Agent on each
-of the available processing Nodes. Nodes which are not responsive are
-marked as {\tt dead} so they may be re-tried. A Node Agent runs on
-each of the individual nodes to execute the tasks as directed by the
-Controller. The Node Agents communicate with the Controller via a
-socket connection.
-
-A Node Agent (which is only running on a Node in the {\tt alive}
-state) may be in one of four modes: {\tt idle}, {\tt busy}, {\tt
-done}, {\tt crash}. A Node Agent which is {\tt busy} currently has a
-task assigned to it which is executing. The pcontrol may only
-assign one task to a Node at a time. A Node Agent which is in the
-{\tt idle} state may have a task assigned to it. When the Node Agent
-detects that a tasks has finished, it changes to either the {\tt done}
-or {\tt crash} states depending on the outcome of the process
-execution. The pcontrol must also respect a list of task
-restrictions which may require specific tasks to run on specific CPUs
-or exclude specific tasks from specific CPUs.
-
-A task being executed by the Node is run in the UNIX user space as a
-forked process. The Node Agent must monitor the standard error and
-standard output of the executing task and save them in separate
-buffers. If the process exits or dies, the Node Agent must detect
-this result and change state appropriately. The Node Agent must
-respond to various commands from the Controller, as follows:
-
-\paragraph{Report status}
-
-The Node Agent returns its state ({\tt idle}, {\tt busy}, {\tt done},
-{\tt crash}) and the exit status of the current processing task, if
-available. The reported exit state, if the process has completed
-without crashing, is the UNIX exit state reported by the task: 0--256
-with 0 indicating a successful completion.
-
-\paragraph{Report stdout}
-
-Send and flush the current stdout buffer. The Node Agent will return
-the complete contents of the stdout buffer via a buffered write and
-flush the buffer when it is finished. The Node Agent will not accept
-more data on the stdout buffer from the current processing task until
-the send is complete and the buffer is flushed. The daemon must
-accept all of the buffer output.
-
-\paragraph{Report stderr}
-
-Identical to `report stdout', but for stderr.
-
-\paragraph{Kill task }
-
-The Node Agent should send a kill signal (\code{KILL} or \code{TERM})
-to the current processing task. When the processing task has exited,
-the Node Agent should set its state to {\tt crash}.
-
-\paragraph{Clear task}
-
-The Node Agent should set its state {\tt idle}. If a processing stage
-is currently running, it should be killed (\code{KILL} or \code{TERM})
-before the task is cleared.
-
-\paragraph{Start processing stage}
-
-The Node Agent forks a specified command. The command should be a
-standard UNIX command without command line redirection or
-backgrounding. The task is run with the same user ID as the Node
-Agent, which is also the same user ID as the Controller.
-
-\subsubsection{Tasks}
-
-The pcontrol accepts tasks from other IPP subsystems. The task
-requests include the specific command to be executed and are in the
-form of a UNIX command which could be performed on any of the
-computing nodes. Any input or output data in the commands must be a
-valid resource regardless of the node on which the task is executed.
-Input and output data resources must be unique where necessary to
-avoid conflicts. It is the responsibility of the task to wait for
-network lags (ie, NFS delays). The pcontrol gives each task a
-unique identifier, which is returned to the requesting entity. The
-requestor may then use that ID to obtain status information on that
-task or to send control signals to the specific task.
-
-Task requests may specify a desired node for the task execution. The
-pcontrol attempts to honor the request if the node is {\tt
-alive}, but will execute it on another node if the requested one is
-{\tt dead} or {\tt off}. Even if a node is {\tt alive}, the IPP
-Controller will choose another node if the specified task is not
-allowed on the requested node. In all other cases, the pcontrol
-waits until the currently executing processes, and processes with
-higher priority, are completed before executing the specified task on
-the requested node.
-
-Task requests may specify an urgency level. The pcontrol
-determines the priority of the task on the basis of both the urgency
-and the age of the request. An executing task must be completed on a
-CPU before any new task is started on that CPU, regardless of
-priority. The urgency levels range from 0 to 2. Tasks with an
-urgency of 1 are scheduled whenever they reach the top of the stack.
-Tasks with an urgency of 2 are sent immediately to the top of the
-stack. Tasks assigned a priority of 0 are maintained in the queue and
-never executed.
-
-It may be useful for the Controller to distinguish between tasks
-dominated by I/O and tasks dominated by data processing. It is
-possible that one of each of these types of tasks may be sent to the
-same node without significantly impacting the system performance.
-Alternatively, it may be necessary to limit a single machine with 2
-CPUs to only one of each of these types of tasks (i.e., one processor
-will be working on I/O while the other is working on processing).
-Such details will be studied by the IfA IPP Team.
-
-The pcontrol monitors the output streams from the executing
-tasks and the exit status of the tasks. Each task is associated with
-a log file, to which all output is written. The status, including the
-exit status, of each task is maintained by the pcontrol so that
-other subsystems may determine if specific tasks have started or
-completed.
-
-\subsubsection{Controller Interfaces}
-
-The pcontrol must accept commands from other IPP subsystems.
-These commands include those which govern the processing of specified
-tasks, those which govern the behavior of specific computing nodes,
-and those which request information from the pcontrol. The IPP
-Controller must be able to halt the execution of a specified task,
-delete an unexecuted task from the task list, change the priority of
-tasks, and change the requested nodes for tasks. The pcontrol
-must also be able to stop the current execution of a task and push it
-to the end of the queue and also change its priority.
-
-The pcontrol must respond to informational requests regarding the
-collection of machines and their states as well as the collection of
-tasks and their states. The pcontrol must monitor the execution
-times of the different tasks and provide summary statistics. Finally,
-the pcontrol must respond to three top-level commands: {\tt finish},
-{\tt stop} and {\tt abort}. When {\tt finish} is requested, no more
-new tasks are accepted on the stack of task, and when all tasks in the
-stack have completed, the pcontrol must exit. When {\tt stop} is
-requested, the currently executing tasks must be completed at which
-point the pcontrol must exit, but tasks remaining in the stack which
-have not been started are flushed. When {\tt abort} is issued, the
-pcontrol immediately kills all executing tasks and exits.
-
+pcontrol are allowed to be in one of several states:
+off, down, idle, busy, and
+done. These states have the following meanings:
+
+Jobs
+
+
+job -host (command and arguments...)
+job +host (command and arguments...)
+
+
+The first case specifies a desired host, while the second specifies a
+required host. It is also possible to specify the special host name
+anyhost, which is equivalent to not specifying a host at all.
+
+
+
+
+Miscellaneous Commands
+
+ Basic Concept
-
- Tasks vs Jobs
-
- Overview
+
+ Tasks vs Jobs
+
+
@@ -93,18 +93,18 @@
-
@@ -134,28 +134,231 @@
- The task.exec macro is executed by psched every 5.0
- seconds. This macro executes a (hypothetical user-defined) UNIX
- command (next.file) which examines the system state, return
- either a filename or the word "none". If the result of this test is
- "none", the task does nothing: no job is constructed. Otherwise, a
- job is constructed using the name of the file returned by the state
- test. Successful jobs have the filename added to the 'copied'
- queue, while failed jobs add the filename to the 'failure' queue.
-
- Parallel vs Local Job Processing
-
- Task Restrictions
-
- Inter-Task and Inter-Job Communications
-
- psched Design
-
- The Opihi Shell
-
- Task List
-
- Job List
-
- pcontrol Interface
-
-
+The task.exec macro is executed by psched every 5.0
+seconds. This macro executes a (hypothetical user-defined) UNIX
+command (next.file) which examines the system state, return
+either a filename or the word "none". If the result of this test is
+"none", the task does nothing: no job is constructed. Otherwise, a
+job is constructed using the name of the file returned by the state
+test. Successful jobs have the filename added to the 'copied'
+queue, while failed jobs add the filename to the 'failure' queue.
+
+
Parallel vs Local Job Processing
+
+Job which are generated by psched tasks may either be run locally
+(forked in the background on the same machine as psched) or run on
+the IPP parallel process controller, pcontrol. The default
+is for the job to be run locally. If a job should be run on the
+parallel controller, this can be specified by including the command
+host (hostname) in the definition of a task. If the value
+of (hostname) is 'anyhost', then pcontrol may select any of its host
+computers to run the job according to its own rules. If the value
+of (hostname) is one of the computers managed by pcontrol, then that
+machine will be selected for the job, if it is available. This
+amounts to a preference to use that machine, but pcontrol is allowed
+to substitute a different machine if it chooses. If the
+host command is given the option -required, then
+pcontrol is forced to use the named host, even if the machine is
+down, unknown, or otherwise unavailable. If the machine is not
+available, pcontrol will simply hold onto the job until the machine
+is available or the job is deleted. Note that psched may delete
+jobs from pcontrol if they remain pending for too long (see
+period -timeout).
+
+
+
+
+It is also possible to specify a host for a task which has not been
+identified to the controller. If such a host is required, the
+controller will simply keep the associated jobs in the pending state
+until such a machine exists. See the pcontrol
+documentation for further discussion of the controller manipuation of
+jobs and hosts.
+
+
+
+
+The time range may be given as a range of absolute dates as follows: + +
+trange YYYY/MM/DD,HH:MM:SS YYYY/MM/DD,HH:MM:SS ++ +where the two dates specify the start and end of the time range. In +either of these date representations, the least-significant elements +of the date and time may be dropped, defaulting to 00 (in the case +of hours, minutes, and seconds) or 01 (in the case of day and +months). Rather than specifying an end date, it is also valid to +specify a time interval from the starting date. The time interval +is specified as a number followed by a unit indicated by a single +letter: d (days), h (hours), m (minutes), s (seconds). + +
+The time range may also be specified as a repeated period of time, +either as a time of day or a day and time of week. In the first +case, the time range is specified as follows: + +
+trange HH:MM:SS HH:MM:SS ++ +where again the least-significant elements may be dropped and +default to 00. This type of restriction defines a time range which +is valid every day. The alternative is to specify a time range +within the week, in the following form: + +
+trange DAY@HH:MM:SS DAY@HH:MM:SS ++ +where the value of DAY may take on any of the three letter +day-of-week names (Sun, Mon, Tue, etc). This restriction specifies +a start and end time within a week which is evaluated for each +week. + +
+Below are several examples of valid time range restrictions + +
+trange 2005/01/01 2005/12/31 (only run during 2005!) +trange 18:00 00:00 (only run from 6pm until midnight) +trange 00:00 06:00 (only run from midnight until 6am) +trange Mon@08:00 Fri@17:00 (only run between Mon morning and Fri afternoon) +trange -exclude 12:00 13:00 (skip 1 hour from noon) ++ +Note that the current definition of trange does not include time +zone information. This means that all times are relative to +UT. This should be addressed by adding a timezone environment +variable to psched and by allowing the trange to define a timezone +offset. + +
+It is also possible to restrict the total number of jobs which are +spawned for a given task. This is done with the nmax +command, which is given as part of the task definition. Once a task +has constructed nmax jobs, it stops task evaluation. It is possible +to redefine the value of nmax at any time by redefining the task. +Any time the task is redefined, the new values for any task concept +will override the existing values for the task concept. + +
+It is always possible for the interprocess communication to be +performed externally: all jobs may simply write results to an +external data source which is queried as part of the task +evaluation. Psched may interact with UNIX programs using Opihi +system interaction functions. These interaction methods include: +the backticks for setting Opihi variables: + +
+$variable = `UNIX Command` ++ +The exec command (which executes a UNIX command) and the backticks +both receive the UNIX command exit status, setting the variable +$STATUS. It is also possible to set a variable list to the output +of a UNIX command: + +
+list var -x "UNIX Command" ++ +In this last case, the values $var:0 - $var:N-1 are set to the value +of the stdout lines from the UNIX command, and the value $var:n is +set to the number of output lines. + +
+Fine-grained control over the job exit status is available with the +task.exit macro command. This allows a task to define an +exit macro which is performed for different exit status conditions. +The argument to the task.exit command is the exit status +value which triggers the macro. This may consist of any valid +numeric exit status value (0-255). It may also have the value +crash, in which case the macro is executed if the program +exited as a result of a signal (ie, segmentation fault, etc). +Finally, if may have the value default, in which case, the +macro is run if no other macro describes the exit status. + +
+Jobs may transmit their results back to psched for further +evaluation through the standard output and standard error streams. +Whenever a job exits, the complete stdout and stderr streams from +the job are pushed onto the psched queues stdout and +stderr. The job exit macros may then parse these queues, +moving the results into other psched / Opihi data containers +(queues, variables, vectors, whatever is appropriate). Note +that currently, the output data is simply pushed onto these output +queues. It is currently the responsibility of the psched programmer +to use or dispose of the data in these queues. This may change in +the future: the queues may be flushed for each job completion. + +
+It is also possible to kill or delete individual jobs by hand with +the commands kill (jobID) or delete (jobID). + +