Changeset 4732 for trunk/Ohana/doc/www/html/IPP-subsystems/pcontrol/index.htm

Timestamp:

Aug 8, 2005, 10:17:52 PM (21 years ago)

Author:

eugene

Message:

fleshed out psched/pcontrol/pclient descriptions

File:

• trunk/Ohana/doc/www/html/IPP-subsystems/pcontrol/index.htm (modified) (2 diffs)

trunk/Ohana/doc/www/html/IPP-subsystems/pcontrol/index.htm

@@ -1 +1 @@
 <meta name=file  content=index>
-<meta name=title content=PCONTROL SUMMARY>
-<meta name=page  content=pcontrol summary>
+<meta name=title content=PCONTROL.SUMMARY>
+<meta name=page  content=pcontrol.summary>
 
 <tt>pcontrol</tt> is the IPP parallel process controller.
 
-<h2>Overview</h2>
+<h3>Overview</h3>
 
 <p>
@@ -18 +18 @@
 tasks in the IPP is <tt>pcontrol</tt>.
 
-<p>
-<tt>pcontrol</tt> interacts with the collection of computers under its
-management and with other subsystems in the IPP.  The IPP Controller
-receives a variety of inputs from other subsystems, described below,
-and initiates actions such as adding a new process to the queue of
-pending tasks.  <tt>pcontrol</tt> also provides information to other
-subsystems on demand about its processing history and current state.
-Each physical computer may have multiple processors; since
-<tt>pcontrol</tt> is managing processing tasks, it treats each
-processor independently.  It is up to the system configuration if each
-computer needs to reserve one of its CPUs to manage background tasks
-or if <tt>pcontrol</tt> should attempt to send one task per CPU and
-let the operating system handle the I/O load.
-
-<h2>hosts</h2>
-
-The Controller maintains a table of available processing computers
+<h3>Host States</h3>
+
+<p>
+<tt>pcontrol</tt> maintains a table of available processing computers
 (<em>hosts</em>) and tracks their status.  Hosts managed by
-<tt>pcontrol</tt> are allowed to be in one of several states, and
-<tt>pcontrol</tt> must interact with it in an appropriate way for each
-of those states.  A Node may be {\tt alive}, {\tt dead} or {\tt off}.
-If the Node is {\tt alive}, it responds to commands from the IPP
-Controller and may be used for tasks subject to other constraints.  If
-it is {\tt dead}, the Node is not responsive and must not be used for
-executing tasks.  <tt>pcontrol</tt> must identify Nodes which have
-died (not responding) and occasionally test them to see if they are
-{\tt alive} again.  Nodes which are {\tt off} are not available for
-tasks and must not be tested.  Nodes may be set to the {\tt off} or
-{\tt dead} states by external subsystems; it is the responsibility of
-<tt>pcontrol</tt> to return a Node to the {\tt alive} state if
-possible.
-
-<tt>pcontrol</tt> must honor requests (normally from the users) to
-change the mode of any computing node on demand between {\tt off} and
-{\tt dead}.  This would normally be done after a Node has been
-rebooted and is released to <tt>pcontrol</tt> for its use.  It
-must also be able to change the list of allowed tasks as requested by
-external commands.
-
-Two example scenarios illustrate the transition between these states,
-and the basic concept of operations for <tt>pcontrol</tt>.  First,
-imagine a computer crashes.  At this point <tt>pcontrol</tt> should
-detect that the Node is no longer responsive and mark it as {\tt
-dead}.  It should occasionally try to re-establish communication with
-the Node, potentially with longer and longer delays between attempts.
-A human could be notified if the Node seems to remain {\tt dead} for a
-very long time.  In another scenario, a person needs to work on a
-Node.  They notify <tt>pcontrol</tt> that the machine is {\tt off},
-perhaps with a prior notification that the machine should be prepared
-to go off.  When work on the machine is complete, it should be placed
-in the {\tt dead} state.  Only when the person is done working and
-testing the machine, and tells <tt>pcontrol</tt> that the machine is
-now {\tt dead} can <tt>pcontrol</tt> attempt to re-start
-communications and re-new processing operations on that Node.
-
-<h2>pclient</h2>
-
-When the Controller starts, it attempts to launch a Node Agent on each
-of the available processing Nodes.  Nodes which are not responsive are
-marked as {\tt dead} so they may be re-tried.  A Node Agent runs on
-each of the individual nodes to execute the tasks as directed by the
-Controller.  The Node Agents communicate with the Controller via a
-socket connection.
-
-A Node Agent (which is only running on a Node in the {\tt alive}
-state) may be in one of four modes: {\tt idle}, {\tt busy}, {\tt
-done}, {\tt crash}.  A Node Agent which is {\tt busy} currently has a
-task assigned to it which is executing.  The <tt>pcontrol</tt> may only
-assign one task to a Node at a time.  A Node Agent which is in the
-{\tt idle} state may have a task assigned to it.  When the Node Agent
-detects that a tasks has finished, it changes to either the {\tt done}
-or {\tt crash} states depending on the outcome of the process
-execution.  The <tt>pcontrol</tt> must also respect a list of task
-restrictions which may require specific tasks to run on specific CPUs
-or exclude specific tasks from specific CPUs.
-
-A task being executed by the Node is run in the UNIX user space as a
-forked process.  The Node Agent must monitor the standard error and
-standard output of the executing task and save them in separate
-buffers.  If the process exits or dies, the Node Agent must detect
-this result and change state appropriately.  The Node Agent must
-respond to various commands from the Controller, as follows:
-
-\paragraph{Report status}
-
-The Node Agent returns its state ({\tt idle}, {\tt busy}, {\tt done},
-{\tt crash}) and the exit status of the current processing task, if
-available.  The reported exit state, if the process has completed
-without crashing, is the UNIX exit state reported by the task: 0--256
-with 0 indicating a successful completion.
-
-\paragraph{Report stdout}
-
-Send and flush the current stdout buffer.  The Node Agent will return
-the complete contents of the stdout buffer via a buffered write and
-flush the buffer when it is finished.  The Node Agent will not accept
-more data on the stdout buffer from the current processing task until
-the send is complete and the buffer is flushed.  The daemon must
-accept all of the buffer output.
-
-\paragraph{Report stderr}
-
-Identical to `report stdout', but for stderr.
-
-\paragraph{Kill task }
-
-The Node Agent should send a kill signal (\code{KILL} or \code{TERM})
-to the current processing task.  When the processing task has exited,
-the Node Agent should set its state to {\tt crash}.
-
-\paragraph{Clear task}
-
-The Node Agent should set its state {\tt idle}.  If a processing stage
-is currently running, it should be killed (\code{KILL} or \code{TERM})
-before the task is cleared.
-
-\paragraph{Start processing stage}
-
-The Node Agent forks a specified command.  The command should be a
-standard UNIX command without command line redirection or
-backgrounding.  The task is run with the same user ID as the Node
-Agent, which is also the same user ID as the Controller.
-
-\subsubsection{Tasks}
-
-The <tt>pcontrol</tt> accepts tasks from other IPP subsystems.  The task
-requests include the specific command to be executed and are in the
-form of a UNIX command which could be performed on any of the
-computing nodes.  Any input or output data in the commands must be a
-valid resource regardless of the node on which the task is executed.
-Input and output data resources must be unique where necessary to
-avoid conflicts.  It is the responsibility of the task to wait for
-network lags (ie, NFS delays).  The <tt>pcontrol</tt> gives each task a
-unique identifier, which is returned to the requesting entity.  The
-requestor may then use that ID to obtain status information on that
-task or to send control signals to the specific task.
-
-Task requests may specify a desired node for the task execution.  The
-<tt>pcontrol</tt> attempts to honor the request if the node is {\tt
-alive}, but will execute it on another node if the requested one is
-{\tt dead} or {\tt off}.  Even if a node is {\tt alive}, the IPP
-Controller will choose another node if the specified task is not
-allowed on the requested node.  In all other cases, the <tt>pcontrol</tt>
-waits until the currently executing processes, and processes with
-higher priority, are completed before executing the specified task on
-the requested node.
-
-Task requests may specify an urgency level.  The <tt>pcontrol</tt>
-determines the priority of the task on the basis of both the urgency
-and the age of the request.  An executing task must be completed on a
-CPU before any new task is started on that CPU, regardless of
-priority.  The urgency levels range from 0 to 2.  Tasks with an
-urgency of 1 are scheduled whenever they reach the top of the stack.
-Tasks with an urgency of 2 are sent immediately to the top of the
-stack. Tasks assigned a priority of 0 are maintained in the queue and
-never executed.
-
-It may be useful for the Controller to distinguish between tasks
-dominated by I/O and tasks dominated by data processing.  It is
-possible that one of each of these types of tasks may be sent to the
-same node without significantly impacting the system performance.
-Alternatively, it may be necessary to limit a single machine with 2
-CPUs to only one of each of these types of tasks (i.e., one processor
-will be working on I/O while the other is working on processing).
-Such details will be studied by the IfA IPP Team.
-
-The <tt>pcontrol</tt> monitors the output streams from the executing
-tasks and the exit status of the tasks.  Each task is associated with
-a log file, to which all output is written.  The status, including the
-exit status, of each task is maintained by the <tt>pcontrol</tt> so that
-other subsystems may determine if specific tasks have started or
-completed.
-
-\subsubsection{Controller Interfaces}
-
-The <tt>pcontrol</tt> must accept commands from other IPP subsystems.
-These commands include those which govern the processing of specified
-tasks, those which govern the behavior of specific computing nodes,
-and those which request information from the <tt>pcontrol</tt>.  The IPP
-Controller must be able to halt the execution of a specified task,
-delete an unexecuted task from the task list, change the priority of
-tasks, and change the requested nodes for tasks.  The <tt>pcontrol</tt>
-must also be able to stop the current execution of a task and push it
-to the end of the queue and also change its priority.
-
-The <tt>pcontrol</tt> must respond to informational requests regarding the
-collection of machines and their states as well as the collection of
-tasks and their states.  The <tt>pcontrol</tt> must monitor the execution
-times of the different tasks and provide summary statistics.  Finally,
-the <tt>pcontrol</tt> must respond to three top-level commands: {\tt finish},
-{\tt stop} and {\tt abort}.  When {\tt finish} is requested, no more
-new tasks are accepted on the stack of task, and when all tasks in the
-stack have completed, the <tt>pcontrol</tt> must exit.  When {\tt stop} is
-requested, the currently executing tasks must be completed at which
-point the <tt>pcontrol</tt> must exit, but tasks remaining in the stack which
-have not been started are flushed.  When {\tt abort} is issued, the
-<tt>pcontrol</tt> immediately kills all executing tasks and exits.
-
+<tt>pcontrol</tt> are allowed to be in one of several states:
+<tt>off</tt>, <tt>down</tt>, <tt>idle</tt>, <tt>busy</tt>, and
+<tt>done</tt>.  These states have the following meanings:
+
+<p>
+If the host is <tt>off</tt>, it is known to pcontrol, but pcontrol
+does not have an active connection to the machine.  Hosts which are
+<tt>off</tt> are not available for jobs, and pcontrol does not attempt
+to initiate a connection to them.
+
+<p>
+When pcontrol is told to consider a machine on, the machine is moved
+from the <tt>off</tt> state to the <tt>down</tt> state.  Pcontrol
+attempts to initiate a connection to the host.  Connections are made
+by running a remote client on the host, using the specified connection
+method.  The connection method may be <tt>ssh</tt>, <tt>rsh</tt>, or
+an equivalent remote shell connection. The choice is specified by the
+COMMAND Opihi variable.  The remote connection starts a dedicated
+remote client which must accept the pcontrol client commands and
+respond appropriately.  The provided remote client is called
+<tt>pclient</tt>, though in principal other equivalent programs could
+be used by setting the Opihi variable SHELL (this feature more
+generally allows a user to specify a path to the remote client, if it
+is not in the user's path).  A pcontrol user may force a host to
+transition to the <tt>off</tt> state with the command <tt>host off
+(hostname)</tt>.  (<em> Note that this command will set only one of
+the connections to the named host to <tt>off</tt>.  If multiple
+connections to a machine have been defined, multiple <tt>off</tt>
+commands must be sent</em>).
+
+<p>
+If the remote connection is successful, the connected host is moved by
+pcontrol from the <tt>down</tt> state to the <tt>idle</tt> state.  If
+the connection is unsuccessful, pcontrol will try again after a
+certain period of time.  If the connection continues to be
+unsuccessful, the retry period is doubled for each successiver
+connection attempt.  If the user wants to force pcontrol to retry the
+connection to a machine (if, for example, the timeout is now very
+long, but the user knows the machine's ethernet cable has been
+re-inserted...), this can be achieved with the command <tt>host retry
+(hostname)</tt>.  A host which is <tt>down</tt> is in the limbo state
+between <tt>off</tt> and <tt>idle</tt>.  
+
+<p>
+Once pcontrol has made a successful connection to the host, the host
+is in the <tt>idle</tt> state.  At this point, it is ready to accept
+jobs from pcontrol for execution.  Pcontrol repeatedly queries the
+hosts to check that they are still alive.  If a host is discovered to
+be unresponsive, and particularly if the remote pipe connection has
+closed, then the machine is moved back to the <tt>down</tt> state.  
+
+<p>
+Hosts which are <tt>idle</tt> may accept a job from pcontrol.  A job simply
+consists of a bare UNIX command, without redirection of standard input
+or standard output.  The host will initiate the job, and pcontrol will
+place the host into the <tt>busy</tt> state.  The remote client, pclient,
+runs the job in the background and will continue to accept input from
+pcontrol.  pcontrol will continue to check the status of the host, and
+now also the status of the specific job.  As before, if the connection
+breaks, pcontrol will migrate the host to the <tt>down</tt> state.  Any job
+already initiated on a host which goes down will be returned for later
+processing, so the job will not be lost. 
+
+<p>
+When the job exits, pclient tells pcontrol that the job is completed,
+and specifies the exit status.  At this point, pcontrol will move the
+host from <tt>busy</tt> to <tt>done</tt> state.  It will stay in this
+state until pcontrol can determine the ending conditions and reset the
+remote client.  pcontrol requests the standard error and standard
+output from the job from pclient.  pcontrol stores this data with its
+information about the completed job, and send a reset command to the
+remote client.  Once these cleanup tasks are successfully completed,
+pcontrol will move the host to the <tt>idle</tt> state, ready for
+further jobs.
+
+<p>
+Each physical computer may have multiple processors.
+<tt>pcontrol</tt> treats each processor independently.  It is up to
+the system configuration if each computer needs to reserve one of its
+CPUs to manage background tasks or if <tt>pcontrol</tt> should attempt
+to send one task per CPU and let the operating system handle the I/O
+load.  <em>some of this behavior will probably be eventually more
+intelligent.  For example, the commands which turn a host on or off
+should be able to do the same operation to all host connections for
+the same machine name.</em>
+
+<p>
+A machine may be completely removed from pcontrol's host tables with
+the command <tt>host delete (hostname)</tt>.  
+
+<h3>Jobs</h3>
+
+<p>
+The <tt>pcontrol</tt> accepts new jobs with the command <tt>job
+...</tt>, in which the ellipsis represents the command and arguments
+of a valid UNIX command.  The commands are run under <tt>sh</tt>, and
+are executed in the user's home directory.  (<em>If it is desired, we
+can easily add a command to tell pclient to perform <tt>cd</tt></em>).
+Users should be wary of the conditions under which the remote jobs are
+run.  If the nodes in question all cross-mount the same home
+directories, multiple jobs which interact with the same named file may
+produce unexpected results.  The controller cannot enforce good
+behavior on the part of the remote jobs; it is the responsibility of
+the user to ensure that conflicts do not arise by, eg, always using
+unique output file names.
+
+<p>
+Other issues may arise from the fact that pcontrol may be choosing any
+of the hosts to run the job.  Typical failures arise if the user does
+not realize that specific jobs do not behave the same on all machines,
+or if a necessary resource (eg, some input data file) is only
+available or accessible from some of the hosts.  It is the
+responsibility of the task to wait for network lags (ie, NFS delays).
+
+<p>
+<tt>pcontrol</tt> gives each task a unique internal identifier (Job
+ID) equivalent to the process ID used in UNIX.  When a job is
+submitted to pcontrol, the command echoes back the Job ID.  This ID
+may be used by other pcontrol commands to obtain information about or
+interact with the job.
+
+<p>
+A job may specify a specific host for the task execution.  The host
+specified for a job may be <b>required</b>, or <b>desired</b>.  In the
+first case, pcontrol, will only run the job on the specified host,
+waiting until it is available before attempting the job.  In the
+second case, pcontrol will attempt to send the job to the specified
+host, but if the host is unavailable (<em>how long? what
+conditions?</em>), pcontrol will allow the job to be sent to an
+alternative host.  <tt>pcontrol</tt> attempts to honor the requests
+for required and desired hosts, giving priority first to required-host
+jobs, then to the desired-host jobs, and finally to all other jobs.
+To specify a host for a job, the following commands are used:
+
+<pre>
+job -host (command and arguments...)
+job +host (command and arguments...)
+</pre>
+
+The first case specifies a desired host, while the second specifies a
+required host.  It is also possible to specify the special host name
+<tt>anyhost</tt>, which is equivalent to not specifying a host at all.
+
+<p>
+<em>Job priority / urgency levels are not implemented at this time.</em>
+
+<p>
+<em>I/O vs CPU tasks are not currently distinguished by pcontrol</em>
+
+<p>
+<tt>pcontrol</tt> stores the stdout and stderr for each completed job.
+To retrieve these data from these streams, the user issues the
+commands <tt>stdout (JobID)</tt> and <tt>stderr (JobID)</tt>.  The
+result is a single line specifying the number of bytes to expect,
+followed by a dump of the buffers, followed by the prompt. It is the
+user's responsibility to relieve pcontrol of this data load by
+deleting jobs once they are no longer needed.  Job deletion is
+performed with the command <tt>delete (JobID)</tt>.
+
+<p>
+Jobs are moved between the following states by pcontrol:
+<ul>
+<li> pending: the job has not yet been executed.
+<li> busy: the job is currently being executed.
+<li> done: the job has completed, but the stdout/stderr has not been processed by
+     pcontrol.
+<li> exit: the job has completed with a valid exit status
+<li>crash: the job has completed with a crash status (exit on signal).
+</ul>
+
+<h3>Miscellaneous Commands</h3>
+
+<p>
+It is possible to check the status of a single host or job with the
+user command <tt>check</tt>.
+
+<p>
+pcontrol continuously examines the stack of jobs, adjusting their
+state as needed and extracting their output when it is ready.  These
+checks are performed in the background, with pcontrol ready to accept
+further commands from the user in the foreground.  These checks are
+performed after every keystroke, and also after an inactivity timeout.
+The interrupt interval defaults to 1 second, but may be adjusted with
+the <tt>pulse</tt> command, which takes as an argument, the number of
+microseconds for the timeout.
+
+<p>
+the pcontrol system status may be examined with the command
+<tt>status</tt>.  This provides a dump of the job stacks and the host
+stacks.  
+
+<p>
+It is possible to list the jobs currently in a specific stack,
+corresponding to the list of jobs with a given state.  This is done
+with the command <tt>jobstack (stackname)</tt>.  The valid stack names are
+pending, busy, exit, crash, and done.  The result is a list of all
+jobs on the specified stack.  This is useful to determine quickly
+which jobs have exited or crashed.  
+
+<p>
+A specific job may be killed with the command <tt>kill (JobID)</tt>.
+This command is only valid for a job in the <tt>busy</tt> state.  Any
+job in the <tt>pending</tt>, <tt>exit</tt>, or <tt>crash</tt> state
+may be deleted with the <tt>delete (JobID)</tt> command.  This is
+necessary to free the memory associated with the job and its output
+streams.
+
+<p>
+The command <tt>verbose (mode)</tt> turns the verbosity of the
+pcontrol operations on or off.
+
+<p>
 The <tt>pcontrol</tt> and the IPP Image Server have related needs for
 information from the combined storage-and-processing nodes regarding
 which nodes are available.  It is not yet clear if this information is
-best stored in a single location (either <tt>pcontrol</tt> or IPP Image
-Server), which provides the information to other systems on demand, or
-if both systems should maintain the information.  Also, it may be
-necessary to distinguish nodes which are available for processing from
-those that are available to serve data as part of the IPP Image
-Server.
-
-The Controller maintains three tables of processing jobs: pending
-stages, active stages, and completed stages.  The pending stages are
-those which have not yet been performed.  The active stages are those
-currently being performed on one of the remote nodes.  The completed
-stages are those which have finished, either successfully or with an
-error state.  The Controller daemon monitors the collection of remote
-clients and sends them new pending stages when they become free.
-
-The <tt>pcontrol</tt> provides a mechanism for users (either other
-programs or humans) to interact with it.  The user interface provides
-commands to check the current processing job queues, the tables of
-successful and failed jobs, to stop or delete jobs, etc.
-
+best stored in a single location (either <tt>pcontrol</tt> or IPP
+Image Server), which provides the information to other systems on
+demand, or if both systems should maintain the information.  Also, it
+may be necessary to distinguish nodes which are available for
+processing from those that are available to serve data as part of the
+IPP Image Server.
+