pcontrol is the IPP parallel process controller.
Overview
The IPP uses a group of computers to store and process images and to
manipulate collections of detections. These computers perform any of
a large number of analysis stages or other processing tasks without
significant interprocess communication. It is necessary to have a
mechanism which initiates computing tasks on the different computers,
which monitors the tasks as they are executed, which handles the
output and the errors from these tasks, and which reacts to the
failure of any of the computing nodes. The system responsible for the
tasks in the IPP is pcontrol.
pcontrol interacts with the collection of computers under its
management and with other subsystems in the IPP. The IPP Controller
receives a variety of inputs from other subsystems, described below,
and initiates actions such as adding a new process to the queue of
pending tasks. pcontrol also provides information to other
subsystems on demand about its processing history and current state.
Each physical computer may have multiple processors; since
pcontrol is managing processing tasks, it treats each
processor independently. It is up to the system configuration if each
computer needs to reserve one of its CPUs to manage background tasks
or if pcontrol should attempt to send one task per CPU and
let the operating system handle the I/O load.
hosts
The Controller maintains a table of available processing computers
(hosts) and tracks their status. Hosts managed by
pcontrol are allowed to be in one of several states, and
pcontrol must interact with it in an appropriate way for each
of those states. A Node may be {\tt alive}, {\tt dead} or {\tt off}.
If the Node is {\tt alive}, it responds to commands from the IPP
Controller and may be used for tasks subject to other constraints. If
it is {\tt dead}, the Node is not responsive and must not be used for
executing tasks. pcontrol must identify Nodes which have
died (not responding) and occasionally test them to see if they are
{\tt alive} again. Nodes which are {\tt off} are not available for
tasks and must not be tested. Nodes may be set to the {\tt off} or
{\tt dead} states by external subsystems; it is the responsibility of
pcontrol to return a Node to the {\tt alive} state if
possible.
pcontrol must honor requests (normally from the users) to
change the mode of any computing node on demand between {\tt off} and
{\tt dead}. This would normally be done after a Node has been
rebooted and is released to pcontrol for its use. It
must also be able to change the list of allowed tasks as requested by
external commands.
Two example scenarios illustrate the transition between these states,
and the basic concept of operations for pcontrol. First,
imagine a computer crashes. At this point pcontrol should
detect that the Node is no longer responsive and mark it as {\tt
dead}. It should occasionally try to re-establish communication with
the Node, potentially with longer and longer delays between attempts.
A human could be notified if the Node seems to remain {\tt dead} for a
very long time. In another scenario, a person needs to work on a
Node. They notify pcontrol that the machine is {\tt off},
perhaps with a prior notification that the machine should be prepared
to go off. When work on the machine is complete, it should be placed
in the {\tt dead} state. Only when the person is done working and
testing the machine, and tells pcontrol that the machine is
now {\tt dead} can pcontrol attempt to re-start
communications and re-new processing operations on that Node.
pclient
When the Controller starts, it attempts to launch a Node Agent on each
of the available processing Nodes. Nodes which are not responsive are
marked as {\tt dead} so they may be re-tried. A Node Agent runs on
each of the individual nodes to execute the tasks as directed by the
Controller. The Node Agents communicate with the Controller via a
socket connection.
A Node Agent (which is only running on a Node in the {\tt alive}
state) may be in one of four modes: {\tt idle}, {\tt busy}, {\tt
done}, {\tt crash}. A Node Agent which is {\tt busy} currently has a
task assigned to it which is executing. The pcontrol may only
assign one task to a Node at a time. A Node Agent which is in the
{\tt idle} state may have a task assigned to it. When the Node Agent
detects that a tasks has finished, it changes to either the {\tt done}
or {\tt crash} states depending on the outcome of the process
execution. The pcontrol must also respect a list of task
restrictions which may require specific tasks to run on specific CPUs
or exclude specific tasks from specific CPUs.
A task being executed by the Node is run in the UNIX user space as a
forked process. The Node Agent must monitor the standard error and
standard output of the executing task and save them in separate
buffers. If the process exits or dies, the Node Agent must detect
this result and change state appropriately. The Node Agent must
respond to various commands from the Controller, as follows:
\paragraph{Report status}
The Node Agent returns its state ({\tt idle}, {\tt busy}, {\tt done},
{\tt crash}) and the exit status of the current processing task, if
available. The reported exit state, if the process has completed
without crashing, is the UNIX exit state reported by the task: 0--256
with 0 indicating a successful completion.
\paragraph{Report stdout}
Send and flush the current stdout buffer. The Node Agent will return
the complete contents of the stdout buffer via a buffered write and
flush the buffer when it is finished. The Node Agent will not accept
more data on the stdout buffer from the current processing task until
the send is complete and the buffer is flushed. The daemon must
accept all of the buffer output.
\paragraph{Report stderr}
Identical to `report stdout', but for stderr.
\paragraph{Kill task }
The Node Agent should send a kill signal (\code{KILL} or \code{TERM})
to the current processing task. When the processing task has exited,
the Node Agent should set its state to {\tt crash}.
\paragraph{Clear task}
The Node Agent should set its state {\tt idle}. If a processing stage
is currently running, it should be killed (\code{KILL} or \code{TERM})
before the task is cleared.
\paragraph{Start processing stage}
The Node Agent forks a specified command. The command should be a
standard UNIX command without command line redirection or
backgrounding. The task is run with the same user ID as the Node
Agent, which is also the same user ID as the Controller.
\subsubsection{Tasks}
The pcontrol accepts tasks from other IPP subsystems. The task
requests include the specific command to be executed and are in the
form of a UNIX command which could be performed on any of the
computing nodes. Any input or output data in the commands must be a
valid resource regardless of the node on which the task is executed.
Input and output data resources must be unique where necessary to
avoid conflicts. It is the responsibility of the task to wait for
network lags (ie, NFS delays). The pcontrol gives each task a
unique identifier, which is returned to the requesting entity. The
requestor may then use that ID to obtain status information on that
task or to send control signals to the specific task.
Task requests may specify a desired node for the task execution. The
pcontrol attempts to honor the request if the node is {\tt
alive}, but will execute it on another node if the requested one is
{\tt dead} or {\tt off}. Even if a node is {\tt alive}, the IPP
Controller will choose another node if the specified task is not
allowed on the requested node. In all other cases, the pcontrol
waits until the currently executing processes, and processes with
higher priority, are completed before executing the specified task on
the requested node.
Task requests may specify an urgency level. The pcontrol
determines the priority of the task on the basis of both the urgency
and the age of the request. An executing task must be completed on a
CPU before any new task is started on that CPU, regardless of
priority. The urgency levels range from 0 to 2. Tasks with an
urgency of 1 are scheduled whenever they reach the top of the stack.
Tasks with an urgency of 2 are sent immediately to the top of the
stack. Tasks assigned a priority of 0 are maintained in the queue and
never executed.
It may be useful for the Controller to distinguish between tasks
dominated by I/O and tasks dominated by data processing. It is
possible that one of each of these types of tasks may be sent to the
same node without significantly impacting the system performance.
Alternatively, it may be necessary to limit a single machine with 2
CPUs to only one of each of these types of tasks (i.e., one processor
will be working on I/O while the other is working on processing).
Such details will be studied by the IfA IPP Team.
The pcontrol monitors the output streams from the executing
tasks and the exit status of the tasks. Each task is associated with
a log file, to which all output is written. The status, including the
exit status, of each task is maintained by the pcontrol so that
other subsystems may determine if specific tasks have started or
completed.
\subsubsection{Controller Interfaces}
The pcontrol must accept commands from other IPP subsystems.
These commands include those which govern the processing of specified
tasks, those which govern the behavior of specific computing nodes,
and those which request information from the pcontrol. The IPP
Controller must be able to halt the execution of a specified task,
delete an unexecuted task from the task list, change the priority of
tasks, and change the requested nodes for tasks. The pcontrol
must also be able to stop the current execution of a task and push it
to the end of the queue and also change its priority.
The pcontrol must respond to informational requests regarding the
collection of machines and their states as well as the collection of
tasks and their states. The pcontrol must monitor the execution
times of the different tasks and provide summary statistics. Finally,
the pcontrol must respond to three top-level commands: {\tt finish},
{\tt stop} and {\tt abort}. When {\tt finish} is requested, no more
new tasks are accepted on the stack of task, and when all tasks in the
stack have completed, the pcontrol must exit. When {\tt stop} is
requested, the currently executing tasks must be completed at which
point the pcontrol must exit, but tasks remaining in the stack which
have not been started are flushed. When {\tt abort} is issued, the
pcontrol immediately kills all executing tasks and exits.
The pcontrol and the IPP Image Server have related needs for
information from the combined storage-and-processing nodes regarding
which nodes are available. It is not yet clear if this information is
best stored in a single location (either pcontrol or IPP Image
Server), which provides the information to other systems on demand, or
if both systems should maintain the information. Also, it may be
necessary to distinguish nodes which are available for processing from
those that are available to serve data as part of the IPP Image
Server.
The Controller maintains three tables of processing jobs: pending
stages, active stages, and completed stages. The pending stages are
those which have not yet been performed. The active stages are those
currently being performed on one of the remote nodes. The completed
stages are those which have finished, either successfully or with an
error state. The Controller daemon monitors the collection of remote
clients and sends them new pending stages when they become free.
The pcontrol provides a mechanism for users (either other
programs or humans) to interact with it. The user interface provides
commands to check the current processing job queues, the tables of
successful and failed jobs, to stop or delete jobs, etc.