Context Navigation

← Previous Change
Next Change →

pantasks.tex

Timestamp:

Jan 19, 2006, 12:58:19 AM (21 years ago)

Author:

eugene

Message:

updates

File:

: 1 edited

trunk/doc/pantasks/pantasks.tex (modified) (19 diffs)

Legend:

: Unmodified
: Added
: Removed

trunk/doc/pantasks/pantasks.tex

-              r6033
+              r6055
 tool which manages the sequencing of data analysis steps and, with the
 related tool `PControl', distributes the data processing across a
 cluster of computers.
+cluster of computers. \tbd{uses the 'opihi' shell-scripting system}
 The purpose of PanTasks is to manage the automatic construction and
 …
 uses a set of rules to define UNIX commands, and their corresponding
 command-line arguments, to be performed on some regular, repeated
+basis.  The utility of PanTasks is that it can easily define an
+analysis system which is completely state-based, as opposed to an
+event-driven system.
+The two basic units of PanTasks operation are the 'task' and the
+'job'.  A 'job' is simply a command the user would execute on a
+command line; it consists of a command along with optional command
+line arguments.  A 'task' is a generic description of a type of job in
+which the details of any specific piece of data are omitted.  The task
+defines the UNIX command which corresponds to the job, and it provides
+rules for determining the identity of data for the job.  The task also
+defines tests which are used to decide if the job may be executed.
+Finally, it defines a polling frequency in which PanTasks should
+attempt to construct a new job for the task.
+For example, we may want to regularly copy files from one location to
+another.  Perhaps the name of the next available file is available in
+a database table, and can be retrieved with the command 'nextFile'.
+Perhaps we wish to check for new files every 60 seconds.  Thus, the
+task is to copy files, while a specific job of this task might be of
+the form \code{cp newfile newpath}.  With PanTasks, we would define a
+copy task somewhat like the following:
+\begin{verbatim}
+  task copyfile
+    periods -exec 60.0
+    task.exec
+      $file = `nextFile`
+      if ($file == "none")
+        break
+      end
+      command cp $file $newpath
+    end
+    task.exit 0
+      queuepush copied $file
+    end
+    task.exit 1
+      queuepush failure $file
+    end
+  end
+\end{verbatim}
+In this simple example, the task is attempted every 60 seconds.  If
+there is no new file (output of 'nextFile' is 'none'), then no job
+results from this task.  If there is a new file, the copy command is
+performed.  This example is deceptively simple because one could
+easily imagine writing a stand-along program which performs the same
+thing.  The important advantages of using PanTasks for this type of
+operation are:
+\begin{itemize}
+\item the output from one job can be used to spawn a number of other tasks.
+\item the success or failure state of each job can be used to spawn
+  other tasks.
+\item the details of the job and data test are kept separated from the
+  rules which connect tasks together.
+\item the relationships between tasks are kept together in a single
+  location.
+\end{itemize}
+In addition to these organizational advantages, PanTasks also provides
+a direct connection to the tool for monitoring parallel jobs,
+pcontrol.  Thus the jobs spawned by PanTasks can be defined to run in
+the background locally or on any of the computers in the parallel
+processing cluster.
+\section{PanTasks : the Scheduler}
+The purpose of PanTasks is to manage the automatic construction and
+execution of inter-related (often repetative) operations. PanTasks uses
+a set of rules to define UNIX commands, and their corresponding
+command-line arguments, to be performed on some regular, repeated
+basis. The utility of PanTasks is that it can easily define an analysis
+system which is completely state-based, as opposed to an event-driven
+system.
+basis.  The utility of PanTasks is that it can easily define and
+manage an analysis system which is completely state-based, as opposed
+to an event-driven system.
 Consider, for example, a telescope which obtains a collection of
 …
 \subsection{Tasks vs Jobs}
+The primary function of PanTasks is to repeatedly perform tasks, and
+execute jobs on the basis of those tasks. A task consists of a set of
+rules which describe system state tests to perform on a regular time
+scale. Based on the results of those state tests, the task will then
+choose whether or not to construct a job. The task also defines
+actions to perform upon the completion of a job, based upon the output
+and exit status of the job. A task thus defines the repeat period. It
+may optionally define valid or invalid time ranges (eg, Mon-Fri or
+:00-17:00, etc). The task may also specify that the job be run
+locally (ie, in the background on the same computer as PanTasks) or
+remotely by the parallel process controller (pcontrol). A job may even
+be restricted to a specific computer managed by pcontrol. An example
+of a simple tasks is given below.
+\begin{verbatim}
+   task datalist
+     command ls /data/foo
+     periods -exec 5.0
+     periods -timeout 50.0
+     periods -poll 1.0
+     task.exit 0
+       queueprint stdout
+       queuedelete stdout
+     end
+     task.exit 1
+       queuepush failure "task failed"
+     end
+   end
+ \end{verbatim}
+This task does not perform any system state tests; it is simply
+constructs a new job every 5.0 seconds. The job in this case is always
+the same: ls /data/foo . When the job finished, if the job exit status
+is 0 (normal UNIX success status), the resulting output is printed to
+the screen. If the job returns an exit status of 1 (a failure), the
+failure queue receives a single entry. Although they are not defined
+in this case, it is also possible to specify the action to be taken if
+the job crashes (does not exit normally) or if it times out (runs
+beyond the specified timeout period). A slightly more complex task
+which performs a state test and constructs a command based on that
+test is shown below
+\begin{verbatim}
+   task datalist
+     periods -exec 5.0
+     periods -timeout 50.0
+     periods -poll 1.0
+     task.exec
+       $file = `next.file`
+       if ($file == "none")
+         break
+       end
+       command cp /data/foo/$file /data/bar
+     end
+     task.exit 0
+       queueprint stdout
+       queuedelete stdout
+       queuepush copied $file
+     end
+     task.exit 1
+       queuepush failure $file
+     end
+   end
+\end{verbatim}
+ The task.exec macro is executed by PanTasks every 5.0 seconds. This
+ macro executes a (hypothetical user-defined) UNIX command (next.file)
+ which examines the system state, return either a filename or the word
+ "none". If the result of this test is "none", the task does nothing:
+ no job is constructed. Otherwise, a job is constructed using the name
+ of the file returned by the state test. Successful jobs have the
+ filename added to the 'copied' queue, while failed jobs add the
+ filename to the 'failure' queue.
+The two basic units of PanTasks operation are the 'task' and the
+'job'.  A 'job' is simply a command the user would execute on a
+command line; it consists of a command along with optional command
+line arguments.  A 'task' is a generic description of a type of job in
+which the details of any specific piece of data are omitted.  The task
+defines the UNIX command which corresponds to the job, and it provides
+rules for determining the identity of data for the job.  The task also
+defines tests which are used to decide if the job may be executed.
+Finally, it defines a polling frequency in which PanTasks should
+attempt to construct a new job for the task.
+For example, we may want to regularly copy files from one location to
+another.  Perhaps the name of the next available file is available in
+a database table, and can be retrieved with the command 'nextFile'.
+Perhaps we wish to check for new files every 60 seconds.  Thus, the
+task is to copy files, while a specific job of this task might be of
+the form \code{cp newfile newpath}.  With PanTasks, we would define a
+copy task somewhat like the following:
+\begin{verbatim}
+  task copyfile
+    periods -exec 60.0
+    task.exec
+      $file = `nextFile`
+      if ($file == "none")
+        break
+      end
+      command cp $file $newpath
+    end
+    task.exit 0
+      queuepush copied $file
+    end
+    task.exit 1
+      queuepush failure $file
+    end
+  end
+\end{verbatim}
+In this simple example, the task is attempted every 60 seconds.  If
+there is no new file (output of 'nextFile' is 'none'), then no job
+results from this task.  If there is a new file, the copy command is
+performed.  This example is deceptively simple because one could
+easily imagine writing a stand-along program which performs the same
+thing.  The important advantages of using PanTasks for this type of
+operation are:
+\begin{itemize}
+\item the output from one job can be used to spawn a number of other tasks.
+\item the success or failure state of each job can be used to spawn
+  other tasks.
+\item the details of the job and data test are kept separated from the
+  rules which connect tasks together.
+\item the relationships between tasks are kept together in a single
+  location.
+\end{itemize}
+In addition to these organizational advantages, PanTasks also provides
+a direct connection to the tool for monitoring parallel jobs,
+pcontrol.  Thus the jobs spawned by PanTasks can be defined to run in
+the background locally or on any of the computers in the parallel
+processing cluster.
 \subsection{Parallel vs Local Job Processing}
 Jobs which are generated by PanTasks tasks may either be run locally
 (forked in the background on the same machine as PanTasks) or run on the
+IPP parallel process controller, pcontrol. The default is for the job
+to be run locally. If a job should be run on the parallel controller,
+this can be specified by including the command host (hostname) in the
+definition of a task. If the value of (hostname) is 'anyhost', then
+pcontrol may select any of its host computers to run the job according
 to its own rules. If the value of (hostname) is one of the computers
+managed by pcontrol, then that machine will be selected for the job,
+if it is available. This amounts to a preference to use that machine,
 but pcontrol is allowed to substitute a different machine if it
 chooses. If the host command is given the option -required, then
+pcontrol is forced to use the named host, even if the machine is down,
+unknown, or otherwise unavailable. If the machine is not available,
 pcontrol will simply hold onto the job until the machine is available
+or the job is deleted. Note that PanTasks may delete jobs from pcontrol
 if they remain pending for too long (see period -timeout).
+(forked in the background on the same machine as PanTasks) or run on
+the IPP parallel process controller, pcontrol. The default is for the
+job to be run locally. If a job should be run on the parallel
+controller, this can be specified by including the command host
+(hostname) in the definition of a task. If the value of (hostname) is
+'anyhost', then pcontrol may select any of its host computers to run
+the job according to its own rules. If the value of (hostname) is one
+of the computers managed by pcontrol, then that machine will be
+selected for the job, if it is available. This amounts to a preference
+to use that machine, but pcontrol is allowed to substitute a different
+machine if it chooses. If the host command is given the option
+-required, then pcontrol is forced to use the named host, even if the
+machine is down, unknown, or otherwise unavailable. If the machine is
+not available, pcontrol will simply hold onto the job until the
+machine is available or the job is deleted. Note that PanTasks may
+delete jobs from pcontrol if they remain pending for too long.
 It is possible to interact directly with the parallel processor to
 …
 identified to the controller. If such a host is required, the
 controller will simply keep the associated jobs in the pending state
 until such a machine exists. See the pcontrol documentation for
+until such a machine exists. See the pcontrol section below for
 further discussion of the controller manipuation of jobs and hosts.
 …
 \item external communications
 \item job exit status
 \item job stdout parsing
+\item job stdout/stderr parsing
 \end{itemize}
 …
 value of the stdout lines from the UNIX command, and the value
 \code{$var:n} is set to the number of output lines.
+% $
 Fine-grained control over the job exit status is available with the
 …
 status.
+Jobs may transmit their results back to PanTasks for further evaluation
+through the standard output and standard error streams. Whenever a job
+exits, the complete stdout and stderr streams from the job are pushed
+onto the PanTasks queues stdout and stderr. The job exit macros may then
+parse these queues, moving the results into other PanTasks / Opihi data
+containers (queues, variables, vectors, whatever is appropriate). Note
+that currently, the output data is simply pushed onto these output
+queues. It is currently the responsibility of the PanTasks programmer to
+use or dispose of the data in these queues. This may change in the
+future: the queues may be flushed for each job completion.
+Jobs may transmit their results back to PanTasks for further
+evaluation through the standard output and standard error
+streams. Whenever a job exits, the complete stdout and stderr streams
+from the job are pushed onto the PanTasks queues \code{stdout} and
+\code{stderr}. The job exit macros may then parse these queues, moving
+the results into other PanTasks / Opihi data containers (queues,
+variables, vectors, whatever is appropriate). Note that currently, the
+output data is simply pushed onto these output queues. It is currently
+the responsibility of the PanTasks programmer to use or dispose of the
+data in these queues. This may change in the future: the queues may be
+flushed for each job completion.
+\subsection{PanTasks Monitoring Loop}
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.01.ps}
+\caption{\label{MonitorLoop} PanTasks Monitor Loop}
+\end{center}
+\end{figure}
+Figure~\ref{MonitorLoop} illustrates the operation of PanTasks.
+Currently, PanTasks is run as a stand-alone foreground process, not a
+server.  In this current operating mode, PanTasks accepts commands
+from the user (left side) via a GNU readline interface.  Readline
+provides a mechanism to schedule an background process which is
+executed on a regular basis, or in the interval after each keystroke.
+This background processing is represented as the loop on the right.
+In this loop, PanTasks checks on the status of pcontrol and on any
+locally spawned background jobs.  It also checks the list of tasks for
+tasks which are ready to be executed.  The tasks and the background
+jobs are kept in rotating queues.  Every time the loop is processed,
+PanTasks runs through a number of the background jobs and tasks,
+attempting to evaluate as many as possible, but stopping after a
+limited number of milliseconds.  This test stage cycles through the
+tasks and/or jobs in their queue, but stops if it examines all of the
+entries in the queue.  If this testing process runs out of time before
+the entire queue is searched, it leaves the sequence intact for the
+next pass.  The timeout is set to keep the keyboard small enough to
+keep the human interaction from being troublesome.
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.02.ps}
+\caption{\label{TaskLoop} PanTasks Task Check Loop}
+\end{center}
+\end{figure}
+Figure~\ref{TaskLoop} shows the interactions performed for each check
+of the list of tasks.  The task timers are examined to see if the
+specific task is ready to be executed.  If so, then the task exec
+macro is executed.  If this macro exit status is false, the loop goes
+to the next task.  If the task exec macro is true, the job command is
+constructed and the job is submitted to the correct location, either
+the controller (pcontrol) or to the queue of local, background jobs.
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.03.ps}
+\caption{\label{JobLoop} PanTasks Job Check Loop}
+\end{center}
+\end{figure}
+Figure~\ref{JobLoop} shows the interactions performed for each check
+of a single background job.  The job timer is checked to see if the
+job has timed out.  Next, the job run status is examined to see if the
+job has finished or not.  If the job has finished, the appropriate
+exit macro is executed and the job results are returned to the rest of
+the PanTasks data structures (ie, stderr and stdout queues are filled
+out).  Users should not define exit macros which perform long running
+jobs, or PanTasks will become quite sluggish to keyboard strokes.
+PanTasks also checks the current status of pcontrol on each loop.
+In this test, PanTasks requests from pcontrol a list of the jobs which
+have finished.  It then attempts to harvest the finished jobs one by
+one and place the results in the approrpiate locations.  This loop is
+also limited to a fixed amount of time; any pcontrol jobs which remain
+unharvested are left for the next pass by PanTasks.
 \subsection{Running the scheduler}
 …
 \end{verbatim}
-\begin{figure}
-\begin{center}
-\includegraphics[scale=0.85]{pics/pantasks.01.ps}
-\caption{\label{queues} PanTasks queues and MDDB tables}
-\end{center}
-\end{figure}
-\begin{figure}
-\begin{center}
-\includegraphics[scale=0.85]{pics/pantasks.02.ps}
-\caption{\label{queues} PanTasks queues and MDDB tables}
-\end{center}
-\end{figure}
-\begin{figure}
-\begin{center}
-\includegraphics[scale=0.85]{pics/pantasks.03.ps}
-\caption{\label{queues} PanTasks queues and MDDB tables}
-\end{center}
-\end{figure}
 \newpage
 \section{pcontrol : the PanTasks parallel controller}
 …
 \subsection{Host States}
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.04.ps}
+\caption{\label{HostStates} PanTasks Host States}
+\end{center}
+\end{figure}
 pcontrol maintains a table of available processing computers (hosts)
 and tracks their status. Hosts managed by pcontrol are allowed to be
+in one of several states: off, down, idle, busy, and done. These
+states have the following meanings:
+If the host is off, it is known to pcontrol, but pcontrol does not
+have an active connection to the machine. Hosts which are off are not
+available for jobs, and pcontrol does not attempt to initiate a
+connection to them.
+in one of several states: \code{off}, \code{down}, \code{idle},
+\code{busy}, and \code{done} (see Figure~\ref{HostStates}). There are
+also two virtual states: \code{new} and \code{delete}.  These states
+have the following meanings:
+If the host is \code{off}, it is known to pcontrol, but pcontrol does
+not have an active connection to the machine. Hosts which are
+\code{off} are not available for jobs, and pcontrol does not attempt to
+initiate a connection to them.  A pcontrol user may force a host to
+transition to the \code{off} state with the command host
+\code{off~(hostname)}. (Note that this command will set only one of
+the connections to the named host to \code{off}. If multiple
+connections to a machine have been defined, multiple \code{off}
+commands must be sent).
 When pcontrol is told to consider a machine on, the machine is moved
+from the off state to the down state. Pcontrol attempts to initiate a
+connection to the host. Connections are made by running a remote
+client on the host, using the specified connection method. The
+connection method may be ssh, rsh, or an equivalent remote shell
+connection. The choice is specified by the COMMAND Opihi variable. The
+remote connection starts a dedicated remote client which must accept
+the pcontrol client commands and respond appropriately. The provided
+remote client is called pclient, though in principal other equivalent
+programs could be used by setting the Opihi variable SHELL (this
+feature more generally allows a user to specify a path to the remote
+client, if it is not in the user's path). A pcontrol user may force a
+host to transition to the off state with the command host off
+(hostname). ( Note that this command will set only one of the
+connections to the named host to off. If multiple connections to a
+machine have been defined, multiple off commands must be sent).
+from the \code{off} state to the \code{down} state. Pcontrol attempts
+to initiate a connection to the host. Connections are made by running
+a remote client on the host, using the specified connection
+method. The connection method may be ssh, rsh, or an equivalent remote
+shell connection. The choice is specified by the \code{COMMAND} Opihi
+variable. The remote connection starts a dedicated remote client which
+must accept the pcontrol client commands and respond
+appropriately. The provided remote client is called {\em pclient},
+though in principal other equivalent programs could be used by setting
+the Opihi variable \code{SHELL}.  This feature more generally allows a
+user to specify a path to the remote client, if it is not in the
+user's path.
 If the remote connection is successful, the connected host is moved by
 pcontrol from the down state to the idle state. If the connection is
+unsuccessful, pcontrol will try again after a certain period of
+time. If the connection continues to be unsuccessful, the retry period
+is doubled for each successiver connection attempt. If the user wants
 to force pcontrol to retry the connection to a machine (if, for
 example, the timeout is now very long, but the user knows the
+pcontrol from the \code{down} state to the \code{idle} state. If the
+connection is unsuccessful, pcontrol will try again after a certain
+period of time. If the connection continues to be unsuccessful, the
+retry period is doubled for each successiver connection attempt. If
+the user wants to force pcontrol to retry the connection to a machine
+(if, for example, the timeout is now very long, but the user knows the
 machine's ethernet cable has been re-inserted...), this can be
+achieved with the command host retry (hostname). A host which is down
+is in the limbo state between off and idle.
+achieved with the command host \code{retry (hostname)}. A host which
+is \code{down} is in the limbo state between \code{off} and
+\code{idle}.
 Once pcontrol has made a successful connection to the host, the host
+is in the idle state. At this point, it is ready to accept jobs from
+pcontrol for execution. Pcontrol repeatedly queries the hosts to check
+that they are still alive. If a host is discovered to be unresponsive,
+and particularly if the remote pipe connection has closed, then the
+machine is moved back to the down state.
+Hosts which are idle may accept a job from pcontrol. A job simply
+consists of a bare UNIX command, without redirection of standard input
+or standard output. The host will initiate the job, and pcontrol will
+place the host into the busy state. The remote client, pclient, runs
+the job in the background and will continue to accept input from
+pcontrol. pcontrol will continue to check the status of the host, and
+now also the status of the specific job. As before, if the connection
+breaks, pcontrol will migrate the host to the down state. Any job
+already initiated on a host which goes down will be returned for later
+processing, so the job will not be lost.
+is in the \code{idle} state. At this point, it is ready to accept jobs
+from pcontrol for execution. Pcontrol periodically queries the hosts
+to check that they are still alive. If a host is discovered to be
+unresponsive, and particularly if the remote pipe connection has
+closed, then the machine is moved back to the \code{down} state.
+Hosts which are \code{idle} may accept a job from pcontrol. A job
+simply consists of a bare UNIX command, without redirection of
+standard input or standard output. The host will initiate the job, and
+pcontrol will place the host into the \code{busy} state. The remote
+client, pclient, runs the job in the background and will continue to
+accept input from pcontrol. pcontrol will continue to check the status
+of the host, and now also the status of the specific job. As before,
+if the connection breaks, pcontrol will migrate the host to the
+\code{down} state. Any job already initiated on a host which goes down
+will be returned to the pool of unexecuted jobs for later processing,
+so the job will not be lost.
 When the job exits, pclient tells pcontrol that the job is completed,
 and specifies the exit status. At this point, pcontrol will move the
+host from busy to done state. It will stay in this state until
+pcontrol can determine the ending conditions and reset the remote
+client. pcontrol requests the standard error and standard output from
+the job from pclient. pcontrol stores this data with its information
+about the completed job, and send a reset command to the remote
+client. Once these cleanup tasks are successfully completed, pcontrol
+will move the host to the idle state, ready for further jobs.
+host from \code{busy} to \code{done} state. It will stay in this state
+until pcontrol can determine the ending conditions and reset the
+remote client. pcontrol requests the standard error and standard
+output from the job from pclient. pcontrol stores this data with its
+information about the completed job, and send a reset command to the
+remote client. Once these cleanup tasks are successfully completed,
+pcontrol will move the host to the \code{idle} state, ready for
+further jobs.
 Each physical computer may have multiple processors. pcontrol treats
 …
 \subsection{Jobs}
+The pcontrol accepts new jobs with the command job ..., in which the
+ellipsis represents the command and arguments of a valid UNIX
+command. The commands are run under sh, and are executed in the user's
+home directory. (If it is desired, we can easily add a command to tell
+pclient to perform cd). Users should be wary of the conditions under
+which the remote jobs are run. If the nodes in question all
+The pcontrol accepts new jobs with the command \code{job ...}, in
+which the ellipsis represents the command and arguments of a valid
+UNIX command. The commands are run under \code{sh}, and are executed
+in the user's home directory. Users should be wary of the conditions
+under which the remote jobs are run. If the nodes in question all
 cross-mount the same home directories, multiple jobs which interact
 with the same named file may produce unexpected results. The
 …
 not realize that specific jobs do not behave the same on all machines,
 or if a necessary resource (eg, some input data file) is only
 available or accessible from some of the hosts. It is the
+available or accessible from some of the hosts. It is also the
 responsibility of the task to wait for network lags (ie, NFS delays).
 …
 pcontrol, the command echoes back the Job ID. This ID may be used by
 other pcontrol commands to obtain information about or interact with
+the job.
+the job.  For example, PanTasks uses the Job ID to examine specific
+jobs.
 A job may specify a specific host for the task execution. The host
 …
 (JobID).
+Jobs are moved between the following states by pcontrol:
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.05.ps}
+\caption{\label{queues} pcontrol job states.  Transitions labeled Ux
+  are issued by the pcontrol user (including PanTasks).  Transitions
+  labeled Px are initiated by pcontrol.  Transitions labeled Tx are
+  the result of the job completion}
+\end{center}
+\end{figure}
+Jobs are moved between the following states by pcontrol (see
+Figure~\ref{JobStates}):
 \begin{itemize}
 \item pending: the job has not yet been executed.
 …
 It is possible to check the status of a single host or job with the
 user command check.
+user command \code{check}.
 pcontrol continuously examines the stack of jobs, adjusting their
 …
 number of microseconds for the timeout.
 the pcontrol system status may be examined with the command
+The pcontrol system status may be examined with the command
 status. This provides a dump of the job stacks and the host stacks.
 …
 have exited or crashed.
+A specific job may be killed with the command kill (JobID). This
+command is only valid for a job in the busy state. Any job in the
+pending, exit, or crash state may be deleted with the delete (JobID)
+command. This is necessary to free the memory associated with the job
+and its output streams.
+The command verbose (mode) turns the verbosity of the pcontrol
+A specific job may be killed with the command \code{kill
+(JobID)}. This command is only valid for a job in the busy state. Any
+job in the pending, exit, or crash state may be deleted with the
+\code{delete (JobID)} command. This is necessary to free the memory
+associated with the job and its output streams.  PanTasks
+automatically performs these job harvesting functions.
+The command \code{verbose (mode)} turns the verbosity of the pcontrol
 operations on or off.
+The pcontrol and the IPP Image Server have related needs for
+information from the combined storage-and-processing nodes regarding
+which nodes are available. It is not yet clear if this information is
+best stored in a single location (either pcontrol or IPP Image
+Server), which provides the information to other systems on demand, or
+if both systems should maintain the information. Also, it may be
+necessary to distinguish nodes which are available for processing from
+those that are available to serve data as part of the IPP Image
+Server.
+\subsection{Command Summary}
+The pcontrol and Nebulous have related needs for information from the
+combined storage-and-processing nodes regarding which nodes are
+available. Currently, the two systems independently examine the
+hardware to judge the availability.  It is not yet clear if this
+information is best stored in a single location (either pcontrol or
+Nebulous), which provides the information to other systems on demand.
+The current implementation allows the IPP system as a whole to
+distinguih nodes which are available for processing from those that
+are available to serve data as part of Nebulous.
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.06.ps}
+\caption{\label{PControlLoop} PControl Job Monitor Loop}
+\end{center}
+\end{figure}
+Figure~\ref{PControlLoop} illustrated the pcontrol job monitor loop.
+This is very similar to the loop in PanTasks.  A background process
+launched by readline during idle periods cycles through the list of
+hosts (children) and migrates jobs to and from them as needed.
+\subsection{pcontrol Command Summary}
 \begin{verbatim}
 …
 \end{verbatim}
-\begin{figure}
-\begin{center}
-\includegraphics[scale=0.85]{pics/pantasks.04.ps}
-\caption{\label{queues} PanTasks queues and MDDB tables}
-\end{center}
-\end{figure}
-\begin{figure}
-\begin{center}
-\includegraphics[scale=0.85]{pics/pantasks.05.ps}
-\caption{\label{queues} PanTasks queues and MDDB tables}
-\end{center}
-\end{figure}
-\begin{figure}
-\begin{center}
-\includegraphics[scale=0.85]{pics/pantasks.06.ps}
-\caption{\label{queues} PanTasks queues and MDDB tables}
-\end{center}
-\end{figure}
 \newpage
 \section{pclient}
 …
 recommended practice is to set up ssh to allow the connection to the
 remote host without additional authentication using the appropriate
+authorized keys (see this article on ssh issues).
+authorized keys.  The security is maintained by the security of ssh
+logins to the computers used by PanTask and pcontrol.
 It is convenient to keep a continuous connection to the remote
 …
 \begin{figure}
 \begin{center}
 \includegraphics[scale=0.85]{pics/pantasks.07.ps}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.07.ps}
 \caption{\label{queues} PanTasks queues and MDDB tables}
 \end{center}

Note: See TracChangeset for help on using the changeset viewer.

Context Navigation

Changeset 6055 for trunk/doc/pantasks/pantasks.tex

Legend:

trunk/doc/pantasks/pantasks.tex

Download in other formats: