Index: trunk/doc/pantasks/pantasks.tex
===================================================================
--- trunk/doc/pantasks/pantasks.tex	(revision 6033)
+++ trunk/doc/pantasks/pantasks.tex	(revision 6055)
@@ -24,5 +24,5 @@
 tool which manages the sequencing of data analysis steps and, with the
 related tool `PControl', distributes the data processing across a
-cluster of computers.
+cluster of computers. \tbd{uses the 'opihi' shell-scripting system}
 
 The purpose of PanTasks is to manage the automatic construction and
@@ -30,82 +30,7 @@
 uses a set of rules to define UNIX commands, and their corresponding
 command-line arguments, to be performed on some regular, repeated
-basis.  The utility of PanTasks is that it can easily define an
-analysis system which is completely state-based, as opposed to an
-event-driven system.  
-
-The two basic units of PanTasks operation are the 'task' and the
-'job'.  A 'job' is simply a command the user would execute on a
-command line; it consists of a command along with optional command
-line arguments.  A 'task' is a generic description of a type of job in
-which the details of any specific piece of data are omitted.  The task
-defines the UNIX command which corresponds to the job, and it provides
-rules for determining the identity of data for the job.  The task also
-defines tests which are used to decide if the job may be executed.
-Finally, it defines a polling frequency in which PanTasks should
-attempt to construct a new job for the task.
-
-For example, we may want to regularly copy files from one location to
-another.  Perhaps the name of the next available file is available in
-a database table, and can be retrieved with the command 'nextFile'.
-Perhaps we wish to check for new files every 60 seconds.  Thus, the
-task is to copy files, while a specific job of this task might be of
-the form \code{cp newfile newpath}.  With PanTasks, we would define a
-copy task somewhat like the following:
-
-\begin{verbatim}
-  task copyfile
-    periods -exec 60.0
-
-    task.exec
-      $file = `nextFile`
-      if ($file == "none")
-        break
-      end
-      command cp $file $newpath
-    end
-
-    task.exit 0
-      queuepush copied $file
-    end
-
-    task.exit 1
-      queuepush failure $file
-    end
-  end
-\end{verbatim}
-
-In this simple example, the task is attempted every 60 seconds.  If
-there is no new file (output of 'nextFile' is 'none'), then no job
-results from this task.  If there is a new file, the copy command is
-performed.  This example is deceptively simple because one could
-easily imagine writing a stand-along program which performs the same
-thing.  The important advantages of using PanTasks for this type of
-operation are: 
-\begin{itemize}
-\item the output from one job can be used to spawn a number of other tasks.
-\item the success or failure state of each job can be used to spawn
-  other tasks.
-\item the details of the job and data test are kept separated from the
-  rules which connect tasks together.  
-\item the relationships between tasks are kept together in a single
-  location.
-\end{itemize}
-
-In addition to these organizational advantages, PanTasks also provides
-a direct connection to the tool for monitoring parallel jobs,
-pcontrol.  Thus the jobs spawned by PanTasks can be defined to run in
-the background locally or on any of the computers in the parallel
-processing cluster.
-
-
-\section{PanTasks : the Scheduler}
-
-The purpose of PanTasks is to manage the automatic construction and
-execution of inter-related (often repetative) operations. PanTasks uses
-a set of rules to define UNIX commands, and their corresponding
-command-line arguments, to be performed on some regular, repeated
-basis. The utility of PanTasks is that it can easily define an analysis
-system which is completely state-based, as opposed to an event-driven
-system.
+basis.  The utility of PanTasks is that it can easily define and
+manage an analysis system which is completely state-based, as opposed
+to an event-driven system.
 
 Consider, for example, a telescope which obtains a collection of
@@ -149,103 +74,87 @@
 \subsection{Tasks vs Jobs}
 
-The primary function of PanTasks is to repeatedly perform tasks, and
-execute jobs on the basis of those tasks. A task consists of a set of
-rules which describe system state tests to perform on a regular time
-scale. Based on the results of those state tests, the task will then
-choose whether or not to construct a job. The task also defines
-actions to perform upon the completion of a job, based upon the output
-and exit status of the job. A task thus defines the repeat period. It
-may optionally define valid or invalid time ranges (eg, Mon-Fri or
-10:00-17:00, etc). The task may also specify that the job be run
-locally (ie, in the background on the same computer as PanTasks) or
-remotely by the parallel process controller (pcontrol). A job may even
-be restricted to a specific computer managed by pcontrol. An example
-of a simple tasks is given below.
-
-\begin{verbatim}
-   task datalist
-     command ls /data/foo
-     periods -exec 5.0
-     periods -timeout 50.0
-     periods -poll 1.0
- 
-     task.exit 0
-       queueprint stdout
-       queuedelete stdout
-     end
-  
-     task.exit 1
-       queuepush failure "task failed"
-     end
-   end
- \end{verbatim}
-
-
-This task does not perform any system state tests; it is simply
-constructs a new job every 5.0 seconds. The job in this case is always
-the same: ls /data/foo . When the job finished, if the job exit status
-is 0 (normal UNIX success status), the resulting output is printed to
-the screen. If the job returns an exit status of 1 (a failure), the
-failure queue receives a single entry. Although they are not defined
-in this case, it is also possible to specify the action to be taken if
-the job crashes (does not exit normally) or if it times out (runs
-beyond the specified timeout period). A slightly more complex task
-which performs a state test and constructs a command based on that
-test is shown below
-
-\begin{verbatim}
-   task datalist
-     periods -exec 5.0
-     periods -timeout 50.0
-     periods -poll 1.0
- 
-     task.exec 
-       $file = `next.file`
-       if ($file == "none")
-         break
-       end
-       command cp /data/foo/$file /data/bar
-     end
- 
-     task.exit 0
-       queueprint stdout
-       queuedelete stdout
-       queuepush copied $file
-     end
-  
-     task.exit 1
-       queuepush failure $file
-     end
-   end
-\end{verbatim}
-
- The task.exec macro is executed by PanTasks every 5.0 seconds. This
- macro executes a (hypothetical user-defined) UNIX command (next.file)
- which examines the system state, return either a filename or the word
- "none". If the result of this test is "none", the task does nothing:
- no job is constructed. Otherwise, a job is constructed using the name
- of the file returned by the state test. Successful jobs have the
- filename added to the 'copied' queue, while failed jobs add the
- filename to the 'failure' queue.
+The two basic units of PanTasks operation are the 'task' and the
+'job'.  A 'job' is simply a command the user would execute on a
+command line; it consists of a command along with optional command
+line arguments.  A 'task' is a generic description of a type of job in
+which the details of any specific piece of data are omitted.  The task
+defines the UNIX command which corresponds to the job, and it provides
+rules for determining the identity of data for the job.  The task also
+defines tests which are used to decide if the job may be executed.
+Finally, it defines a polling frequency in which PanTasks should
+attempt to construct a new job for the task.
+
+For example, we may want to regularly copy files from one location to
+another.  Perhaps the name of the next available file is available in
+a database table, and can be retrieved with the command 'nextFile'.
+Perhaps we wish to check for new files every 60 seconds.  Thus, the
+task is to copy files, while a specific job of this task might be of
+the form \code{cp newfile newpath}.  With PanTasks, we would define a
+copy task somewhat like the following:
+
+\begin{verbatim}
+  task copyfile
+    periods -exec 60.0
+
+    task.exec
+      $file = `nextFile`
+      if ($file == "none")
+        break
+      end
+      command cp $file $newpath
+    end
+
+    task.exit 0
+      queuepush copied $file
+    end
+
+    task.exit 1
+      queuepush failure $file
+    end
+  end
+\end{verbatim}
+
+In this simple example, the task is attempted every 60 seconds.  If
+there is no new file (output of 'nextFile' is 'none'), then no job
+results from this task.  If there is a new file, the copy command is
+performed.  This example is deceptively simple because one could
+easily imagine writing a stand-along program which performs the same
+thing.  The important advantages of using PanTasks for this type of
+operation are: 
+\begin{itemize}
+\item the output from one job can be used to spawn a number of other tasks.
+\item the success or failure state of each job can be used to spawn
+  other tasks.
+\item the details of the job and data test are kept separated from the
+  rules which connect tasks together.  
+\item the relationships between tasks are kept together in a single
+  location.
+\end{itemize}
+
+In addition to these organizational advantages, PanTasks also provides
+a direct connection to the tool for monitoring parallel jobs,
+pcontrol.  Thus the jobs spawned by PanTasks can be defined to run in
+the background locally or on any of the computers in the parallel
+processing cluster.
 
 \subsection{Parallel vs Local Job Processing}
 
 Jobs which are generated by PanTasks tasks may either be run locally
-(forked in the background on the same machine as PanTasks) or run on the
-IPP parallel process controller, pcontrol. The default is for the job
-to be run locally. If a job should be run on the parallel controller,
-this can be specified by including the command host (hostname) in the
-definition of a task. If the value of (hostname) is 'anyhost', then
-pcontrol may select any of its host computers to run the job according
-to its own rules. If the value of (hostname) is one of the computers
-managed by pcontrol, then that machine will be selected for the job,
-if it is available. This amounts to a preference to use that machine,
-but pcontrol is allowed to substitute a different machine if it
-chooses. If the host command is given the option -required, then
-pcontrol is forced to use the named host, even if the machine is down,
-unknown, or otherwise unavailable. If the machine is not available,
-pcontrol will simply hold onto the job until the machine is available
-or the job is deleted. Note that PanTasks may delete jobs from pcontrol
-if they remain pending for too long (see period -timeout).
+(forked in the background on the same machine as PanTasks) or run on
+the IPP parallel process controller, pcontrol. The default is for the
+job to be run locally. If a job should be run on the parallel
+controller, this can be specified by including the command host
+(hostname) in the definition of a task. If the value of (hostname) is
+'anyhost', then pcontrol may select any of its host computers to run
+the job according to its own rules. If the value of (hostname) is one
+of the computers managed by pcontrol, then that machine will be
+selected for the job, if it is available. This amounts to a preference
+to use that machine, but pcontrol is allowed to substitute a different
+machine if it chooses. If the host command is given the option
+-required, then pcontrol is forced to use the named host, even if the
+machine is down, unknown, or otherwise unavailable. If the machine is
+not available, pcontrol will simply hold onto the job until the
+machine is available or the job is deleted. Note that PanTasks may
+delete jobs from pcontrol if they remain pending for too long.
 
 It is possible to interact directly with the parallel processor to
@@ -287,5 +196,5 @@
 identified to the controller. If such a host is required, the
 controller will simply keep the associated jobs in the pending state
-until such a machine exists. See the pcontrol documentation for
+until such a machine exists. See the pcontrol section below for
 further discussion of the controller manipuation of jobs and hosts.
 
@@ -373,5 +282,5 @@
 \item external communications
 \item job exit status
-\item job stdout parsing 
+\item job stdout/stderr parsing 
 \end{itemize}
 
@@ -399,4 +308,5 @@
 value of the stdout lines from the UNIX command, and the value
 \code{$var:n} is set to the number of output lines.
+% $
 
 Fine-grained control over the job exit status is available with the
@@ -411,14 +321,82 @@
 status.
 
-Jobs may transmit their results back to PanTasks for further evaluation
-through the standard output and standard error streams. Whenever a job
-exits, the complete stdout and stderr streams from the job are pushed
-onto the PanTasks queues stdout and stderr. The job exit macros may then
-parse these queues, moving the results into other PanTasks / Opihi data
-containers (queues, variables, vectors, whatever is appropriate). Note
-that currently, the output data is simply pushed onto these output
-queues. It is currently the responsibility of the PanTasks programmer to
-use or dispose of the data in these queues. This may change in the
-future: the queues may be flushed for each job completion.
+Jobs may transmit their results back to PanTasks for further
+evaluation through the standard output and standard error
+streams. Whenever a job exits, the complete stdout and stderr streams
+from the job are pushed onto the PanTasks queues \code{stdout} and
+\code{stderr}. The job exit macros may then parse these queues, moving
+the results into other PanTasks / Opihi data containers (queues,
+variables, vectors, whatever is appropriate). Note that currently, the
+output data is simply pushed onto these output queues. It is currently
+the responsibility of the PanTasks programmer to use or dispose of the
+data in these queues. This may change in the future: the queues may be
+flushed for each job completion.
+
+\subsection{PanTasks Monitoring Loop}
+
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.01.ps}
+\caption{\label{MonitorLoop} PanTasks Monitor Loop}
+\end{center}
+\end{figure}
+
+Figure~\ref{MonitorLoop} illustrates the operation of PanTasks.
+Currently, PanTasks is run as a stand-alone foreground process, not a
+server.  In this current operating mode, PanTasks accepts commands
+from the user (left side) via a GNU readline interface.  Readline
+provides a mechanism to schedule an background process which is
+executed on a regular basis, or in the interval after each keystroke.
+This background processing is represented as the loop on the right.
+In this loop, PanTasks checks on the status of pcontrol and on any
+locally spawned background jobs.  It also checks the list of tasks for
+tasks which are ready to be executed.  The tasks and the background
+jobs are kept in rotating queues.  Every time the loop is processed,
+PanTasks runs through a number of the background jobs and tasks,
+attempting to evaluate as many as possible, but stopping after a
+limited number of milliseconds.  This test stage cycles through the
+tasks and/or jobs in their queue, but stops if it examines all of the
+entries in the queue.  If this testing process runs out of time before
+the entire queue is searched, it leaves the sequence intact for the
+next pass.  The timeout is set to keep the keyboard small enough to
+keep the human interaction from being troublesome.
+
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.02.ps}
+\caption{\label{TaskLoop} PanTasks Task Check Loop}
+\end{center}
+\end{figure}
+
+Figure~\ref{TaskLoop} shows the interactions performed for each check
+of the list of tasks.  The task timers are examined to see if the
+specific task is ready to be executed.  If so, then the task exec
+macro is executed.  If this macro exit status is false, the loop goes
+to the next task.  If the task exec macro is true, the job command is
+constructed and the job is submitted to the correct location, either
+the controller (pcontrol) or to the queue of local, background jobs.
+
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.03.ps}
+\caption{\label{JobLoop} PanTasks Job Check Loop}
+\end{center}
+\end{figure}
+
+Figure~\ref{JobLoop} shows the interactions performed for each check
+of a single background job.  The job timer is checked to see if the
+job has timed out.  Next, the job run status is examined to see if the
+job has finished or not.  If the job has finished, the appropriate
+exit macro is executed and the job results are returned to the rest of
+the PanTasks data structures (ie, stderr and stdout queues are filled
+out).  Users should not define exit macros which perform long running
+jobs, or PanTasks will become quite sluggish to keyboard strokes.
+
+PanTasks also checks the current status of pcontrol on each loop.
+In this test, PanTasks requests from pcontrol a list of the jobs which
+have finished.  It then attempts to harvest the finished jobs one by
+one and place the results in the approrpiate locations.  This loop is
+also limited to a fixed amount of time; any pcontrol jobs which remain
+unharvested are left for the next pass by PanTasks.
 
 \subsection{Running the scheduler}
@@ -458,25 +436,4 @@
 \end{verbatim}
 
-\begin{figure}
-\begin{center}
-\includegraphics[scale=0.85]{pics/pantasks.01.ps}
-\caption{\label{queues} PanTasks queues and MDDB tables}
-\end{center}
-\end{figure}
-
-\begin{figure}
-\begin{center}
-\includegraphics[scale=0.85]{pics/pantasks.02.ps}
-\caption{\label{queues} PanTasks queues and MDDB tables}
-\end{center}
-\end{figure}
-
-\begin{figure}
-\begin{center}
-\includegraphics[scale=0.85]{pics/pantasks.03.ps}
-\caption{\label{queues} PanTasks queues and MDDB tables}
-\end{center}
-\end{figure}
-
 \newpage
 \section{pcontrol : the PanTasks parallel controller}
@@ -494,69 +451,83 @@
 \subsection{Host States}
 
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.04.ps}
+\caption{\label{HostStates} PanTasks Host States}
+\end{center}
+\end{figure}
+
 pcontrol maintains a table of available processing computers (hosts)
 and tracks their status. Hosts managed by pcontrol are allowed to be
-in one of several states: off, down, idle, busy, and done. These
-states have the following meanings:
-
-If the host is off, it is known to pcontrol, but pcontrol does not
-have an active connection to the machine. Hosts which are off are not
-available for jobs, and pcontrol does not attempt to initiate a
-connection to them.
+in one of several states: \code{off}, \code{down}, \code{idle},
+\code{busy}, and \code{done} (see Figure~\ref{HostStates}). There are
+also two virtual states: \code{new} and \code{delete}.  These states
+have the following meanings:
+
+If the host is \code{off}, it is known to pcontrol, but pcontrol does
+not have an active connection to the machine. Hosts which are
+\code{off} are not available for jobs, and pcontrol does not attempt to
+initiate a connection to them.  A pcontrol user may force a host to
+transition to the \code{off} state with the command host
+\code{off~(hostname)}. (Note that this command will set only one of
+the connections to the named host to \code{off}. If multiple
+connections to a machine have been defined, multiple \code{off}
+commands must be sent).
 
 When pcontrol is told to consider a machine on, the machine is moved
-from the off state to the down state. Pcontrol attempts to initiate a
-connection to the host. Connections are made by running a remote
-client on the host, using the specified connection method. The
-connection method may be ssh, rsh, or an equivalent remote shell
-connection. The choice is specified by the COMMAND Opihi variable. The
-remote connection starts a dedicated remote client which must accept
-the pcontrol client commands and respond appropriately. The provided
-remote client is called pclient, though in principal other equivalent
-programs could be used by setting the Opihi variable SHELL (this
-feature more generally allows a user to specify a path to the remote
-client, if it is not in the user's path). A pcontrol user may force a
-host to transition to the off state with the command host off
-(hostname). ( Note that this command will set only one of the
-connections to the named host to off. If multiple connections to a
-machine have been defined, multiple off commands must be sent).
+from the \code{off} state to the \code{down} state. Pcontrol attempts
+to initiate a connection to the host. Connections are made by running
+a remote client on the host, using the specified connection
+method. The connection method may be ssh, rsh, or an equivalent remote
+shell connection. The choice is specified by the \code{COMMAND} Opihi
+variable. The remote connection starts a dedicated remote client which
+must accept the pcontrol client commands and respond
+appropriately. The provided remote client is called {\em pclient},
+though in principal other equivalent programs could be used by setting
+the Opihi variable \code{SHELL}.  This feature more generally allows a
+user to specify a path to the remote client, if it is not in the
+user's path.
 
 If the remote connection is successful, the connected host is moved by
-pcontrol from the down state to the idle state. If the connection is
-unsuccessful, pcontrol will try again after a certain period of
-time. If the connection continues to be unsuccessful, the retry period
-is doubled for each successiver connection attempt. If the user wants
-to force pcontrol to retry the connection to a machine (if, for
-example, the timeout is now very long, but the user knows the
+pcontrol from the \code{down} state to the \code{idle} state. If the
+connection is unsuccessful, pcontrol will try again after a certain
+period of time. If the connection continues to be unsuccessful, the
+retry period is doubled for each successiver connection attempt. If
+the user wants to force pcontrol to retry the connection to a machine
+(if, for example, the timeout is now very long, but the user knows the
 machine's ethernet cable has been re-inserted...), this can be
-achieved with the command host retry (hostname). A host which is down
-is in the limbo state between off and idle.
+achieved with the command host \code{retry (hostname)}. A host which
+is \code{down} is in the limbo state between \code{off} and
+\code{idle}.
 
 Once pcontrol has made a successful connection to the host, the host
-is in the idle state. At this point, it is ready to accept jobs from
-pcontrol for execution. Pcontrol repeatedly queries the hosts to check
-that they are still alive. If a host is discovered to be unresponsive,
-and particularly if the remote pipe connection has closed, then the
-machine is moved back to the down state.
-
-Hosts which are idle may accept a job from pcontrol. A job simply
-consists of a bare UNIX command, without redirection of standard input
-or standard output. The host will initiate the job, and pcontrol will
-place the host into the busy state. The remote client, pclient, runs
-the job in the background and will continue to accept input from
-pcontrol. pcontrol will continue to check the status of the host, and
-now also the status of the specific job. As before, if the connection
-breaks, pcontrol will migrate the host to the down state. Any job
-already initiated on a host which goes down will be returned for later
-processing, so the job will not be lost.
+is in the \code{idle} state. At this point, it is ready to accept jobs
+from pcontrol for execution. Pcontrol periodically queries the hosts
+to check that they are still alive. If a host is discovered to be
+unresponsive, and particularly if the remote pipe connection has
+closed, then the machine is moved back to the \code{down} state.
+
+Hosts which are \code{idle} may accept a job from pcontrol. A job
+simply consists of a bare UNIX command, without redirection of
+standard input or standard output. The host will initiate the job, and
+pcontrol will place the host into the \code{busy} state. The remote
+client, pclient, runs the job in the background and will continue to
+accept input from pcontrol. pcontrol will continue to check the status
+of the host, and now also the status of the specific job. As before,
+if the connection breaks, pcontrol will migrate the host to the
+\code{down} state. Any job already initiated on a host which goes down
+will be returned to the pool of unexecuted jobs for later processing,
+so the job will not be lost.
 
 When the job exits, pclient tells pcontrol that the job is completed,
 and specifies the exit status. At this point, pcontrol will move the
-host from busy to done state. It will stay in this state until
-pcontrol can determine the ending conditions and reset the remote
-client. pcontrol requests the standard error and standard output from
-the job from pclient. pcontrol stores this data with its information
-about the completed job, and send a reset command to the remote
-client. Once these cleanup tasks are successfully completed, pcontrol
-will move the host to the idle state, ready for further jobs.
+host from \code{busy} to \code{done} state. It will stay in this state
+until pcontrol can determine the ending conditions and reset the
+remote client. pcontrol requests the standard error and standard
+output from the job from pclient. pcontrol stores this data with its
+information about the completed job, and send a reset command to the
+remote client. Once these cleanup tasks are successfully completed,
+pcontrol will move the host to the \code{idle} state, ready for
+further jobs.
 
 Each physical computer may have multiple processors. pcontrol treats
@@ -574,10 +545,9 @@
 \subsection{Jobs}
 
-The pcontrol accepts new jobs with the command job ..., in which the
-ellipsis represents the command and arguments of a valid UNIX
-command. The commands are run under sh, and are executed in the user's
-home directory. (If it is desired, we can easily add a command to tell
-pclient to perform cd). Users should be wary of the conditions under
-which the remote jobs are run. If the nodes in question all
+The pcontrol accepts new jobs with the command \code{job ...}, in
+which the ellipsis represents the command and arguments of a valid
+UNIX command. The commands are run under \code{sh}, and are executed
+in the user's home directory. Users should be wary of the conditions
+under which the remote jobs are run. If the nodes in question all
 cross-mount the same home directories, multiple jobs which interact
 with the same named file may produce unexpected results. The
@@ -590,5 +560,5 @@
 not realize that specific jobs do not behave the same on all machines,
 or if a necessary resource (eg, some input data file) is only
-available or accessible from some of the hosts. It is the
+available or accessible from some of the hosts. It is also the
 responsibility of the task to wait for network lags (ie, NFS delays).
 
@@ -597,5 +567,6 @@
 pcontrol, the command echoes back the Job ID. This ID may be used by
 other pcontrol commands to obtain information about or interact with
-the job.
+the job.  For example, PanTasks uses the Job ID to examine specific
+jobs.
 
 A job may specify a specific host for the task execution. The host
@@ -634,6 +605,16 @@
 (JobID).
 
-Jobs are moved between the following states by pcontrol:
-
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.05.ps}
+\caption{\label{queues} pcontrol job states.  Transitions labeled Ux
+  are issued by the pcontrol user (including PanTasks).  Transitions
+  labeled Px are initiated by pcontrol.  Transitions labeled Tx are
+  the result of the job completion}
+\end{center}
+\end{figure}
+
+Jobs are moved between the following states by pcontrol (see
+Figure~\ref{JobStates}):
 \begin{itemize}
 \item pending: the job has not yet been executed.
@@ -647,5 +628,5 @@
 
 It is possible to check the status of a single host or job with the
-user command check.
+user command \code{check}.
 
 pcontrol continuously examines the stack of jobs, adjusting their
@@ -658,5 +639,5 @@
 number of microseconds for the timeout.
 
-the pcontrol system status may be examined with the command
+The pcontrol system status may be examined with the command
 status. This provides a dump of the job stacks and the host stacks.
 
@@ -668,24 +649,37 @@
 have exited or crashed.
 
-A specific job may be killed with the command kill (JobID). This
-command is only valid for a job in the busy state. Any job in the
-pending, exit, or crash state may be deleted with the delete (JobID)
-command. This is necessary to free the memory associated with the job
-and its output streams.
-
-The command verbose (mode) turns the verbosity of the pcontrol
+A specific job may be killed with the command \code{kill
+(JobID)}. This command is only valid for a job in the busy state. Any
+job in the pending, exit, or crash state may be deleted with the
+\code{delete (JobID)} command. This is necessary to free the memory
+associated with the job and its output streams.  PanTasks
+automatically performs these job harvesting functions.
+
+The command \code{verbose (mode)} turns the verbosity of the pcontrol
 operations on or off.
 
-The pcontrol and the IPP Image Server have related needs for
-information from the combined storage-and-processing nodes regarding
-which nodes are available. It is not yet clear if this information is
-best stored in a single location (either pcontrol or IPP Image
-Server), which provides the information to other systems on demand, or
-if both systems should maintain the information. Also, it may be
-necessary to distinguish nodes which are available for processing from
-those that are available to serve data as part of the IPP Image
-Server.
-
-\subsection{Command Summary}
+The pcontrol and Nebulous have related needs for information from the
+combined storage-and-processing nodes regarding which nodes are
+available. Currently, the two systems independently examine the
+hardware to judge the availability.  It is not yet clear if this
+information is best stored in a single location (either pcontrol or
+Nebulous), which provides the information to other systems on demand.
+The current implementation allows the IPP system as a whole to
+distinguih nodes which are available for processing from those that
+are available to serve data as part of Nebulous.
+
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.06.ps}
+\caption{\label{PControlLoop} PControl Job Monitor Loop}
+\end{center}
+\end{figure}
+
+Figure~\ref{PControlLoop} illustrated the pcontrol job monitor loop.
+This is very similar to the loop in PanTasks.  A background process
+launched by readline during idle periods cycles through the list of
+hosts (children) and migrates jobs to and from them as needed.
+
+\subsection{pcontrol Command Summary}
 
 \begin{verbatim}
@@ -703,25 +697,4 @@
 \end{verbatim}
 
-\begin{figure}
-\begin{center}
-\includegraphics[scale=0.85]{pics/pantasks.04.ps}
-\caption{\label{queues} PanTasks queues and MDDB tables}
-\end{center}
-\end{figure}
-
-\begin{figure}
-\begin{center}
-\includegraphics[scale=0.85]{pics/pantasks.05.ps}
-\caption{\label{queues} PanTasks queues and MDDB tables}
-\end{center}
-\end{figure}
-
-\begin{figure}
-\begin{center}
-\includegraphics[scale=0.85]{pics/pantasks.06.ps}
-\caption{\label{queues} PanTasks queues and MDDB tables}
-\end{center}
-\end{figure}
-
 \newpage
 \section{pclient}
@@ -739,5 +712,6 @@
 recommended practice is to set up ssh to allow the connection to the
 remote host without additional authentication using the appropriate
-authorized keys (see this article on ssh issues).
+authorized keys.  The security is maintained by the security of ssh
+logins to the computers used by PanTask and pcontrol.  
 
 It is convenient to keep a continuous connection to the remote
@@ -768,5 +742,5 @@
 \begin{figure}
 \begin{center}
-\includegraphics[scale=0.85]{pics/pantasks.07.ps}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.07.ps}
 \caption{\label{queues} PanTasks queues and MDDB tables}
 \end{center}