Index: /trunk/doc/Makefile
===================================================================
--- /trunk/doc/Makefile	(revision 6054)
+++ /trunk/doc/Makefile	(revision 6055)
@@ -1,5 +1,5 @@
 # Makefile for all IPP main engineering documents
 
-DIR = pslib modules design hardware ipptools pantasks psphot misc
+DIR = pslib modules design hardware dvo ipptools pantasks psphot misc
 
 all:
Index: /trunk/doc/design/ippSSDD.tex
===================================================================
--- /trunk/doc/design/ippSSDD.tex	(revision 6054)
+++ /trunk/doc/design/ippSSDD.tex	(revision 6055)
@@ -1,3 +1,3 @@
-%%% $Id: ippSSDD.tex,v 1.5 2006-01-19 06:49:50 eugene Exp $
+%%% $Id: ippSSDD.tex,v 1.6 2006-01-19 10:58:19 eugene Exp $
 \documentclass[panstarrs]{panstarrs}
 
@@ -779,4 +779,8 @@
 \subsection{AP Database}
 
+\tbd{this section needs to be updated with current implemetation; the
+  DVO SDD contains much of this information, but needs to be fleshed
+  out in places.}
+
 \subsubsection{Corresponding Requirements}
 
@@ -1199,392 +1203,676 @@
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\subsection{Controller}
-\label{sec:Controller}
-
-\subsubsection{Corresponding Requirements}
-
-The Controller must meet the requirements specified in Section 3.4.4
-of the Pan-STARRS PS-1 IPP SRS (PSDC-430-005).  The design must meet
-requirements 3.4.4.1 - 3.4.4.7.  In particular, the Controller / Node
-Agent architecture is chosen to control the I/O flow between the
-Controller and the individual processes so that blocking on the I/O
-from many remote processes does not saturate the Controller
-processing.
-
-\subsubsection{Overview}
+\subsection{PanTasks : the IPP Scheduler}
+
+PanTasks is the IPP tool which manages the sequencing of data analysis
+steps and, with the related tool `PControl', distributes the data
+processing across a cluster of computers. \tbd{describe the 'opihi'
+shell-scripting system}
+
+The purpose of PanTasks is to manage the automatic construction and
+execution of inter-related (often repetative) operations.  PanTasks
+uses a set of rules to define UNIX commands, and their corresponding
+command-line arguments, to be performed on some regular, repeated
+basis.  The utility of PanTasks is that it can easily define and
+manage an analysis system which is completely state-based, as opposed
+to an event-driven system.
+
+Consider, for example, a telescope which obtains a collection of
+images over the course of a night. Every minute or two, it takes an
+image and writes the image to some disk. An event-driven analysis
+system would involve having the telescope initiate a process at the
+end of the exposure. This process would perform an analysis, write
+some output, then send trigger another process. This type of operation
+works very well for a simple set up with reliable hardware. Such a
+system becomes more difficult to maintain when hardware failures occur
+or when multiple systems need to interact with each other. When
+failures occur, the triggering information (the events) is easily
+lost, thus some mechanisms are needed to detect these failures and
+either re-send the trigger or send an alternative failure-mode
+trigger. Or, if two systems need to interact, one or the other system
+must block for results from the first. Stopping and restarting such an
+analysis system is very delicate since the appropriate triggers must
+be set up some how, eg by noticing which images have not succeeded and
+restarting them at the appropriate stage. All of these types of
+methods of handling complexity and failures are essentially
+state-based rules. PanTasks allows the easy definition of a totally
+state-based analysis system.
+
+In a state-based system, some mechanism examines the state of the
+system and decides which actions to perform based on the current
+state. In the illustration above, the mechanism could examine the
+images available (either by examining the disk or by examining the
+state of a data table) and decide to perform an operation based on
+what images are available. This makes it very easy to handle
+complexity and errors. If an analysis fails, the state either is not
+successfully updated or the error state is recorded, both situations
+being easy to detect and easy to handle. Restarting the system simply
+involves starting the state-monitoring mechanism. Combining results
+from multiple input sources simply involves watching for the multiple
+inputs to be available. PanTasks provides a mechanism to define state
+monitors, and to define the actions which are performed when those
+states occur. PanTasks action consist of initiating UNIX commands, where
+the arguments of those commands may depend on the results of the state
+tests.  
+
+\subsubsection{Tasks vs Jobs}
+
+The two basic units of PanTasks operation are the 'task' and the
+'job'.  A 'job' is simply a command the user would execute on a
+command line; it consists of a command along with optional command
+line arguments.  A 'task' is a generic description of a type of job in
+which the details of any specific piece of data are omitted.  The task
+defines the UNIX command which corresponds to the job, and it provides
+rules for determining the identity of data for the job.  The task also
+defines tests which are used to decide if the job may be executed.
+Finally, it defines a polling frequency in which PanTasks should
+attempt to construct a new job for the task.
+
+For example, we may want to regularly copy files from one location to
+another.  Perhaps the name of the next available file is available in
+a database table, and can be retrieved with the command 'nextFile'.
+Perhaps we wish to check for new files every 60 seconds.  Thus, the
+task is to copy files, while a specific job of this task might be of
+the form \code{cp newfile newpath}.  With PanTasks, we would define a
+copy task somewhat like the following:
+
+\begin{verbatim}
+  task copyfile
+    periods -exec 60.0
+
+    task.exec
+      $file = `nextFile`
+      if ($file == "none")
+        break
+      end
+      command cp $file $newpath
+    end
+
+    task.exit 0
+      queuepush copied $file
+    end
+
+    task.exit 1
+      queuepush failure $file
+    end
+  end
+\end{verbatim}
+
+In this simple example, the task is attempted every 60 seconds.  If
+there is no new file (output of 'nextFile' is 'none'), then no job
+results from this task.  If there is a new file, the copy command is
+performed.  This example is deceptively simple because one could
+easily imagine writing a stand-along program which performs the same
+thing.  The important advantages of using PanTasks for this type of
+operation are: 
+\begin{itemize}
+\item the output from one job can be used to spawn a number of other tasks.
+\item the success or failure state of each job can be used to spawn
+  other tasks.
+\item the details of the job and data test are kept separated from the
+  rules which connect tasks together.  
+\item the relationships between tasks are kept together in a single
+  location.
+\end{itemize}
+
+In addition to these organizational advantages, PanTasks also provides
+a direct connection to the tool for monitoring parallel jobs,
+pcontrol.  Thus the jobs spawned by PanTasks can be defined to run in
+the background locally or on any of the computers in the parallel
+processing cluster.
+
+\subsubsection{Parallel vs Local Job Processing}
+
+Jobs which are generated by PanTasks tasks may either be run locally
+(forked in the background on the same machine as PanTasks) or run on
+the IPP parallel process controller, pcontrol. The default is for the
+job to be run locally. If a job should be run on the parallel
+controller, this can be specified by including the command host
+(hostname) in the definition of a task. If the value of (hostname) is
+'anyhost', then pcontrol may select any of its host computers to run
+the job according to its own rules. If the value of (hostname) is one
+of the computers managed by pcontrol, then that machine will be
+selected for the job, if it is available. This amounts to a preference
+to use that machine, but pcontrol is allowed to substitute a different
+machine if it chooses. If the host command is given the option
+-required, then pcontrol is forced to use the named host, even if the
+machine is down, unknown, or otherwise unavailable. If the machine is
+not available, pcontrol will simply hold onto the job until the
+machine is available or the job is deleted. Note that PanTasks may
+delete jobs from pcontrol if they remain pending for too long.
+
+It is possible to interact directly with the parallel processor to
+examine the current status, halt the parallel processor, etc. Commands
+to the parallel processor are defined under the controller
+command. The following controller commands are available:
+
+\begin{itemize}
+
+\item controller host (command) (hostname): Manage the parallel
+  controller collection of hosts. This command can be used to add a
+  new host, the delete one of the existing hosts, to turn a host on or
+  off, and to check the status of a host
+
+  \begin{itemize}
+  \item controller host add (hostname): add a new host.
+
+  \item controller host delete (hostname): delete a host.
+
+  \item controller host on (hostname): tell pcontrol that the host is on.
+
+  \item controller host off (hostname): tell pcontrol that the host is off.
+
+  \item controller host retry (hostname): tell pcontrol to retry the host connection.
+
+  \item controller host check (hostname): check the current status of a host. 
+  \end{itemize}
+
+  \item controller exit: stop controller execution.
+
+  \item controller status: report controller current status.
+
+  \item controller check: check job or host status.
+
+  \item controller output: print accumulated messages from the controller. 
+\end{itemize}
+
+It is also possible to specify a host for a task which has not been
+identified to the controller. If such a host is required, the
+controller will simply keep the associated jobs in the pending state
+until such a machine exists. See the pcontrol section below for
+further discussion of the controller manipuation of jobs and hosts.
+
+\subsubsection{Task Restrictions}
+
+Tasks may have restrictions on when they create jobs and how
+frequently they create jobs. The task command trange is used to
+specify a valid or invalid time range for a task. A valid time range
+limits the task evaluation to that time period. An invalid time range
+excludes task evaluation from the time period. Any number of time
+range restrictions may be defined, and the union of all restrictions
+will define if a job may be created. By default, the time range is an
+inclusive time range: the task is evaluated only if the current time
+falls within the specified time range. Alternatively, if the -exclude
+flag is given, the time range is exclusive, in which case the task is
+not evaluated if the current time falls within this range.
+
+The time range may be given as a range of absolute dates as follows:
+
+\begin{verbatim}
+ trange YYYY/MM/DD,HH:MM:SS YYYY/MM/DD,HH:MM:SS 
+\end{verbatim}
+
+where the two dates specify the start and end of the time range. In
+either of these date representations, the least-significant elements
+of the date and time may be dropped, defaulting to 00 (in the case of
+hours, minutes, and seconds) or 01 (in the case of day and
+months). Rather than specifying an end date, it is also valid to
+specify a time interval from the starting date. The time interval is
+specified as a number followed by a unit indicated by a single letter:
+d (days), h (hours), m (minutes), s (seconds).
+
+The time range may also be specified as a repeated period of time,
+either as a time of day or a day and time of week. In the first case,
+the time range is specified as follows:
+
+ 
+\begin{verbatim}
+ trange HH:MM:SS HH:MM:SS
+ \end{verbatim}
+
+
+where again the least-significant elements may be dropped and default
+to 00. This type of restriction defines a time range which is valid
+every day. The alternative is to specify a time range within the week,
+in the following form:
+
+\begin{verbatim}
+ trange DAY@HH:MM:SS DAY@HH:MM:SS
+\end{verbatim}
+
+where the value of DAY may take on any of the three letter day-of-week
+names (Sun, Mon, Tue, etc). This restriction specifies a start and end
+time within a week which is evaluated for each week.
+
+Below are several examples of valid time range restrictions
+
+\begin{verbatim}
+ trange 2005/01/01 2005/12/31   (only run during 2005!)
+ trange 18:00 00:00             (only run from 6pm until midnight)
+ trange 00:00 06:00             (only run from midnight until 6am)
+ trange Mon@08:00 Fri@17:00     (only run between Mon morning and Fri afternoon)
+ trange -exclude 12:00 13:00    (skip 1 hour from noon)
+\end{verbatim}
+
+Note that the current definition of trange does not include time zone
+information. This means that all times are relative to UT. This should
+be addressed by adding a timezone environment variable to PanTasks and
+by allowing the trange to define a timezone offset.
+
+It is also possible to restrict the total number of jobs which are
+spawned for a given task. This is done with the nmax command, which is
+given as part of the task definition. Once a task has constructed nmax
+jobs, it stops task evaluation. It is possible to redefine the value
+of nmax at any time by redefining the task. Any time the task is
+redefined, the new values for any task concept will override the
+existing values for the task concept.
+
+\subsubsection{Inter-Task and Inter-Job Communications}
+
+There are several ways in which the results of jobs may be used to
+influence other jobs. These include:
+
+\begin{itemize}
+\item external communications
+\item job exit status
+\item job stdout/stderr parsing 
+\end{itemize}
+
+It is always possible for the interprocess communication to be
+performed externally: all jobs may simply write results to an external
+data source which is queried as part of the task evaluation. PanTasks
+may interact with UNIX programs using Opihi system interaction
+functions. These interaction methods include: the backticks for
+setting Opihi variables:
+
+\begin{verbatim}
+ $variable = `UNIX Command`
+\end{verbatim}
+
+The exec command (which executes a UNIX command) and the backticks
+both receive the UNIX command exit status, setting the variable
+\code{$STATUS}. It is also possible to set a variable list to the
+output of a UNIX command:
+
+\begin{verbatim}
+ list var -x "UNIX Command"
+\end{verbatim}
+
+In this last case, the values \code{$var:0 - $var:N-1} are set to the
+value of the stdout lines from the UNIX command, and the value
+\code{$var:n} is set to the number of output lines.
+% $
+
+Fine-grained control over the job exit status is available with the
+task.exit macro command. This allows a task to define an exit macro
+which is performed for different exit status conditions. The argument
+to the task.exit command is the exit status value which triggers the
+macro. This may consist of any valid numeric exit status value
+(0-255). It may also have the value crash, in which case the macro is
+executed if the program exited as a result of a signal (ie,
+segmentation fault, etc). Finally, if may have the value default, in
+which case, the macro is run if no other macro describes the exit
+status.
+
+Jobs may transmit their results back to PanTasks for further
+evaluation through the standard output and standard error
+streams. Whenever a job exits, the complete stdout and stderr streams
+from the job are pushed onto the PanTasks queues \code{stdout} and
+\code{stderr}. The job exit macros may then parse these queues, moving
+the results into other PanTasks / Opihi data containers (queues,
+variables, vectors, whatever is appropriate). Note that currently, the
+output data is simply pushed onto these output queues. It is currently
+the responsibility of the PanTasks programmer to use or dispose of the
+data in these queues. This may change in the future: the queues may be
+flushed for each job completion.
+
+\subsubsection{PanTasks Monitoring Loop}
 
 \begin{figure}
 \begin{center}
-\resizebox{4.5in}{!}{\includegraphics{pics/Controller}}
-\caption{Schematic illustration of the Controller components}
-\label{fig:Controller}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.01.ps}
+\caption{\label{MonitorLoop} PanTasks Monitor Loop}
 \end{center}
 \end{figure}
 
+Figure~\ref{MonitorLoop} illustrates the operation of PanTasks.
+Currently, PanTasks is run as a stand-alone foreground process, not a
+server.  In this current operating mode, PanTasks accepts commands
+from the user (left side) via a GNU readline interface.  Readline
+provides a mechanism to schedule an background process which is
+executed on a regular basis, or in the interval after each keystroke.
+This background processing is represented as the loop on the right.
+In this loop, PanTasks checks on the status of pcontrol and on any
+locally spawned background jobs.  It also checks the list of tasks for
+tasks which are ready to be executed.  The tasks and the background
+jobs are kept in rotating queues.  Every time the loop is processed,
+PanTasks runs through a number of the background jobs and tasks,
+attempting to evaluate as many as possible, but stopping after a
+limited number of milliseconds.  This test stage cycles through the
+tasks and/or jobs in their queue, but stops if it examines all of the
+entries in the queue.  If this testing process runs out of time before
+the entire queue is searched, it leaves the sequence intact for the
+next pass.  The timeout is set to keep the keyboard small enough to
+keep the human interaction from being troublesome.
+
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.02.ps}
+\caption{\label{TaskLoop} PanTasks Task Check Loop}
+\end{center}
+\end{figure}
+
+Figure~\ref{TaskLoop} shows the interactions performed for each check
+of the list of tasks.  The task timers are examined to see if the
+specific task is ready to be executed.  If so, then the task exec
+macro is executed.  If this macro exit status is false, the loop goes
+to the next task.  If the task exec macro is true, the job command is
+constructed and the job is submitted to the correct location, either
+the controller (pcontrol) or to the queue of local, background jobs.
+
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.03.ps}
+\caption{\label{JobLoop} PanTasks Job Check Loop}
+\end{center}
+\end{figure}
+
+Figure~\ref{JobLoop} shows the interactions performed for each check
+of a single background job.  The job timer is checked to see if the
+job has timed out.  Next, the job run status is examined to see if the
+job has finished or not.  If the job has finished, the appropriate
+exit macro is executed and the job results are returned to the rest of
+the PanTasks data structures (ie, stderr and stdout queues are filled
+out).  Users should not define exit macros which perform long running
+jobs, or PanTasks will become quite sluggish to keyboard strokes.
+
+PanTasks also checks the current status of pcontrol on each loop.
+In this test, PanTasks requests from pcontrol a list of the jobs which
+have finished.  It then attempts to harvest the finished jobs one by
+one and place the results in the approrpiate locations.  This loop is
+also limited to a fixed amount of time; any pcontrol jobs which remain
+unharvested are left for the next pass by PanTasks.
+
+\subsubsection{Running the scheduler}
+
+Once a set of tasks has been defined, the scheduler can be
+started. The scheduler will run in the background, at regular
+intervals examining the collection of tasks and jobs. In these
+periods, the scheduler attempts to construct new jobs and checks on
+the status of jobs which may have finished, either locally or on the
+controller. To start the scheduler, give the command run. To stop the
+scheduler, given the command stop. The current status of the
+scheduler, controller, and any jobs which have been spawned are listed
+with the status command.
+
+It is also possible to kill or delete individual jobs by hand with the
+commands kill (jobID) or delete (jobID).  Other features
+
+\subsection{pcontrol : the PanTasks parallel controller}
+
 The IPP uses a group of computers to store and process images and to
-manipulate collections of detections.  These computers perform any of
-a large number of analysis stages or other processing tasks without
-significant interprocess communication.  It is necessary to have a
+manipulate collections of detections. These computers perform any of a
+large number of analysis stages or other processing tasks without
+significant interprocess communication. It is necessary to have a
 mechanism which initiates computing tasks on the different computers,
 which monitors the tasks as they are executed, which handles the
 output and the errors from these tasks, and which reacts to the
-failure of any of the computing nodes.  The system responsible for the
-tasks in the IPP is the IPP Controller.
-
-The IPP Controller interacts with the collection of computers under
-its management and with other subsystems in the IPP.  The IPP
-Controller receives a variety of inputs from other subsystems,
-described below, and initiates actions such as adding a new process to
-the queue of pending tasks.  The IPP Controller also provides
-information to other subsystems on demand about its processing history
-and current state.  Each physical computer may have multiple
-processors; since the IPP Controller is managing processing tasks, it
-treats each processor independently.  It is up to the system
-configuration if each computer needs to reserve one of its CPUs to
-manage background tasks or if the IPP Controller should attempt to
-send one task per CPU and let the operating system handle the I/O
-load.  The relationship between the different components of the
-Controller is illustrated in Figure~\ref{fig:Controller} and discussed
-below.
-
-\subsubsection{Nodes}
-
-The Controller maintains a table of available processing computers
-(`Nodes') and tracks the status of these Nodes.  Nodes managed by the
-IPP Controller are allowed to be in one of several states, and the IPP
-Controller must interact with it in an appropriate way for each of
-those states.  A Node may be {\tt alive}, {\tt dead} or {\tt off}.
-If the Node is {\tt alive}, it responds to commands from the IPP
-Controller and may be used for tasks subject to other constraints.  If
-it is {\tt dead}, the Node is not responsive and must not be used
-for executing tasks.  The IPP Controller must identify Nodes which
-have died (not responding) and occasionally test them to see if they
-are {\tt alive} again.  Nodes which are {\tt off} are not
-available for tasks and must not be tested.  Nodes may be set to
-the {\tt off} or {\tt dead} states by external subsystems; it is the
-responsibility of the IPP Controller to return a Node to the {\tt
-alive} state if possible.
-
-The IPP Controller must honor requests (normally from the users) to
-change the mode of any computing node on demand between {\tt off} and
-{\tt dead}.  This would normally be done after a Node has been
-rebooted and is released to the IPP Controller for its use.  It must
-also be able to change the list of allowed tasks as requested by
-external commands.
-
-Two example scenarios illustrate the transition between these states,
-and the basic concept of operations for the IPP Controller.  First,
-imagine a computer crashes.  At this point the IPP Controller should
-detect that the Node is no longer responsive and mark it as {\tt
-dead}.  It should occasionally try to re-establish communication with
-the Node, potentially with longer and longer delays between attempts.
-A human could be notified if the Node seems to remain {\tt dead} for a
-very long time.  In another scenario, a person needs to work on a
-Node.  They notify the IPP Controller that the machine is {\tt off},
-perhaps with a prior notification that the machine should be prepared
-to go off.  When work on the machine is complete, it should be placed
-in the {\tt dead} state.  Only when the person is done working and
-testing the machine, and tells the IPP Controller that the machine is
-now {\tt dead} can the IPP Controller attempt to re-start
-communications and re-new processing operations on that Node.
-
-\subsubsection{Node Agents}
-
-When the Controller starts, it attempts to launch a Node Agent on each
-of the available processing Nodes.  Nodes which are not responsive are
-marked as {\tt dead} so they may be re-tried.  A Node Agent runs on
-each of the individual nodes to execute the tasks as directed by the
-Controller.  The Node Agents communicate with the Controller via a
-socket connection.
-
-A Node Agent (which is only running on a Node in the {\tt alive}
-state) may be in one of four modes: {\tt idle}, {\tt busy}, {\tt
-done}, {\tt crash}.  A Node Agent which is {\tt busy} currently has a
-task assigned to it which is executing.  The IPP Controller may only
-assign one task to a Node at a time.  A Node Agent which is in the
-{\tt idle} state may have a task assigned to it.  When the Node Agent
-detects that a tasks has finished, it changes to either the {\tt done}
-or {\tt crash} states depending on the outcome of the process
-execution.  The IPP Controller must also respect a list of task
-restrictions which may require specific tasks to run on specific CPUs
-or exclude specific tasks from specific CPUs.
-
-A task being executed by the Node is run in the UNIX user space as a
-forked process.  The Node Agent must monitor the standard error and
-standard output of the executing task and save them in separate
-buffers.  If the process exits or dies, the Node Agent must detect
-this result and change state appropriately.  The Node Agent must
-respond to various commands from the Controller, as follows:
-
-\paragraph{Report status}
-
-The Node Agent returns its state ({\tt idle}, {\tt busy}, {\tt done},
-{\tt crash}) and the exit status of the current processing task, if
-available.  The reported exit state, if the process has completed
-without crashing, is the UNIX exit state reported by the task: 0--256
-with 0 indicating a successful completion.
-
-\paragraph{Report stdout}
-
-Send and flush the current stdout buffer.  The Node Agent will return
-the complete contents of the stdout buffer via a buffered write and
-flush the buffer when it is finished.  The Node Agent will not accept
-more data on the stdout buffer from the current processing task until
-the send is complete and the buffer is flushed.  The daemon must
-accept all of the buffer output.
-
-\paragraph{Report stderr}
-
-Identical to `report stdout', but for stderr.
-
-\paragraph{Kill task }
-
-The Node Agent should send a kill signal (\code{KILL} or \code{TERM})
-to the current processing task.  When the processing task has exited,
-the Node Agent should set its state to {\tt crash}.
-
-\paragraph{Clear task}
-
-The Node Agent should set its state {\tt idle}.  If a processing stage
-is currently running, it should be killed (\code{KILL} or \code{TERM})
-before the task is cleared.
-
-\paragraph{Start processing stage}
-
-The Node Agent forks a specified command.  The command should be a
-standard UNIX command without command line redirection or
-backgrounding.  The task is run with the same user ID as the Node
-Agent, which is also the same user ID as the Controller.
-
-\subsubsection{Tasks}
-
-The IPP Controller accepts tasks from other IPP subsystems.  The task
-requests include the specific command to be executed and are in the
-form of a UNIX command which could be performed on any of the
-computing nodes.  Any input or output data in the commands must be a
-valid resource regardless of the node on which the task is executed.
-Input and output data resources must be unique where necessary to
-avoid conflicts.  It is the responsibility of the task to wait for
-network lags (ie, NFS delays).  The IPP Controller gives each task a
-unique identifier, which is returned to the requesting entity.  The
-requestor may then use that ID to obtain status information on that
-task or to send control signals to the specific task.
-
-Task requests may specify a desired node for the task execution.  The
-IPP Controller attempts to honor the request if the node is {\tt
-alive}, but will execute it on another node if the requested one is
-{\tt dead} or {\tt off}.  Even if a node is {\tt alive}, the IPP
-Controller will choose another node if the specified task is not
-allowed on the requested node.  In all other cases, the IPP Controller
-waits until the currently executing processes, and processes with
-higher priority, are completed before executing the specified task on
-the requested node.
-
-Task requests may specify an urgency level.  The IPP Controller
-determines the priority of the task on the basis of both the urgency
-and the age of the request.  An executing task must be completed on a
-CPU before any new task is started on that CPU, regardless of
-priority.  The urgency levels range from 0 to 2.  Tasks with an
-urgency of 1 are scheduled whenever they reach the top of the stack.
-Tasks with an urgency of 2 are sent immediately to the top of the
-stack. Tasks assigned a priority of 0 are maintained in the queue and
-never executed.
-
-It may be useful for the Controller to distinguish between tasks
-dominated by I/O and tasks dominated by data processing.  It is
-possible that one of each of these types of tasks may be sent to the
-same node without significantly impacting the system performance.
-Alternatively, it may be necessary to limit a single machine with 2
-CPUs to only one of each of these types of tasks (i.e., one processor
-will be working on I/O while the other is working on processing).
-Such details will be studied by the IfA IPP Team.
-
-The IPP Controller monitors the output streams from the executing
-tasks and the exit status of the tasks.  Each task is associated with
-a log file, to which all output is written.  The status, including the
-exit status, of each task is maintained by the IPP Controller so that
-other subsystems may determine if specific tasks have started or
-completed.
-
-\subsubsection{Controller Interfaces}
-
-The IPP Controller must accept commands from other IPP subsystems.
-These commands include those which govern the processing of specified
-tasks, those which govern the behavior of specific computing nodes,
-and those which request information from the IPP Controller.  The IPP
-Controller must be able to halt the execution of a specified task,
-delete an unexecuted task from the task list, change the priority of
-tasks, and change the requested nodes for tasks.  The IPP Controller
-must also be able to stop the current execution of a task and push it
-to the end of the queue and also change its priority.
-
-The IPP Controller must respond to informational requests regarding the
-collection of machines and their states as well as the collection of
-tasks and their states.  The IPP Controller must monitor the execution
-times of the different tasks and provide summary statistics.  Finally,
-the IPP Controller must respond to three top-level commands: {\tt finish},
-{\tt stop} and {\tt abort}.  When {\tt finish} is requested, no more
-new tasks are accepted on the stack of task, and when all tasks in the
-stack have completed, the IPP Controller must exit.  When {\tt stop} is
-requested, the currently executing tasks must be completed at which
-point the IPP Controller must exit, but tasks remaining in the stack which
-have not been started are flushed.  When {\tt abort} is issued, the
-IPP Controller immediately kills all executing tasks and exits.
-
-The IPP Controller and the IPP Image Server have related needs for
-information from the combined storage-and-processing nodes regarding
-which nodes are available.  It is not yet clear if this information is
-best stored in a single location (either IPP Controller or IPP Image
-Server), which provides the information to other systems on demand, or
-if both systems should maintain the information.  Also, it may be
-necessary to distinguish nodes which are available for processing from
-those that are available to serve data as part of the IPP Image
-Server.
-
-The Controller maintains three tables of processing jobs: pending
-stages, active stages, and completed stages.  The pending stages are
-those which have not yet been performed.  The active stages are those
-currently being performed on one of the remote nodes.  The completed
-stages are those which have finished, either successfully or with an
-error state.  The Controller daemon monitors the collection of remote
-clients and sends them new pending stages when they become free.
-
-The IPP Controller provides a mechanism for users (either other
-programs or humans) to interact with it.  The user interface provides
-commands to check the current processing job queues, the tables of
-successful and failed jobs, to stop or delete jobs, etc.
-
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-
-\subsection{Scheduler}
-\label{sec:scheduler}
-
-\subsubsection{Corresponding Requirements}
-
-The Scheduler must meet the requirements specified in Section 3.4.5 of
-the Pan-STARRS PS-1 IPP SRS (PSDC-430-005).  The design must meet
-requirements 3.4.5.1 - 3.4.5.7.  In particular, the Task / Test
-division is chosen to prevent the Scheduler from blocking while an
-analysis process is performed.  Scheduling requirements will be met by
-defining appropriate Test periods for the different Tasks.
-
-\subsubsection{Overview}
-
-The IPP is responsible for a variety of analysis jobs: processing of
-the science images through several stages; routine assessment of the
-detrend (instrumental calibration) images used in processing the
-science images; construction of replacement detrend images when
-needed; generation of astrometric and photometric reference catalogs
-based on the collected dataset; and the performance of test analysis
-programs.  At any point, decisions need to be made about which of
-these tasks should be performed, based on an analysis of the contents
-of the metadata database, the requirements of the people monitoring
-the IPP, and the near-term observing plans.  The IPP Scheduler is the
-mechanism that assesses these various inputs to guide the decisions
-and initiate the actions.
-
-The IPP Scheduler acts as an interface between several components of
-the IPP and also between the IPP and external agents such as OTIS and
-the users who must monitor the behavior of the IPP.  The IPP Scheduler
-may be viewed as the central brain of the IPP.
-Figure~\ref{fig:Scheduler} illustrates the design of the IPP
-Scheduler.
-
-\subsubsection{Scheduler Tasks and Tests}
-
-The IPP Scheduler performs two types of actions.  'Tasks' are
-long-running programs which are executed by the Controller.  These are
-not only background tasks, but are distributed computing tasks.
-Examples of these include the science analysis tasks (eg, Phase 1, 2,
-3, 4), the Calibration construction tasks, and data copy tasks (such
-as copying images and metadata from the summit system).  'Tests' are
-short-running programs which are used to decide which tasks should be
-run.  Tests should be designed to return immediately ($< 100 ms$) and
-are not run in the background; the Scheduler will block until the test
-is complete.  The IPP Scheduler daemon, which runs continuously,
-performs tests (eg, queries of the IPP Metadata Database, queries of
-OTIS, checks of the IPP hardware status, etc).  Based on these tests,
-the daemon defines appropriate tasks and sends them to the Controller.
-When tasks are completed, their results may be used by the Scheduler
-to update the external systems (update the Metadata Database), or the
-tasks themselves may send their results directly to the Metadata
-Database or other subsystems.  Based on the successful completion (or
-not!) of the tasks, and the new state of entries in the Metadata
-Database, the Scheduler can define new tasks.
+failure of any of the computing nodes. The system responsible for the
+tasks in the IPP is pcontrol.
+
+\subsubsection{Host States}
 
 \begin{figure}
 \begin{center}
-\resizebox{6in}{!}{\includegraphics{pics/Scheduler}}
-\caption{ \label{fig:Scheduler} IPP Scheduler}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.04.ps}
+\caption{\label{HostStates} PanTasks Host States}
 \end{center}
 \end{figure}
 
-The IPP Scheduler sends tasks to the IPP Controller for execution.
-While the IPP Scheduler chooses the tasks to be performed, it is the
-IPP Controller's responsibility to manage the specific tasks executing
-on a given processing node.  This division of responsibilities allows
-the different functionalities of the IPP Scheduler and the IPP
-Controller to be isolated and encapsulated.  With this separation, the
-IPP Controller does not information about the details of the tasks it
-executes, while the IPP Scheduler does not need to monitor the
-computer hardware.
-
-Communication between the IPP Scheduler and the IPP Controller is
-bi-directional; the IPP Scheduler sends tasks to the IPP Controller,
-while the IPP Controller informs the IPP Scheduler of the outcome of
-those tasks.  For the PS-1 IPP, the IPP Scheduler and the IPP
-Controller are distinct, interacting software components.  The
-interface mechanisms are described in Section~\ref{sec:interfaces}.
-
-\subsubsection{Task Rules}
-
-The IPP Scheduler takes as input a collection of rules which define
-the dependency of tasks on certain tests.  The IPP Scheduler must
-choose between several types of analysis tasks based on those rules
-and on results of the tests.  The timescale on which different tasks
-(and their related tests) are executed may vary from 10s of seconds to
-hours, days, or even as long as a week.  The list of tasks which the
-IPP Scheduler must decide between, and the relevant timescale, follow:
+pcontrol maintains a table of available processing computers (hosts)
+and tracks their status. Hosts managed by pcontrol are allowed to be
+in one of several states: \code{off}, \code{down}, \code{idle},
+\code{busy}, and \code{done} (see Figure~\ref{HostStates}). There are
+also two virtual states: \code{new} and \code{delete}.  These states
+have the following meanings:
+
+If the host is \code{off}, it is known to pcontrol, but pcontrol does
+not have an active connection to the machine. Hosts which are
+\code{off} are not available for jobs, and pcontrol does not attempt to
+initiate a connection to them.  A pcontrol user may force a host to
+transition to the \code{off} state with the command host
+\code{off~(hostname)}. (Note that this command will set only one of
+the connections to the named host to \code{off}. If multiple
+connections to a machine have been defined, multiple \code{off}
+commands must be sent).
+
+When pcontrol is told to consider a machine on, the machine is moved
+from the \code{off} state to the \code{down} state. Pcontrol attempts
+to initiate a connection to the host. Connections are made by running
+a remote client on the host, using the specified connection
+method. The connection method may be ssh, rsh, or an equivalent remote
+shell connection. The choice is specified by the \code{COMMAND} Opihi
+variable. The remote connection starts a dedicated remote client which
+must accept the pcontrol client commands and respond
+appropriately. The provided remote client is called {\em pclient},
+though in principal other equivalent programs could be used by setting
+the Opihi variable \code{SHELL}.  This feature more generally allows a
+user to specify a path to the remote client, if it is not in the
+user's path.
+
+If the remote connection is successful, the connected host is moved by
+pcontrol from the \code{down} state to the \code{idle} state. If the
+connection is unsuccessful, pcontrol will try again after a certain
+period of time. If the connection continues to be unsuccessful, the
+retry period is doubled for each successiver connection attempt. If
+the user wants to force pcontrol to retry the connection to a machine
+(if, for example, the timeout is now very long, but the user knows the
+machine's ethernet cable has been re-inserted...), this can be
+achieved with the command host \code{retry (hostname)}. A host which
+is \code{down} is in the limbo state between \code{off} and
+\code{idle}.
+
+Once pcontrol has made a successful connection to the host, the host
+is in the \code{idle} state. At this point, it is ready to accept jobs
+from pcontrol for execution. Pcontrol periodically queries the hosts
+to check that they are still alive. If a host is discovered to be
+unresponsive, and particularly if the remote pipe connection has
+closed, then the machine is moved back to the \code{down} state.
+
+Hosts which are \code{idle} may accept a job from pcontrol. A job
+simply consists of a bare UNIX command, without redirection of
+standard input or standard output. The host will initiate the job, and
+pcontrol will place the host into the \code{busy} state. The remote
+client, pclient, runs the job in the background and will continue to
+accept input from pcontrol. pcontrol will continue to check the status
+of the host, and now also the status of the specific job. As before,
+if the connection breaks, pcontrol will migrate the host to the
+\code{down} state. Any job already initiated on a host which goes down
+will be returned to the pool of unexecuted jobs for later processing,
+so the job will not be lost.
+
+When the job exits, pclient tells pcontrol that the job is completed,
+and specifies the exit status. At this point, pcontrol will move the
+host from \code{busy} to \code{done} state. It will stay in this state
+until pcontrol can determine the ending conditions and reset the
+remote client. pcontrol requests the standard error and standard
+output from the job from pclient. pcontrol stores this data with its
+information about the completed job, and send a reset command to the
+remote client. Once these cleanup tasks are successfully completed,
+pcontrol will move the host to the \code{idle} state, ready for
+further jobs.
+
+Each physical computer may have multiple processors. pcontrol treats
+each processor independently. It is up to the system configuration if
+each computer needs to reserve one of its CPUs to manage background
+tasks or if pcontrol should attempt to send one task per CPU and let
+the operating system handle the I/O load. some of this behavior will
+probably be eventually more intelligent. For example, the commands
+which turn a host on or off should be able to do the same operation to
+all host connections for the same machine name.
+
+A machine may be completely removed from pcontrol's host tables with
+the command host delete (hostname).
+
+\subsubsection{Jobs}
+
+The pcontrol accepts new jobs with the command \code{job ...}, in
+which the ellipsis represents the command and arguments of a valid
+UNIX command. The commands are run under \code{sh}, and are executed
+in the user's home directory. Users should be wary of the conditions
+under which the remote jobs are run. If the nodes in question all
+cross-mount the same home directories, multiple jobs which interact
+with the same named file may produce unexpected results. The
+controller cannot enforce good behavior on the part of the remote
+jobs; it is the responsibility of the user to ensure that conflicts do
+not arise by, eg, always using unique output file names.
+
+Other issues may arise from the fact that pcontrol may be choosing any
+of the hosts to run the job. Typical failures arise if the user does
+not realize that specific jobs do not behave the same on all machines,
+or if a necessary resource (eg, some input data file) is only
+available or accessible from some of the hosts. It is also the
+responsibility of the task to wait for network lags (ie, NFS delays).
+
+pcontrol gives each task a unique internal identifier (Job ID)
+equivalent to the process ID used in UNIX. When a job is submitted to
+pcontrol, the command echoes back the Job ID. This ID may be used by
+other pcontrol commands to obtain information about or interact with
+the job.  For example, PanTasks uses the Job ID to examine specific
+jobs.
+
+A job may specify a specific host for the task execution. The host
+specified for a job may be required, or desired. In the first case,
+pcontrol, will only run the job on the specified host, waiting until
+it is available before attempting the job. In the second case,
+pcontrol will attempt to send the job to the specified host, but if
+the host is unavailable (how long? what conditions?), pcontrol will
+allow the job to be sent to an alternative host. pcontrol attempts to
+honor the requests for required and desired hosts, giving priority
+first to required-host jobs, then to the desired-host jobs, and
+finally to all other jobs. To specify a host for a job, the following
+commands are used:
+
+\begin{verbatim}
+ job -host (command and arguments...)
+ job +host (command and arguments...)
+\end{verbatim}
+
+
+The first case specifies a desired host, while the second specifies a
+required host. It is also possible to specify the special host name
+anyhost, which is equivalent to not specifying a host at all.
+
+Job priority / urgency levels are not implemented at this time.
+
+I/O vs CPU tasks are not currently distinguished by pcontrol
+
+pcontrol stores the stdout and stderr for each completed job. To
+retrieve these data from these streams, the user issues the commands
+stdout (JobID) and stderr (JobID). The result is a single line
+specifying the number of bytes to expect, followed by a dump of the
+buffers, followed by the prompt. It is the user's responsibility to
+relieve pcontrol of this data load by deleting jobs once they are no
+longer needed. Job deletion is performed with the command delete
+(JobID).
+
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.05.ps}
+\caption{\label{queues} pcontrol job states.  Transitions labeled Ux
+  are issued by the pcontrol user (including PanTasks).  Transitions
+  labeled Px are initiated by pcontrol.  Transitions labeled Tx are
+  the result of the job completion}
+\end{center}
+\end{figure}
+
+Jobs are moved between the following states by pcontrol (see
+Figure~\ref{JobStates}):
 \begin{itemize}
-\item moving data from the Summit pixel server ($\sim 30$ second timescales)
-\item running the science analysis stages ($\sim 30$ second timescales)
-\item testing the validity of the current detrend images ($\sim$
-  nightly)
-\item constructing new detrend images ($\sim$ weekly)
+\item pending: the job has not yet been executed.
+\item busy: the job is currently being executed.
+\item done: the job has completed, but the stdout/stderr has not been processed by pcontrol.
+\item exit: the job has completed with a valid exit status
+\item crash: the job has completed with a crash status (exit on signal). 
 \end{itemize}
-The scheduler may be viewed as a complex state machine.  The goal is
-to design the scheduler so that rules may be specified independently
-from the engine which parses the rules to determine which specific jobs
-to send to the controller.
-
-\subsubsection{User Interface}
-
-The IPP Scheduler shall possess a user interface which allows a human
-operator, or other processes, to monitor the current state of the
-Scheduler.  Users have the option to specify that a particular task or
-set of tasks is of higher or lower urgency (as defined in
-Section~\ref{sec:Controller}) than the norm, or to schedule a
-particular tasks on a different timescale from the basic rule.
-
-The IPP Scheduler defines the operating state of the IPP and shares
-the same set of states:
-\begin{itemize}
-\item active state
-\item interactive state
-\item paused state
-\end{itemize}
-When the IPP Scheduler is in the {\em active state}, it performs the
-most appropriate of all possible tasks at a particular time.  When the
-IPP Scheduler is in the {\em interactive state}, it performs only a
-specific requested action regardless of the outcome of the decision
-trees.  In addition, in the interactive state, the IPP Scheduler must
-only perform the requested actions and not attempt to perform the
-other normally-required actions.  The only exception to this exclusion
-is that, in the interactive state, data is still copied from the
-summit system.  An additional IPP state is the {\em paused state},
-intended for tests or maintenance, in which case the IPP Scheduler
-does not perform even the data copy tasks.  Every task is performed on
-demand by the user.  A user command sets the IPP Scheduler in one of
-these three states, {\em active}, {\em interactive}, and {\em paused}.
+
+\subsubsection{Miscellaneous Commands}
+
+It is possible to check the status of a single host or job with the
+user command \code{check}.
+
+pcontrol continuously examines the stack of jobs, adjusting their
+state as needed and extracting their output when it is ready. These
+checks are performed in the background, with pcontrol ready to accept
+further commands from the user in the foreground. These checks are
+performed after every keystroke, and also after an inactivity
+timeout. The interrupt interval defaults to 1 second, but may be
+adjusted with the pulse command, which takes as an argument, the
+number of microseconds for the timeout.
+
+The pcontrol system status may be examined with the command
+status. This provides a dump of the job stacks and the host stacks.
+
+It is possible to list the jobs currently in a specific stack,
+corresponding to the list of jobs with a given state. This is done
+with the command jobstack (stackname). The valid stack names are
+pending, busy, exit, crash, and done. The result is a list of all jobs
+on the specified stack. This is useful to determine quickly which jobs
+have exited or crashed.
+
+A specific job may be killed with the command \code{kill
+(JobID)}. This command is only valid for a job in the busy state. Any
+job in the pending, exit, or crash state may be deleted with the
+\code{delete (JobID)} command. This is necessary to free the memory
+associated with the job and its output streams.  PanTasks
+automatically performs these job harvesting functions.
+
+The command \code{verbose (mode)} turns the verbosity of the pcontrol
+operations on or off.
+
+The pcontrol and Nebulous have related needs for information from the
+combined storage-and-processing nodes regarding which nodes are
+available. Currently, the two systems independently examine the
+hardware to judge the availability.  It is not yet clear if this
+information is best stored in a single location (either pcontrol or
+Nebulous), which provides the information to other systems on demand.
+The current implementation allows the IPP system as a whole to
+distinguih nodes which are available for processing from those that
+are available to serve data as part of Nebulous.
+
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.06.ps}
+\caption{\label{PControlLoop} PControl Job Monitor Loop}
+\end{center}
+\end{figure}
+
+Figure~\ref{PControlLoop} illustrated the pcontrol job monitor loop.
+This is very similar to the loop in PanTasks.  A background process
+launched by readline during idle periods cycles through the list of
+hosts (children) and migrates jobs to and from them as needed.
+
+\subsubsection{pclient}
+
+pclient is the remote process monitor for pcontrol, the parallel
+process controller.
+
+The program pclient is used to support the remote jobs which are run
+on the remote hosts by pcontrol. The concept of pclient is to act as a
+buffer between the job running on the remote host and pcontrol. The
+pcontrol design uses (by default) ssh connections initiated by
+pcontrol to the remote hosts. These connections execute the remote
+program of pclient. The use of a remote login process lets the UNIX
+system take care of the user authentication issues. In this case, the
+recommended practice is to set up ssh to allow the connection to the
+remote host without additional authentication using the appropriate
+authorized keys.  The security is maintained by the security of ssh
+logins to the computers used by PanTask and pcontrol.  
+
+It is convenient to keep a continuous connection to the remote
+hosts. This avoids incurring the overhead of authentication for each
+command which is executed, while keeping a high-quality user
+authentication process in place.
+
+pclient acts as a buffer between pcontrol and the remote background
+process, allowing the continuous connection to remain viable without
+samping pcontrol with output from the jobs.  
+
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.07.ps}
+\caption{\label{queues} PanTasks queues and MDDB tables}
+\end{center}
+\end{figure}
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
@@ -2507,21 +2795,18 @@
 object analysis in the other analysis stages.
 
-\subsection{AstroRef: Astrometric Reference Catalog creation}
-
-\tbd{needs to be fleshed out substantially}
-
-This processing stage shall use many observations over a given time
-period to fit a consistent global astrometric solution, resulting in a
-high quality and internally-consistent astrometric catalog that may be
-published.
-
-\subsection{PhotoRef: Photometric Reference Catalog creation}
-
-\tbd{needs to be fleshed out substantially}
-
-This processing stage shall use many observations over a given time
-period to fit a consistent global photometric solution, resulting in a
-high quality and internally-consistent photometric catalog that may be
-published.
+\section{IPPtools}
+
+Above, we discussed PanTasks, the IPP scheduler which determines the
+new jobs to run and distributes them to computers across the network.
+PanTasks is a general tool; by it self it does not define the specific
+analysis tasks that the IPP requires.  The previous few sections
+discussed in detail the analysis which is performed by the IPP
+analysis stages.  IPPtools is the collection of PanTasks scripts,
+Metadata Database interaction programs, and other tools used to
+definet the specific analysis stages of the IPP. 
+
+\tbd{this section needs to be fleshed out with a summary of the
+  ippTools functions.  The stand-alone IPPTools document gives a
+  detailed discussion of these issues}.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
@@ -3968,4 +4253,136 @@
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
+\section{PanTasks Example : Pcopy.pro}
+
+Below is an example script for psched which demonstrates the
+scheduling system. This parallel-copying script implements the
+Pan-STARRS image copying system, which requests images from the summit
+and copies them to the appropriate computer. The first task in the
+script queries an external system for new image names with the
+function new.images. In the case of Pan-STARRS, this would be a
+request from OTIS, the observatory controlling system. The second task
+initiates the individual image copies, with separate CCDs being copied
+to separate computers. This script uses the concept of having specific
+machines assigned to specific CCDs (as Pan-STARRS intends to
+operate). The association is determined by calling the external
+function chip.host, providing the identifier of the chip in
+question. This returns an appropriate host. The copy.image function
+copies the file and also sends a message to the summit system to
+inform it that the image has been successfully copied.
+
+\begin{verbatim}
+verbose on
+ queueinit newImages
+ exec echo 0 > new.last
+ exec cp -f raw.list new.list
+ 
+ controller host add po01
+ controller host add po02
+ controller host add po03
+ controller host add po04
+ 
+ # identify the images ready for copy 
+ # new entries are added to queue newImages
+ # need to compare the new list with the ones already being processed
+ task	       new.images
+   command      new.images
+   host         local
+ 
+   periods      -poll 1
+   periods      -exec 5
+   periods      -timeout 5
+ 
+   # success
+   task.exit    0
+     local i j Nstdout Nimages
+     # compare output with new.image queue
+     # keep only new entries
+     queuesize stdout -var Nstdout
+     for i 0 $Nstdout
+       queuepop stdout -var line
+       queuepush newImages -uniq -key 0 "$line"
+     end
+   end
+ 
+   # locked list
+   task.exit    1
+     echo       "new.images: exec failure"
+     $new.image.failure ++
+   end
+ 
+   # default exit status
+   task.exit    default
+     echo       "new.images: unknown exit status: $EXIT"
+     $new.image.failure ++
+   end
+ 
+   # operation times out?
+   task.exit    timeout
+     echo       "new.images: timeout"
+     $new.image.failure ++
+   end
+ end
+ 
+ # copy new images, sending job to desired host
+ task	       copy.images
+   periods      -poll 0.2
+   periods      -exec 1
+   periods      -timeout 5
+ 
+   task.exec
+     queuesize  newImages -var N
+     if ($N == 0) break
+     # if ($network == 0) break
+     # if ($filesystem == 1) break
+     
+     queuepop newImages -var line
+     list tmp -split $line
+     $filename   = $tmp:0
+     $chip       = $tmp:1
+     $state      = $tmp:2
+     if ($state == new) 
+       # copy this image
+       queuepush newImages -replace -key 0 "$filename $chip run"
+     else
+       # ignore this image
+       queuepush newImages -replace -key 0 "$filename $chip $state"
+       break
+     end
+     # echo $chip
+     $host = `chip.host $chip`
+     # echo $host
+     host $host
+     # echo "starting copy for $filename on $host..."
+     command copy.image $filename $chip
+   end
+ 
+   # can I have access to argc,argv?
+ 
+   # success
+   task.exit    0
+     echo "done copy..."
+     queuepop stdout -var line
+     list tmp -split $line
+     $filename   = $tmp:0
+     $chip       = $tmp:1
+     exec mark.image $filename
+     queuepush newImages -replace -key 0 "$filename $chip copy"
+   end
+ 
+   # default exit status
+   task.exit    default
+     echo       "new.images: unknown exit status: $EXIT"
+     $new.image.failure ++
+   end
+ 
+   # operation times out?
+   task.exit    timeout
+     echo       "new.images: timeout"
+     $new.image.failure ++
+   end
+ end
+ 
+\end{verbatim}
+
 \input{glossary.tex}
 
@@ -3976,5 +4393,4 @@
 * output data products
 * DVO
-* PanTasks
 * ipptools / ippMonitor?
 * analysis stages, versions and iterations
Index: /trunk/doc/dvo/dvo.tex
===================================================================
--- /trunk/doc/dvo/dvo.tex	(revision 6054)
+++ /trunk/doc/dvo/dvo.tex	(revision 6055)
@@ -19,5 +19,94 @@
 \pagenumbering{arabic}
 
-\subsection{Photometric systems and the DVO Photcodes}
+\section{Overview}
+
+DVO, the Desktop Virtual Observatory, is a software system which
+stores data related to astronomical objects derived from various
+sources, and provides mechanisms to related multiple detections
+together as astronomical objects.  DVO deals with two related
+concepts: {\em objects} and {\em detections}.  The {\em objects} are
+descriptions of astronomical objects while the {\em detections} are
+the specific measurements of those objects, typically measured from
+astronomical images.  A collection of {\em detections} may be used to
+derive average quantities which describe a particular {\em object}.  A
+third class of measurement to be considered are those supplied by
+external references.  Such measurements may be treated as {\em
+detections}, with the caveat that access to the raw measurements and
+metadata are usually unavailable: the reported measurements and errors
+must be accepted as they are reported.
+
+DVO stores the collections of detections which were derived from
+specific images.  It provides a mechanism to determine the image from
+which a specific detection was derived, and in conjunction with the
+Image Server locate the corresponding data file.  DVO also makes it
+possible to extract all detections derived from a specific image and
+to determine quantities such as the pixel coordinates of the detection
+on the image.
+
+DVO also has the capability to associate multiple detections of a
+specific object.  Several major classes of objects will be present,
+each of which must be handled correctly.  DVO distinguished the
+following types of objects.
+
+{\bf Stars, compact galaxies, and QSOs} will have nearly fixed
+locations relative to other distant stars, with only small deviations
+for individual measurements.  The association between multiple
+detections of such objects is made on the basis of their coincident
+positions.  DVO determines the average position of the object and the
+deviations of the individual detections from that average on the basis
+of the ensemble of individual detection.
+
+{\bf Solar System Objects} do not have a fixed location.  Detections
+of such objects are linked by their orbits, and depend on both the
+position and the time of the image.  DVO does not attempt to make this
+link; this is the role of the MOPS system.  However, it has the
+ability to accept identifications made externally with specified
+detections and to return the identifier of the moving object
+associated with the specific detections.  These associations also
+include descriptive information such as the offset of the detection
+from the predicted location of the detection based on the orbit.  This
+functionality is required to allow DVO to ignore known moving object
+detections from other types of queries.
+
+{\bf High-proper-motion objects} in the general vicinity of the solar
+system fall in between these first two classes of objects.  Their
+proper motion and parallax response is significant enough ($>0.2$
+arcsec in 1 year) that they are not well-described by an average
+location and a collection of offsets.  These objects are better
+described by a distance and a proper motion vector.  DVO provides the
+association between the specific detections and an average object
+which includes finite parallax and proper motion.
+
+{\bf Orphaned detections} are not associated with a specific
+astronomical object of any of the above classes.  Most of these will
+be spurious (not representing real objects), some will be from solar
+system objects for which orbits are not yet determined, some will be
+from faint stars near the detection limits, and some will be from
+short-term transients which have only been detected once.  DVO
+maintains these detections until they have been associated with one of
+the objects above.  DVO provides mechanisms by which individual
+detections may be migrated back and forth between the orphan state and
+association with an astronomical object.
+
+DVO stores the information about the detection, the related objects,
+and the images which provided the measurements.  For every detection,
+DVO provides the mechanisms to link the detection back to the image
+which supplied it.  DVO also provides the capability to determine the
+images containing a specific location but for which no detection was
+made.  The minimum set of information which must be carried for these
+non-detections is the image and the associated object or orphan.
+
+DVO also stores the relationships between various
+photometric systems and the evolution of that relationship.  It
+provides mechanisms to convert between the measured instrumental
+magnitude of a detection with a specific filter, detector, and
+telescope, and at a particular time and the implied magnitude in the
+average Pan-STARRS photometry system, given a determined set of
+calibrations.  It also provides the capability to convert magnitudes
+in one system to the magnitudes in another system; an example of such
+a conversion is between the average Pan-STARRS filter systems and the
+various reference systems appropriate for those filters.
+
+\section{Photometric systems and the DVO Photcodes}
 
 One of the major roles of DVO is to relate different photometric
@@ -26,8 +115,8 @@
 number of different detectors.  We may have observations from
 different telescopes in similar filters.  We may have reference data
-related to some filter, but obtained and published by other
-observers.  We would like to related these measurements together in
-optimal ways, making use of whatever information we have available.
-DVO provides several mechanisms to enable these relationships.
+related to some filter, but obtained and published by other observers.
+We would like to related these measurements together in optimal ways,
+making use of whatever information we have available.  DVO provides
+several mechanisms to enable these relationships.
 
 We identify three distinct types of photometry measurements within
@@ -208,93 +297,4 @@
 parameters.
 
-\section{Overview}
-
-DVO, the Desktop Virtual Observatory, is a software system which
-stores data related to astronomical objects derived from various
-sources, and provides mechanisms to related multiple detections
-together as astronomical objects.  DVO deals with two related
-concepts: {\em objects} and {\em detections}.  The {\em objects} are
-descriptions of astronomical objects while the {\em detections} are
-the specific measurements of those objects, typically measured from
-astronomical images.  A collection of {\em detections} may be used to
-derive average quantities which describe a particular {\em object}.  A
-third class of measurement to be considered are those supplied by
-external references.  Such measurements may be treated as {\em
-detections}, with the caveat that access to the raw measurements and
-metadata are usually unavailable: the reported measurements and errors
-must be accepted as they are reported.
-
-DVO stores the collections of detections which were derived from
-specific images.  It provides a mechanism to determine the image from
-which a specific detection was derived, and in conjunction with the
-Image Server locate the corresponding data file.  DVO also makes it
-possible to extract all detections derived from a specific image and
-to determine quantities such as the pixel coordinates of the detection
-on the image.
-
-DVO also has the capability to associate multiple detections of a
-specific object.  Several major classes of objects will be present,
-each of which must be handled correctly.  DVO distinguished the
-following types of objects.
-
-{\bf Stars, compact galaxies, and QSOs} will have nearly fixed
-locations relative to other distant stars, with only small deviations
-for individual measurements.  The association between multiple
-detections of such objects is made on the basis of their coincident
-positions.  DVO determines the average position of the object and the
-deviations of the individual detections from that average on the basis
-of the ensemble of individual detection.
-
-{\bf Solar System Objects} do not have a fixed location.  Detections
-of such objects are linked by their orbits, and depend on both the
-position and the time of the image.  DVO does not attempt to make this
-link; this is the role of the MOPS system.  However, it has the
-ability to accept identifications made externally with specified
-detections and to return the identifier of the moving object
-associated with the specific detections.  These associations also
-include descriptive information such as the offset of the detection
-from the predicted location of the detection based on the orbit.  This
-functionality is required to allow DVO to ignore known moving object
-detections from other types of queries.
-
-{\bf High-proper-motion objects} in the general vicinity of the solar
-system fall in between these first two classes of objects.  Their
-proper motion and parallax response is significant enough ($>0.2$
-arcsec in 1 year) that they are not well-described by an average
-location and a collection of offsets.  These objects are better
-described by a distance and a proper motion vector.  DVO provides the
-association between the specific detections and an average object
-which includes finite parallax and proper motion.
-
-{\bf Orphaned detections} are not associated with a specific
-astronomical object of any of the above classes.  Most of these will
-be spurious (not representing real objects), some will be from solar
-system objects for which orbits are not yet determined, some will be
-from faint stars near the detection limits, and some will be from
-short-term transients which have only been detected once.  DVO
-maintains these detections until they have been associated with one of
-the objects above.  DVO provides mechanisms by which individual
-detections may be migrated back and forth between the orphan state and
-association with an astronomical object.
-
-DVO stores the information about the detection, the related objects,
-and the images which provided the measurements.  For every detection,
-DVO provides the mechanisms to link the detection back to the image
-which supplied it.  DVO also provides the capability to determine the
-images containing a specific location but for which no detection was
-made.  The minimum set of information which must be carried for these
-non-detections is the image and the associated object or orphan.
-
-DVO also stores the relationships between various
-photometric systems and the evolution of that relationship.  It
-provides mechanisms to convert between the measured instrumental
-magnitude of a detection with a specific filter, detector, and
-telescope, and at a particular time and the implied magnitude in the
-average Pan-STARRS photometry system, given a determined set of
-calibrations.  It also provides the capability to convert magnitudes
-in one system to the magnitudes in another system; an example of such
-a conversion is between the average Pan-STARRS filter systems and the
-various reference systems appropriate for those filters.
-
 \section{DVO Database Tables}
 
@@ -499,4 +499,6 @@
 data types and input/output methods without significant re-coding.
 
+\tbd{DVO mysql table storage is not yet implemented}
+
 \section{addstar : Insert Image \& Detection Set}
 
@@ -505,6 +507,4 @@
 \caption{\label{catalog} \small a figure }
 \end{figure}
-
-\tbd{fill out discussion of the addstar client/server implementation}
 
 One of the most basic operations needed by DVO is to insert a
@@ -527,123 +527,167 @@
 faint orphans.
 
-\subsection{addstar -refs : Insert Reference Objects} 
-
-This operation is very similar to the previous one.  A collection of
-reference objects are added to the database as a collection of
-detections.  The reference photometry should in general be given its
-own photometry code.  The reference data is different from the image
-detection set because the associated image information is not
-included.  Thus, no corresponding images are added to the database.
-
-\section{relphot : Relative Photometry Analysis}
+A wide range of options are available to addstar.  These can be used
+to modify the object matching rules, to reduce the number of tables
+which are updated, to specify the output data format, and so forth.  A
+few options modify the behavoir in substantial ways, as discussed in
+the two sections below.
+
+\tbd{flesh out discussion of the options}
+
+\subsection{Insert Reference Objects} 
+
+\code{addstar -ref (filename)}
+
+This mode of addstar reads a text file and adds the listed objects to
+the database as a reference photcode type.  A collection of reference
+objects are added to the database as a collection of detections.  The
+reference photometry should in general be given its own photometry
+code.  The reference data is different from the image detection set
+because the associated image information is not included.  Thus, no
+corresponding images are added to the database.
+
+\subsection{Insert Catalog Objects} 
+
+\code{addstar -cat (name) -region ra ra dec dec}
+
+In this mode, any of several all-sky or large-scale reference catalogs
+are used for the input sources.  The catalog objects are added to the
+database as reference objects.  The valid catalogs consist of 2MASS,
+USNO, GSC.  Tycho and USNO-B will be added shortly.  Specific
+photcode names are defined for each of these catalogs, and must be
+appropriately requested and defined in the photcode table.  The
+optional region restriction limits the insert to a subset of the sky.
+The user does not always want to add 50GB of 2MASS detections to any
+DVO database... 
+
+\subsection{Addstar Client/Server Interactions}
+
+DVO currently uses stand-alone programs which are run from the command
+line (like addstar, or the programs listed below), or it works with
+the interactive DVO shell, which allows the user to query portions of
+the database.  These programs all interact with the database tables
+directly, making use of file locking to prevent conflicts.  
+
+Unlike the other DVO programs (currently), it is possible to run
+addstar as a client/server system.  In this configuration, the program
+\code{addstard} is launched to run in the background as a server.  It
+monitors a socket waiting for clients to contact it.  The client
+program, \code{addstarc} appears to the user identical to the
+stand-alone addstar.  However, rather than directly insert data into
+the database, \code{addstarc} contacts the addstar server and sends it
+the detections and associated image data (along with the information
+about the user options).  The daemons accepts the incoming data and
+then loads this data into the database, just as the stand-alone
+addstar does.
+
+The purpose of the addstar client/server design is three-fold.  First,
+the client can be used by processes to send data to the DVO database
+and then immediately exit.  The addstar loading process is one of the
+more time-critical functions within the IPP.  However, unlike the
+other portions of the IPP, the addstar processes must operate in
+serial, at least when they are updating the same portion of the sky
+(or the image table).  If the IPP analysis routines all needed to run
+the stand-alone addstar program, they would eventually block waiting
+for each addstar to complete, preventing other processing from
+continuing.  The addstar client / server model allows the processing
+node to invoke the addstar client, sending the data to the addstar
+server.  The addstar server will then be the entity that manages the
+serialization of the incoming data stream.  The addstar server has two
+threads which run in parallel.  One thread monitors the socket and
+accepts new data sets from addstar clients, adding the data to an
+internal queue.  The other thread pulls data off of the queue and
+updates the database with the data.  
+
+A second advantage of the client/server interaction is that only the
+new detections need to be sent across the network.  To update the
+database, addstar must load the average objects for the region from
+the database tables.  In the stand-alone mode, the addstar program
+loads this data via NFS across the network from whatever device stores
+the addstar tables.  In the client/server model, the addstar server
+always runs locally on the machine which holds the database tables.
+Thus, for the server, all database access is local disk access.
+
+The final advantage of the client/server model is that it enables the
+parallel database model, which is not yet implemented as of Jan
+2006. In this model, there are multiple addstar servers.  Each one has
+a fraction of the sky in the local tables.  The identification of
+which table is managed by this host/addstar server is stored in the
+SkyRegion table.  The addstar server simply accepts incoming
+detections from the addstar clients.  Any detections which it receives
+which fall within the boundaries of tables that it manages are updated
+as normal.  The server then identifies the other addstar servers which
+are responsible for the other detections.  It then sends these
+detections to those servers using the same socket communication used
+by the addstar clients.  The addstar server must also be ready accept
+detections from other addstar servers.  This relationship is
+completely parallel, and any addstar client may send its data to any
+addstar server, letting the servers hash out who owns what.  The only
+difficulty with this model is in handling sources near the boundaries
+of the tables.  Note that this issues exists whether those tables are
+distributed across multiple machines or not.
+
+Addstar uses the following strategy to handle detections on the table
+boundaries.  Detections are first added to each table completely
+ignoring the neighboring tables.  A detection which is close to the
+boundary may either be associated with an average object contained
+within the table, or not.  If it is, the detection is associated with
+that average object.  If not, a new average object is created at the
+location of the detection.  So far, this process is identical to the
+behavoir in the middle of the table.  One a longer time-scale, a
+process is run which mediates the table boundaries. In this analysis,
+the two neighboring tables are simultaneously examined.  The border
+region, in a strip wider than the correlation radius, is examined in
+detail.  If two objects within the border region fall within 2x the
+correlation radius of each other, their individual detections are
+re-examined.  These detections are re-added to a temporary table which
+encompases the overlap.  the resulting objects will in general have
+detections from either side of the boundary.  The average objects are
+kept within the table as normal, but the detections are allowed to
+migrate between the tables to stay with their object.  \tbd{this
+boundary cleanup process is not implemented to date}.
+
+\section{Relphot : Relative Photometry Analysis}
 
 This operation uses the overlaps of images and multiple observations
 of the same objects to determine the relative photometry zero-points
-for a collection of images.  This is a task that wil be run much more
+for a collection of images.  This is a task that is run much more
 infrequently than the object insertion tasks.
 
-\section{relphot}
-
-\begin{verbatim}
-    * load data
-          o images: match photcode and time range, reset flags
-          o measure: select subset matching restritions on: photcode, time, dM, Minst, Mag, dophot == 1 
-    * iterate to find Mcal, image.flags:
-    * write out modified Mcal values to image table
-    * write out modified Mcal values to measure table
-    * write out modified Mrel values to average table 
-\end{verbatim}
-
-relphot has two primary purposes:
-
-\begin{verbatim}
-    * calculate Mcal for images / measures
-    * calculate Mrel for stars 
-\end{verbatim}
-
-relphot can also be used to determine the mosaic grid used to generate photometrically corrected flats (-grid option).
-
-\subsection{data exclusion}
-
-relphot uses only a subset of the photometry data to calculate Mcal
-and Mrel. In the first stage, calculation of the Mcal values, relphot
-loads the photometry data from each relevant catalog and creates an
-internal subset catalog with the function bcatalog, excluding some of
-the irrelevant data. In addition, it uses flags to mark some of the
-data as invalid for the processing. bcatalog exclusions
-
-\begin{verbatim}
-    * measure.photcode not equivalent to requested photcode
-    * measure.dophot != 1
-    * measure.Mcat > MAG_LIM
-    * measure.dM > SIGMA_LIM
-    * measure.Minst out of range (ImagMin - ImagMax) [optional]
-    * measure.t out of range (TSTART, TSTOP) 
-\end{verbatim}
-
-flagged data 
-flagged image data
-
-images can be flagged by setting bits of image.code stars can be
-flagged by setting bits of average.code measures can be flagged by
-setting bits of measure.flag
-
-\begin{verbatim}
-image.code
-ID_IMAGE_NOCAL : ignore, irrelevant 
-ID_IMAGE_POOR : image measured bad
-ID_IMAGE_SKIP : externally known bad dMcal > VALUE 
-FLAG_IMAGE_SCATTER clean_images fabs(Mcal) > VALUE 
-FLAG_IMAGE_ZEROPT clean_images dMcal > VALUE 
-FLAG_IMAGE_SCATTER clean_mosaics fabs(Mcal) > VALUE
-FLAG_IMAGE_ZEROPT clean_mosaics mark_images does not seem to do
-anything useful? 
-average.code Ngood < MEAS_TOOFEW setMrel Ngood <
-MEAS_TOOFEW clean_measures ChiSq > STAR_CHISQ clean_stars dM >
-STAR_SCATTER clean_stars average.code (STAR_BAD) is not saved by
-relphot: it is set by clean_stars, clean_measures, and setMrel, but
-not setMrelOutput. STAR_BAD should only be internal since it depends
-on the photcode, but is not associated with a specific photcode in the
-data. Just in case, it is reset to 0 in setMrelFinal. measure.flag X,Y
-out of range setExclusions 3 sigma clipping clean_measures
-\end{verbatim}
-
-setting Mrel final value
-
-setMrelFinal is used to set the final average.Mrel values. We do this
-in 4 stages. In each stage, we set the Mrel values for stars which
-have not already been set, based on the current exclusion settings. At
-successive stages, we relax the exclusions, allowing the more spurious
-objects to have a valid Mrel value to be set. In this loop, we
-actually run setMrelOutput twice: once to get the approximate Mrel
-value, then we flag the outlier measurements with
-\code{clean_measure}, then we redetermine the Mrel values on this
-basis, and mark the stars for exclusion from the next iteration.
-
-\begin{verbatim}
- exclude on
-  photcode       0 1 2 3
-  time range     0 1 2 3
-  MEAS_POOR      0 1 2 3
-  MEAS_TOOFEW    0 1 2 3
-  dophot == 10   0 1 2 
-  inst mag       0 1 2 
-  dophot != 1,2  0 1  
-  ID_IMAGE_POOR  0 1
-  ID_IMAGE_SKIP  0 1
-  dophot != 1    0
-  measure.dM     0 
-\end{verbatim}
- 
-for all relphot runs, Mrel is re-calculated, and measures are marked at least if they are outliers in mag or ccd area. setMrel.output needs to do a few things differently from setMrel:
-
-\begin{verbatim}
-    * set measure.Mcal (skipped in setMrel.basic)
-    * set average.Mrel if N < TOO_FEW (not STAR_BAD) (optional!)
-    * use MAX (stats.error, stats.sigma) (optionally)
-    * allow STAR_BAD? 
-\end{verbatim}
-
-\section{uniphot : Zero Point Analysis}
+The relphot analysis is currently performed with a single Sky region
+as the starting point.  All images (or all chips from all mosaic
+iamges) which overlap the sky region are identified in the image
+table.  This set of images are considered set A.  Next, all skyregions
+which are overlapped by all of these images are selected.  Finally,
+all additional images which overlapped the new regions only are
+selected.  These are considered as image set B.  The image selections
+are also restricted to images of a single, user-selected photcode.  
+
+All of the objects and detections which are contributed by the images
+in sample A are extracted from the average and measure tables.  Only a
+subset of the detections for which the S/N is greater than a
+user-selected limit are kept.  Other restrictions, such as time range
+or instrumental magnitude ranges may also be specified.  The
+collection of average objects, their detections, and the images from
+which they were derived now define a system of photometry equations.
+In this system, every image has a calibration offset magnitude
+($M_{cal}$), every object has an average magnitude in a relative
+system ($M_{rel}$), and every detection of that object has a magnitude
+defined by the equation $M = M_{rel} + M_{cal}$.  The goal is to solve
+for the values of $M_{ref}$ and $M_{cal}$.  
+
+There are two points to note about this operation.  First, the system
+of equations is generally much too large to solve directly; we must
+use an iterative technique to converge on a solution. Second, it is
+important in the analysis to use robust averaging and identify
+detections, stars, or images which are deviant in some way.  These
+should be marked and given set weight in the solution.  These cases
+may represent poorly measured objects (perhaps detections on or near a
+bad column), variable stars, and images obtained in poor weather
+conditions.
+
+Relphot can also be used to determine the mosaic grid used to generate
+photometrically corrected flats (-grid option).
+
+\section{Uniphot : Zero Point Analysis}
 
 This operation uses the time history of relative photometry zero
@@ -1482,2 +1526,85 @@
 \end{itemize}
 
+
+
+\subsection{Relphot data exclusion}
+
+relphot uses only a subset of the photometry data to calculate Mcal
+and Mrel. In the first stage, calculation of the Mcal values, relphot
+loads the photometry data from each relevant catalog and creates an
+internal subset catalog with the function bcatalog, excluding some of
+the irrelevant data. In addition, it uses flags to mark some of the
+data as invalid for the processing. bcatalog exclusions
+
+\begin{verbatim}
+    * measure.photcode not equivalent to requested photcode
+    * measure.dophot != 1
+    * measure.Mcat > MAG_LIM
+    * measure.dM > SIGMA_LIM
+    * measure.Minst out of range (ImagMin - ImagMax) [optional]
+    * measure.t out of range (TSTART, TSTOP) 
+\end{verbatim}
+
+flagged data 
+flagged image data
+
+images can be flagged by setting bits of image.code stars can be
+flagged by setting bits of average.code measures can be flagged by
+setting bits of measure.flag
+
+\begin{verbatim}
+image.code
+ID_IMAGE_NOCAL : ignore, irrelevant 
+ID_IMAGE_POOR : image measured bad
+ID_IMAGE_SKIP : externally known bad dMcal > VALUE 
+FLAG_IMAGE_SCATTER clean_images fabs(Mcal) > VALUE 
+FLAG_IMAGE_ZEROPT clean_images dMcal > VALUE 
+FLAG_IMAGE_SCATTER clean_mosaics fabs(Mcal) > VALUE
+FLAG_IMAGE_ZEROPT clean_mosaics mark_images does not seem to do
+anything useful? 
+average.code Ngood < MEAS_TOOFEW setMrel Ngood <
+MEAS_TOOFEW clean_measures ChiSq > STAR_CHISQ clean_stars dM >
+STAR_SCATTER clean_stars average.code (STAR_BAD) is not saved by
+relphot: it is set by clean_stars, clean_measures, and setMrel, but
+not setMrelOutput. STAR_BAD should only be internal since it depends
+on the photcode, but is not associated with a specific photcode in the
+data. Just in case, it is reset to 0 in setMrelFinal. measure.flag X,Y
+out of range setExclusions 3 sigma clipping clean_measures
+\end{verbatim}
+
+setting Mrel final value
+
+setMrelFinal is used to set the final average.Mrel values. We do this
+in 4 stages. In each stage, we set the Mrel values for stars which
+have not already been set, based on the current exclusion settings. At
+successive stages, we relax the exclusions, allowing the more spurious
+objects to have a valid Mrel value to be set. In this loop, we
+actually run setMrelOutput twice: once to get the approximate Mrel
+value, then we flag the outlier measurements with
+\code{clean_measure}, then we redetermine the Mrel values on this
+basis, and mark the stars for exclusion from the next iteration.
+
+\begin{verbatim}
+ exclude on
+  photcode       0 1 2 3
+  time range     0 1 2 3
+  MEAS_POOR      0 1 2 3
+  MEAS_TOOFEW    0 1 2 3
+  dophot == 10   0 1 2 
+  inst mag       0 1 2 
+  dophot != 1,2  0 1  
+  ID_IMAGE_POOR  0 1
+  ID_IMAGE_SKIP  0 1
+  dophot != 1    0
+  measure.dM     0 
+\end{verbatim}
+ 
+for all relphot runs, Mrel is re-calculated, and measures are marked at least if they are outliers in mag or ccd area. setMrel.output needs to do a few things differently from setMrel:
+
+\begin{verbatim}
+    * set measure.Mcal (skipped in setMrel.basic)
+    * set average.Mrel if N < TOO_FEW (not STAR_BAD) (optional!)
+    * use MAX (stats.error, stats.sigma) (optionally)
+    * allow STAR_BAD? 
+\end{verbatim}
+
Index: /trunk/doc/pantasks/pantasks.tex
===================================================================
--- /trunk/doc/pantasks/pantasks.tex	(revision 6054)
+++ /trunk/doc/pantasks/pantasks.tex	(revision 6055)
@@ -24,5 +24,5 @@
 tool which manages the sequencing of data analysis steps and, with the
 related tool `PControl', distributes the data processing across a
-cluster of computers.
+cluster of computers. \tbd{uses the 'opihi' shell-scripting system}
 
 The purpose of PanTasks is to manage the automatic construction and
@@ -30,82 +30,7 @@
 uses a set of rules to define UNIX commands, and their corresponding
 command-line arguments, to be performed on some regular, repeated
-basis.  The utility of PanTasks is that it can easily define an
-analysis system which is completely state-based, as opposed to an
-event-driven system.  
-
-The two basic units of PanTasks operation are the 'task' and the
-'job'.  A 'job' is simply a command the user would execute on a
-command line; it consists of a command along with optional command
-line arguments.  A 'task' is a generic description of a type of job in
-which the details of any specific piece of data are omitted.  The task
-defines the UNIX command which corresponds to the job, and it provides
-rules for determining the identity of data for the job.  The task also
-defines tests which are used to decide if the job may be executed.
-Finally, it defines a polling frequency in which PanTasks should
-attempt to construct a new job for the task.
-
-For example, we may want to regularly copy files from one location to
-another.  Perhaps the name of the next available file is available in
-a database table, and can be retrieved with the command 'nextFile'.
-Perhaps we wish to check for new files every 60 seconds.  Thus, the
-task is to copy files, while a specific job of this task might be of
-the form \code{cp newfile newpath}.  With PanTasks, we would define a
-copy task somewhat like the following:
-
-\begin{verbatim}
-  task copyfile
-    periods -exec 60.0
-
-    task.exec
-      $file = `nextFile`
-      if ($file == "none")
-        break
-      end
-      command cp $file $newpath
-    end
-
-    task.exit 0
-      queuepush copied $file
-    end
-
-    task.exit 1
-      queuepush failure $file
-    end
-  end
-\end{verbatim}
-
-In this simple example, the task is attempted every 60 seconds.  If
-there is no new file (output of 'nextFile' is 'none'), then no job
-results from this task.  If there is a new file, the copy command is
-performed.  This example is deceptively simple because one could
-easily imagine writing a stand-along program which performs the same
-thing.  The important advantages of using PanTasks for this type of
-operation are: 
-\begin{itemize}
-\item the output from one job can be used to spawn a number of other tasks.
-\item the success or failure state of each job can be used to spawn
-  other tasks.
-\item the details of the job and data test are kept separated from the
-  rules which connect tasks together.  
-\item the relationships between tasks are kept together in a single
-  location.
-\end{itemize}
-
-In addition to these organizational advantages, PanTasks also provides
-a direct connection to the tool for monitoring parallel jobs,
-pcontrol.  Thus the jobs spawned by PanTasks can be defined to run in
-the background locally or on any of the computers in the parallel
-processing cluster.
-
-
-\section{PanTasks : the Scheduler}
-
-The purpose of PanTasks is to manage the automatic construction and
-execution of inter-related (often repetative) operations. PanTasks uses
-a set of rules to define UNIX commands, and their corresponding
-command-line arguments, to be performed on some regular, repeated
-basis. The utility of PanTasks is that it can easily define an analysis
-system which is completely state-based, as opposed to an event-driven
-system.
+basis.  The utility of PanTasks is that it can easily define and
+manage an analysis system which is completely state-based, as opposed
+to an event-driven system.
 
 Consider, for example, a telescope which obtains a collection of
@@ -149,103 +74,87 @@
 \subsection{Tasks vs Jobs}
 
-The primary function of PanTasks is to repeatedly perform tasks, and
-execute jobs on the basis of those tasks. A task consists of a set of
-rules which describe system state tests to perform on a regular time
-scale. Based on the results of those state tests, the task will then
-choose whether or not to construct a job. The task also defines
-actions to perform upon the completion of a job, based upon the output
-and exit status of the job. A task thus defines the repeat period. It
-may optionally define valid or invalid time ranges (eg, Mon-Fri or
-10:00-17:00, etc). The task may also specify that the job be run
-locally (ie, in the background on the same computer as PanTasks) or
-remotely by the parallel process controller (pcontrol). A job may even
-be restricted to a specific computer managed by pcontrol. An example
-of a simple tasks is given below.
-
-\begin{verbatim}
-   task datalist
-     command ls /data/foo
-     periods -exec 5.0
-     periods -timeout 50.0
-     periods -poll 1.0
- 
-     task.exit 0
-       queueprint stdout
-       queuedelete stdout
-     end
-  
-     task.exit 1
-       queuepush failure "task failed"
-     end
-   end
- \end{verbatim}
-
-
-This task does not perform any system state tests; it is simply
-constructs a new job every 5.0 seconds. The job in this case is always
-the same: ls /data/foo . When the job finished, if the job exit status
-is 0 (normal UNIX success status), the resulting output is printed to
-the screen. If the job returns an exit status of 1 (a failure), the
-failure queue receives a single entry. Although they are not defined
-in this case, it is also possible to specify the action to be taken if
-the job crashes (does not exit normally) or if it times out (runs
-beyond the specified timeout period). A slightly more complex task
-which performs a state test and constructs a command based on that
-test is shown below
-
-\begin{verbatim}
-   task datalist
-     periods -exec 5.0
-     periods -timeout 50.0
-     periods -poll 1.0
- 
-     task.exec 
-       $file = `next.file`
-       if ($file == "none")
-         break
-       end
-       command cp /data/foo/$file /data/bar
-     end
- 
-     task.exit 0
-       queueprint stdout
-       queuedelete stdout
-       queuepush copied $file
-     end
-  
-     task.exit 1
-       queuepush failure $file
-     end
-   end
-\end{verbatim}
-
- The task.exec macro is executed by PanTasks every 5.0 seconds. This
- macro executes a (hypothetical user-defined) UNIX command (next.file)
- which examines the system state, return either a filename or the word
- "none". If the result of this test is "none", the task does nothing:
- no job is constructed. Otherwise, a job is constructed using the name
- of the file returned by the state test. Successful jobs have the
- filename added to the 'copied' queue, while failed jobs add the
- filename to the 'failure' queue.
+The two basic units of PanTasks operation are the 'task' and the
+'job'.  A 'job' is simply a command the user would execute on a
+command line; it consists of a command along with optional command
+line arguments.  A 'task' is a generic description of a type of job in
+which the details of any specific piece of data are omitted.  The task
+defines the UNIX command which corresponds to the job, and it provides
+rules for determining the identity of data for the job.  The task also
+defines tests which are used to decide if the job may be executed.
+Finally, it defines a polling frequency in which PanTasks should
+attempt to construct a new job for the task.
+
+For example, we may want to regularly copy files from one location to
+another.  Perhaps the name of the next available file is available in
+a database table, and can be retrieved with the command 'nextFile'.
+Perhaps we wish to check for new files every 60 seconds.  Thus, the
+task is to copy files, while a specific job of this task might be of
+the form \code{cp newfile newpath}.  With PanTasks, we would define a
+copy task somewhat like the following:
+
+\begin{verbatim}
+  task copyfile
+    periods -exec 60.0
+
+    task.exec
+      $file = `nextFile`
+      if ($file == "none")
+        break
+      end
+      command cp $file $newpath
+    end
+
+    task.exit 0
+      queuepush copied $file
+    end
+
+    task.exit 1
+      queuepush failure $file
+    end
+  end
+\end{verbatim}
+
+In this simple example, the task is attempted every 60 seconds.  If
+there is no new file (output of 'nextFile' is 'none'), then no job
+results from this task.  If there is a new file, the copy command is
+performed.  This example is deceptively simple because one could
+easily imagine writing a stand-along program which performs the same
+thing.  The important advantages of using PanTasks for this type of
+operation are: 
+\begin{itemize}
+\item the output from one job can be used to spawn a number of other tasks.
+\item the success or failure state of each job can be used to spawn
+  other tasks.
+\item the details of the job and data test are kept separated from the
+  rules which connect tasks together.  
+\item the relationships between tasks are kept together in a single
+  location.
+\end{itemize}
+
+In addition to these organizational advantages, PanTasks also provides
+a direct connection to the tool for monitoring parallel jobs,
+pcontrol.  Thus the jobs spawned by PanTasks can be defined to run in
+the background locally or on any of the computers in the parallel
+processing cluster.
 
 \subsection{Parallel vs Local Job Processing}
 
 Jobs which are generated by PanTasks tasks may either be run locally
-(forked in the background on the same machine as PanTasks) or run on the
-IPP parallel process controller, pcontrol. The default is for the job
-to be run locally. If a job should be run on the parallel controller,
-this can be specified by including the command host (hostname) in the
-definition of a task. If the value of (hostname) is 'anyhost', then
-pcontrol may select any of its host computers to run the job according
-to its own rules. If the value of (hostname) is one of the computers
-managed by pcontrol, then that machine will be selected for the job,
-if it is available. This amounts to a preference to use that machine,
-but pcontrol is allowed to substitute a different machine if it
-chooses. If the host command is given the option -required, then
-pcontrol is forced to use the named host, even if the machine is down,
-unknown, or otherwise unavailable. If the machine is not available,
-pcontrol will simply hold onto the job until the machine is available
-or the job is deleted. Note that PanTasks may delete jobs from pcontrol
-if they remain pending for too long (see period -timeout).
+(forked in the background on the same machine as PanTasks) or run on
+the IPP parallel process controller, pcontrol. The default is for the
+job to be run locally. If a job should be run on the parallel
+controller, this can be specified by including the command host
+(hostname) in the definition of a task. If the value of (hostname) is
+'anyhost', then pcontrol may select any of its host computers to run
+the job according to its own rules. If the value of (hostname) is one
+of the computers managed by pcontrol, then that machine will be
+selected for the job, if it is available. This amounts to a preference
+to use that machine, but pcontrol is allowed to substitute a different
+machine if it chooses. If the host command is given the option
+-required, then pcontrol is forced to use the named host, even if the
+machine is down, unknown, or otherwise unavailable. If the machine is
+not available, pcontrol will simply hold onto the job until the
+machine is available or the job is deleted. Note that PanTasks may
+delete jobs from pcontrol if they remain pending for too long.
 
 It is possible to interact directly with the parallel processor to
@@ -287,5 +196,5 @@
 identified to the controller. If such a host is required, the
 controller will simply keep the associated jobs in the pending state
-until such a machine exists. See the pcontrol documentation for
+until such a machine exists. See the pcontrol section below for
 further discussion of the controller manipuation of jobs and hosts.
 
@@ -373,5 +282,5 @@
 \item external communications
 \item job exit status
-\item job stdout parsing 
+\item job stdout/stderr parsing 
 \end{itemize}
 
@@ -399,4 +308,5 @@
 value of the stdout lines from the UNIX command, and the value
 \code{$var:n} is set to the number of output lines.
+% $
 
 Fine-grained control over the job exit status is available with the
@@ -411,14 +321,82 @@
 status.
 
-Jobs may transmit their results back to PanTasks for further evaluation
-through the standard output and standard error streams. Whenever a job
-exits, the complete stdout and stderr streams from the job are pushed
-onto the PanTasks queues stdout and stderr. The job exit macros may then
-parse these queues, moving the results into other PanTasks / Opihi data
-containers (queues, variables, vectors, whatever is appropriate). Note
-that currently, the output data is simply pushed onto these output
-queues. It is currently the responsibility of the PanTasks programmer to
-use or dispose of the data in these queues. This may change in the
-future: the queues may be flushed for each job completion.
+Jobs may transmit their results back to PanTasks for further
+evaluation through the standard output and standard error
+streams. Whenever a job exits, the complete stdout and stderr streams
+from the job are pushed onto the PanTasks queues \code{stdout} and
+\code{stderr}. The job exit macros may then parse these queues, moving
+the results into other PanTasks / Opihi data containers (queues,
+variables, vectors, whatever is appropriate). Note that currently, the
+output data is simply pushed onto these output queues. It is currently
+the responsibility of the PanTasks programmer to use or dispose of the
+data in these queues. This may change in the future: the queues may be
+flushed for each job completion.
+
+\subsection{PanTasks Monitoring Loop}
+
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.01.ps}
+\caption{\label{MonitorLoop} PanTasks Monitor Loop}
+\end{center}
+\end{figure}
+
+Figure~\ref{MonitorLoop} illustrates the operation of PanTasks.
+Currently, PanTasks is run as a stand-alone foreground process, not a
+server.  In this current operating mode, PanTasks accepts commands
+from the user (left side) via a GNU readline interface.  Readline
+provides a mechanism to schedule an background process which is
+executed on a regular basis, or in the interval after each keystroke.
+This background processing is represented as the loop on the right.
+In this loop, PanTasks checks on the status of pcontrol and on any
+locally spawned background jobs.  It also checks the list of tasks for
+tasks which are ready to be executed.  The tasks and the background
+jobs are kept in rotating queues.  Every time the loop is processed,
+PanTasks runs through a number of the background jobs and tasks,
+attempting to evaluate as many as possible, but stopping after a
+limited number of milliseconds.  This test stage cycles through the
+tasks and/or jobs in their queue, but stops if it examines all of the
+entries in the queue.  If this testing process runs out of time before
+the entire queue is searched, it leaves the sequence intact for the
+next pass.  The timeout is set to keep the keyboard small enough to
+keep the human interaction from being troublesome.
+
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.02.ps}
+\caption{\label{TaskLoop} PanTasks Task Check Loop}
+\end{center}
+\end{figure}
+
+Figure~\ref{TaskLoop} shows the interactions performed for each check
+of the list of tasks.  The task timers are examined to see if the
+specific task is ready to be executed.  If so, then the task exec
+macro is executed.  If this macro exit status is false, the loop goes
+to the next task.  If the task exec macro is true, the job command is
+constructed and the job is submitted to the correct location, either
+the controller (pcontrol) or to the queue of local, background jobs.
+
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.03.ps}
+\caption{\label{JobLoop} PanTasks Job Check Loop}
+\end{center}
+\end{figure}
+
+Figure~\ref{JobLoop} shows the interactions performed for each check
+of a single background job.  The job timer is checked to see if the
+job has timed out.  Next, the job run status is examined to see if the
+job has finished or not.  If the job has finished, the appropriate
+exit macro is executed and the job results are returned to the rest of
+the PanTasks data structures (ie, stderr and stdout queues are filled
+out).  Users should not define exit macros which perform long running
+jobs, or PanTasks will become quite sluggish to keyboard strokes.
+
+PanTasks also checks the current status of pcontrol on each loop.
+In this test, PanTasks requests from pcontrol a list of the jobs which
+have finished.  It then attempts to harvest the finished jobs one by
+one and place the results in the approrpiate locations.  This loop is
+also limited to a fixed amount of time; any pcontrol jobs which remain
+unharvested are left for the next pass by PanTasks.
 
 \subsection{Running the scheduler}
@@ -458,25 +436,4 @@
 \end{verbatim}
 
-\begin{figure}
-\begin{center}
-\includegraphics[scale=0.85]{pics/pantasks.01.ps}
-\caption{\label{queues} PanTasks queues and MDDB tables}
-\end{center}
-\end{figure}
-
-\begin{figure}
-\begin{center}
-\includegraphics[scale=0.85]{pics/pantasks.02.ps}
-\caption{\label{queues} PanTasks queues and MDDB tables}
-\end{center}
-\end{figure}
-
-\begin{figure}
-\begin{center}
-\includegraphics[scale=0.85]{pics/pantasks.03.ps}
-\caption{\label{queues} PanTasks queues and MDDB tables}
-\end{center}
-\end{figure}
-
 \newpage
 \section{pcontrol : the PanTasks parallel controller}
@@ -494,69 +451,83 @@
 \subsection{Host States}
 
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.04.ps}
+\caption{\label{HostStates} PanTasks Host States}
+\end{center}
+\end{figure}
+
 pcontrol maintains a table of available processing computers (hosts)
 and tracks their status. Hosts managed by pcontrol are allowed to be
-in one of several states: off, down, idle, busy, and done. These
-states have the following meanings:
-
-If the host is off, it is known to pcontrol, but pcontrol does not
-have an active connection to the machine. Hosts which are off are not
-available for jobs, and pcontrol does not attempt to initiate a
-connection to them.
+in one of several states: \code{off}, \code{down}, \code{idle},
+\code{busy}, and \code{done} (see Figure~\ref{HostStates}). There are
+also two virtual states: \code{new} and \code{delete}.  These states
+have the following meanings:
+
+If the host is \code{off}, it is known to pcontrol, but pcontrol does
+not have an active connection to the machine. Hosts which are
+\code{off} are not available for jobs, and pcontrol does not attempt to
+initiate a connection to them.  A pcontrol user may force a host to
+transition to the \code{off} state with the command host
+\code{off~(hostname)}. (Note that this command will set only one of
+the connections to the named host to \code{off}. If multiple
+connections to a machine have been defined, multiple \code{off}
+commands must be sent).
 
 When pcontrol is told to consider a machine on, the machine is moved
-from the off state to the down state. Pcontrol attempts to initiate a
-connection to the host. Connections are made by running a remote
-client on the host, using the specified connection method. The
-connection method may be ssh, rsh, or an equivalent remote shell
-connection. The choice is specified by the COMMAND Opihi variable. The
-remote connection starts a dedicated remote client which must accept
-the pcontrol client commands and respond appropriately. The provided
-remote client is called pclient, though in principal other equivalent
-programs could be used by setting the Opihi variable SHELL (this
-feature more generally allows a user to specify a path to the remote
-client, if it is not in the user's path). A pcontrol user may force a
-host to transition to the off state with the command host off
-(hostname). ( Note that this command will set only one of the
-connections to the named host to off. If multiple connections to a
-machine have been defined, multiple off commands must be sent).
+from the \code{off} state to the \code{down} state. Pcontrol attempts
+to initiate a connection to the host. Connections are made by running
+a remote client on the host, using the specified connection
+method. The connection method may be ssh, rsh, or an equivalent remote
+shell connection. The choice is specified by the \code{COMMAND} Opihi
+variable. The remote connection starts a dedicated remote client which
+must accept the pcontrol client commands and respond
+appropriately. The provided remote client is called {\em pclient},
+though in principal other equivalent programs could be used by setting
+the Opihi variable \code{SHELL}.  This feature more generally allows a
+user to specify a path to the remote client, if it is not in the
+user's path.
 
 If the remote connection is successful, the connected host is moved by
-pcontrol from the down state to the idle state. If the connection is
-unsuccessful, pcontrol will try again after a certain period of
-time. If the connection continues to be unsuccessful, the retry period
-is doubled for each successiver connection attempt. If the user wants
-to force pcontrol to retry the connection to a machine (if, for
-example, the timeout is now very long, but the user knows the
+pcontrol from the \code{down} state to the \code{idle} state. If the
+connection is unsuccessful, pcontrol will try again after a certain
+period of time. If the connection continues to be unsuccessful, the
+retry period is doubled for each successiver connection attempt. If
+the user wants to force pcontrol to retry the connection to a machine
+(if, for example, the timeout is now very long, but the user knows the
 machine's ethernet cable has been re-inserted...), this can be
-achieved with the command host retry (hostname). A host which is down
-is in the limbo state between off and idle.
+achieved with the command host \code{retry (hostname)}. A host which
+is \code{down} is in the limbo state between \code{off} and
+\code{idle}.
 
 Once pcontrol has made a successful connection to the host, the host
-is in the idle state. At this point, it is ready to accept jobs from
-pcontrol for execution. Pcontrol repeatedly queries the hosts to check
-that they are still alive. If a host is discovered to be unresponsive,
-and particularly if the remote pipe connection has closed, then the
-machine is moved back to the down state.
-
-Hosts which are idle may accept a job from pcontrol. A job simply
-consists of a bare UNIX command, without redirection of standard input
-or standard output. The host will initiate the job, and pcontrol will
-place the host into the busy state. The remote client, pclient, runs
-the job in the background and will continue to accept input from
-pcontrol. pcontrol will continue to check the status of the host, and
-now also the status of the specific job. As before, if the connection
-breaks, pcontrol will migrate the host to the down state. Any job
-already initiated on a host which goes down will be returned for later
-processing, so the job will not be lost.
+is in the \code{idle} state. At this point, it is ready to accept jobs
+from pcontrol for execution. Pcontrol periodically queries the hosts
+to check that they are still alive. If a host is discovered to be
+unresponsive, and particularly if the remote pipe connection has
+closed, then the machine is moved back to the \code{down} state.
+
+Hosts which are \code{idle} may accept a job from pcontrol. A job
+simply consists of a bare UNIX command, without redirection of
+standard input or standard output. The host will initiate the job, and
+pcontrol will place the host into the \code{busy} state. The remote
+client, pclient, runs the job in the background and will continue to
+accept input from pcontrol. pcontrol will continue to check the status
+of the host, and now also the status of the specific job. As before,
+if the connection breaks, pcontrol will migrate the host to the
+\code{down} state. Any job already initiated on a host which goes down
+will be returned to the pool of unexecuted jobs for later processing,
+so the job will not be lost.
 
 When the job exits, pclient tells pcontrol that the job is completed,
 and specifies the exit status. At this point, pcontrol will move the
-host from busy to done state. It will stay in this state until
-pcontrol can determine the ending conditions and reset the remote
-client. pcontrol requests the standard error and standard output from
-the job from pclient. pcontrol stores this data with its information
-about the completed job, and send a reset command to the remote
-client. Once these cleanup tasks are successfully completed, pcontrol
-will move the host to the idle state, ready for further jobs.
+host from \code{busy} to \code{done} state. It will stay in this state
+until pcontrol can determine the ending conditions and reset the
+remote client. pcontrol requests the standard error and standard
+output from the job from pclient. pcontrol stores this data with its
+information about the completed job, and send a reset command to the
+remote client. Once these cleanup tasks are successfully completed,
+pcontrol will move the host to the \code{idle} state, ready for
+further jobs.
 
 Each physical computer may have multiple processors. pcontrol treats
@@ -574,10 +545,9 @@
 \subsection{Jobs}
 
-The pcontrol accepts new jobs with the command job ..., in which the
-ellipsis represents the command and arguments of a valid UNIX
-command. The commands are run under sh, and are executed in the user's
-home directory. (If it is desired, we can easily add a command to tell
-pclient to perform cd). Users should be wary of the conditions under
-which the remote jobs are run. If the nodes in question all
+The pcontrol accepts new jobs with the command \code{job ...}, in
+which the ellipsis represents the command and arguments of a valid
+UNIX command. The commands are run under \code{sh}, and are executed
+in the user's home directory. Users should be wary of the conditions
+under which the remote jobs are run. If the nodes in question all
 cross-mount the same home directories, multiple jobs which interact
 with the same named file may produce unexpected results. The
@@ -590,5 +560,5 @@
 not realize that specific jobs do not behave the same on all machines,
 or if a necessary resource (eg, some input data file) is only
-available or accessible from some of the hosts. It is the
+available or accessible from some of the hosts. It is also the
 responsibility of the task to wait for network lags (ie, NFS delays).
 
@@ -597,5 +567,6 @@
 pcontrol, the command echoes back the Job ID. This ID may be used by
 other pcontrol commands to obtain information about or interact with
-the job.
+the job.  For example, PanTasks uses the Job ID to examine specific
+jobs.
 
 A job may specify a specific host for the task execution. The host
@@ -634,6 +605,16 @@
 (JobID).
 
-Jobs are moved between the following states by pcontrol:
-
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.05.ps}
+\caption{\label{queues} pcontrol job states.  Transitions labeled Ux
+  are issued by the pcontrol user (including PanTasks).  Transitions
+  labeled Px are initiated by pcontrol.  Transitions labeled Tx are
+  the result of the job completion}
+\end{center}
+\end{figure}
+
+Jobs are moved between the following states by pcontrol (see
+Figure~\ref{JobStates}):
 \begin{itemize}
 \item pending: the job has not yet been executed.
@@ -647,5 +628,5 @@
 
 It is possible to check the status of a single host or job with the
-user command check.
+user command \code{check}.
 
 pcontrol continuously examines the stack of jobs, adjusting their
@@ -658,5 +639,5 @@
 number of microseconds for the timeout.
 
-the pcontrol system status may be examined with the command
+The pcontrol system status may be examined with the command
 status. This provides a dump of the job stacks and the host stacks.
 
@@ -668,24 +649,37 @@
 have exited or crashed.
 
-A specific job may be killed with the command kill (JobID). This
-command is only valid for a job in the busy state. Any job in the
-pending, exit, or crash state may be deleted with the delete (JobID)
-command. This is necessary to free the memory associated with the job
-and its output streams.
-
-The command verbose (mode) turns the verbosity of the pcontrol
+A specific job may be killed with the command \code{kill
+(JobID)}. This command is only valid for a job in the busy state. Any
+job in the pending, exit, or crash state may be deleted with the
+\code{delete (JobID)} command. This is necessary to free the memory
+associated with the job and its output streams.  PanTasks
+automatically performs these job harvesting functions.
+
+The command \code{verbose (mode)} turns the verbosity of the pcontrol
 operations on or off.
 
-The pcontrol and the IPP Image Server have related needs for
-information from the combined storage-and-processing nodes regarding
-which nodes are available. It is not yet clear if this information is
-best stored in a single location (either pcontrol or IPP Image
-Server), which provides the information to other systems on demand, or
-if both systems should maintain the information. Also, it may be
-necessary to distinguish nodes which are available for processing from
-those that are available to serve data as part of the IPP Image
-Server.
-
-\subsection{Command Summary}
+The pcontrol and Nebulous have related needs for information from the
+combined storage-and-processing nodes regarding which nodes are
+available. Currently, the two systems independently examine the
+hardware to judge the availability.  It is not yet clear if this
+information is best stored in a single location (either pcontrol or
+Nebulous), which provides the information to other systems on demand.
+The current implementation allows the IPP system as a whole to
+distinguih nodes which are available for processing from those that
+are available to serve data as part of Nebulous.
+
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.06.ps}
+\caption{\label{PControlLoop} PControl Job Monitor Loop}
+\end{center}
+\end{figure}
+
+Figure~\ref{PControlLoop} illustrated the pcontrol job monitor loop.
+This is very similar to the loop in PanTasks.  A background process
+launched by readline during idle periods cycles through the list of
+hosts (children) and migrates jobs to and from them as needed.
+
+\subsection{pcontrol Command Summary}
 
 \begin{verbatim}
@@ -703,25 +697,4 @@
 \end{verbatim}
 
-\begin{figure}
-\begin{center}
-\includegraphics[scale=0.85]{pics/pantasks.04.ps}
-\caption{\label{queues} PanTasks queues and MDDB tables}
-\end{center}
-\end{figure}
-
-\begin{figure}
-\begin{center}
-\includegraphics[scale=0.85]{pics/pantasks.05.ps}
-\caption{\label{queues} PanTasks queues and MDDB tables}
-\end{center}
-\end{figure}
-
-\begin{figure}
-\begin{center}
-\includegraphics[scale=0.85]{pics/pantasks.06.ps}
-\caption{\label{queues} PanTasks queues and MDDB tables}
-\end{center}
-\end{figure}
-
 \newpage
 \section{pclient}
@@ -739,5 +712,6 @@
 recommended practice is to set up ssh to allow the connection to the
 remote host without additional authentication using the appropriate
-authorized keys (see this article on ssh issues).
+authorized keys.  The security is maintained by the security of ssh
+logins to the computers used by PanTask and pcontrol.  
 
 It is convenient to keep a continuous connection to the remote
@@ -768,5 +742,5 @@
 \begin{figure}
 \begin{center}
-\includegraphics[scale=0.85]{pics/pantasks.07.ps}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.07.ps}
 \caption{\label{queues} PanTasks queues and MDDB tables}
 \end{center}