Changeset 6055 for trunk/doc/design/ippSSDD.tex

Timestamp:

Jan 19, 2006, 12:58:19 AM (21 years ago)

Author:

eugene

Message:

updates

File:

• trunk/doc/design/ippSSDD.tex (modified) (6 diffs)

trunk/doc/design/ippSSDD.tex

@@ -1 @@
-%%% $Id: ippSSDD.tex,v 1.5 2006-01-19 06:49:50 eugene Exp $
+%%% $Id: ippSSDD.tex,v 1.6 2006-01-19 10:58:19 eugene Exp $
 \documentclass[panstarrs]{panstarrs}
 
@@ -779 +779 @@
 \subsection{AP Database}
 
+\tbd{this section needs to be updated with current implemetation; the
+  DVO SDD contains much of this information, but needs to be fleshed
+  out in places.}
+
 \subsubsection{Corresponding Requirements}
 
@@ -1199 +1203 @@
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-\subsection{Controller}
-\label{sec:Controller}
-
-\subsubsection{Corresponding Requirements}
-
-The Controller must meet the requirements specified in Section 3.4.4
-of the Pan-STARRS PS-1 IPP SRS (PSDC-430-005).  The design must meet
-requirements 3.4.4.1 - 3.4.4.7.  In particular, the Controller / Node
-Agent architecture is chosen to control the I/O flow between the
-Controller and the individual processes so that blocking on the I/O
-from many remote processes does not saturate the Controller
-processing.
-
-\subsubsection{Overview}
+\subsection{PanTasks : the IPP Scheduler}
+
+PanTasks is the IPP tool which manages the sequencing of data analysis
+steps and, with the related tool `PControl', distributes the data
+processing across a cluster of computers. \tbd{describe the 'opihi'
+shell-scripting system}
+
+The purpose of PanTasks is to manage the automatic construction and
+execution of inter-related (often repetative) operations.  PanTasks
+uses a set of rules to define UNIX commands, and their corresponding
+command-line arguments, to be performed on some regular, repeated
+basis.  The utility of PanTasks is that it can easily define and
+manage an analysis system which is completely state-based, as opposed
+to an event-driven system.
+
+Consider, for example, a telescope which obtains a collection of
+images over the course of a night. Every minute or two, it takes an
+image and writes the image to some disk. An event-driven analysis
+system would involve having the telescope initiate a process at the
+end of the exposure. This process would perform an analysis, write
+some output, then send trigger another process. This type of operation
+works very well for a simple set up with reliable hardware. Such a
+system becomes more difficult to maintain when hardware failures occur
+or when multiple systems need to interact with each other. When
+failures occur, the triggering information (the events) is easily
+lost, thus some mechanisms are needed to detect these failures and
+either re-send the trigger or send an alternative failure-mode
+trigger. Or, if two systems need to interact, one or the other system
+must block for results from the first. Stopping and restarting such an
+analysis system is very delicate since the appropriate triggers must
+be set up some how, eg by noticing which images have not succeeded and
+restarting them at the appropriate stage. All of these types of
+methods of handling complexity and failures are essentially
+state-based rules. PanTasks allows the easy definition of a totally
+state-based analysis system.
+
+In a state-based system, some mechanism examines the state of the
+system and decides which actions to perform based on the current
+state. In the illustration above, the mechanism could examine the
+images available (either by examining the disk or by examining the
+state of a data table) and decide to perform an operation based on
+what images are available. This makes it very easy to handle
+complexity and errors. If an analysis fails, the state either is not
+successfully updated or the error state is recorded, both situations
+being easy to detect and easy to handle. Restarting the system simply
+involves starting the state-monitoring mechanism. Combining results
+from multiple input sources simply involves watching for the multiple
+inputs to be available. PanTasks provides a mechanism to define state
+monitors, and to define the actions which are performed when those
+states occur. PanTasks action consist of initiating UNIX commands, where
+the arguments of those commands may depend on the results of the state
+tests.  
+
+\subsubsection{Tasks vs Jobs}
+
+The two basic units of PanTasks operation are the 'task' and the
+'job'.  A 'job' is simply a command the user would execute on a
+command line; it consists of a command along with optional command
+line arguments.  A 'task' is a generic description of a type of job in
+which the details of any specific piece of data are omitted.  The task
+defines the UNIX command which corresponds to the job, and it provides
+rules for determining the identity of data for the job.  The task also
+defines tests which are used to decide if the job may be executed.
+Finally, it defines a polling frequency in which PanTasks should
+attempt to construct a new job for the task.
+
+For example, we may want to regularly copy files from one location to
+another.  Perhaps the name of the next available file is available in
+a database table, and can be retrieved with the command 'nextFile'.
+Perhaps we wish to check for new files every 60 seconds.  Thus, the
+task is to copy files, while a specific job of this task might be of
+the form \code{cp newfile newpath}.  With PanTasks, we would define a
+copy task somewhat like the following:
+
+\begin{verbatim}
+  task copyfile
+    periods -exec 60.0
+
+    task.exec
+      $file = `nextFile`
+      if ($file == "none")
+        break
+      end
+      command cp $file $newpath
+    end
+
+    task.exit 0
+      queuepush copied $file
+    end
+
+    task.exit 1
+      queuepush failure $file
+    end
+  end
+\end{verbatim}
+
+In this simple example, the task is attempted every 60 seconds.  If
+there is no new file (output of 'nextFile' is 'none'), then no job
+results from this task.  If there is a new file, the copy command is
+performed.  This example is deceptively simple because one could
+easily imagine writing a stand-along program which performs the same
+thing.  The important advantages of using PanTasks for this type of
+operation are: 
+\begin{itemize}
+\item the output from one job can be used to spawn a number of other tasks.
+\item the success or failure state of each job can be used to spawn
+  other tasks.
+\item the details of the job and data test are kept separated from the
+  rules which connect tasks together.  
+\item the relationships between tasks are kept together in a single
+  location.
+\end{itemize}
+
+In addition to these organizational advantages, PanTasks also provides
+a direct connection to the tool for monitoring parallel jobs,
+pcontrol.  Thus the jobs spawned by PanTasks can be defined to run in
+the background locally or on any of the computers in the parallel
+processing cluster.
+
+\subsubsection{Parallel vs Local Job Processing}
+
+Jobs which are generated by PanTasks tasks may either be run locally
+(forked in the background on the same machine as PanTasks) or run on
+the IPP parallel process controller, pcontrol. The default is for the
+job to be run locally. If a job should be run on the parallel
+controller, this can be specified by including the command host
+(hostname) in the definition of a task. If the value of (hostname) is
+'anyhost', then pcontrol may select any of its host computers to run
+the job according to its own rules. If the value of (hostname) is one
+of the computers managed by pcontrol, then that machine will be
+selected for the job, if it is available. This amounts to a preference
+to use that machine, but pcontrol is allowed to substitute a different
+machine if it chooses. If the host command is given the option
+-required, then pcontrol is forced to use the named host, even if the
+machine is down, unknown, or otherwise unavailable. If the machine is
+not available, pcontrol will simply hold onto the job until the
+machine is available or the job is deleted. Note that PanTasks may
+delete jobs from pcontrol if they remain pending for too long.
+
+It is possible to interact directly with the parallel processor to
+examine the current status, halt the parallel processor, etc. Commands
+to the parallel processor are defined under the controller
+command. The following controller commands are available:
+
+\begin{itemize}
+
+\item controller host (command) (hostname): Manage the parallel
+  controller collection of hosts. This command can be used to add a
+  new host, the delete one of the existing hosts, to turn a host on or
+  off, and to check the status of a host
+
+  \begin{itemize}
+  \item controller host add (hostname): add a new host.
+
+  \item controller host delete (hostname): delete a host.
+
+  \item controller host on (hostname): tell pcontrol that the host is on.
+
+  \item controller host off (hostname): tell pcontrol that the host is off.
+
+  \item controller host retry (hostname): tell pcontrol to retry the host connection.
+
+  \item controller host check (hostname): check the current status of a host. 
+  \end{itemize}
+
+  \item controller exit: stop controller execution.
+
+  \item controller status: report controller current status.
+
+  \item controller check: check job or host status.
+
+  \item controller output: print accumulated messages from the controller. 
+\end{itemize}
+
+It is also possible to specify a host for a task which has not been
+identified to the controller. If such a host is required, the
+controller will simply keep the associated jobs in the pending state
+until such a machine exists. See the pcontrol section below for
+further discussion of the controller manipuation of jobs and hosts.
+
+\subsubsection{Task Restrictions}
+
+Tasks may have restrictions on when they create jobs and how
+frequently they create jobs. The task command trange is used to
+specify a valid or invalid time range for a task. A valid time range
+limits the task evaluation to that time period. An invalid time range
+excludes task evaluation from the time period. Any number of time
+range restrictions may be defined, and the union of all restrictions
+will define if a job may be created. By default, the time range is an
+inclusive time range: the task is evaluated only if the current time
+falls within the specified time range. Alternatively, if the -exclude
+flag is given, the time range is exclusive, in which case the task is
+not evaluated if the current time falls within this range.
+
+The time range may be given as a range of absolute dates as follows:
+
+\begin{verbatim}
+ trange YYYY/MM/DD,HH:MM:SS YYYY/MM/DD,HH:MM:SS 
+\end{verbatim}
+
+where the two dates specify the start and end of the time range. In
+either of these date representations, the least-significant elements
+of the date and time may be dropped, defaulting to 00 (in the case of
+hours, minutes, and seconds) or 01 (in the case of day and
+months). Rather than specifying an end date, it is also valid to
+specify a time interval from the starting date. The time interval is
+specified as a number followed by a unit indicated by a single letter:
+d (days), h (hours), m (minutes), s (seconds).
+
+The time range may also be specified as a repeated period of time,
+either as a time of day or a day and time of week. In the first case,
+the time range is specified as follows:
+
+ 
+\begin{verbatim}
+ trange HH:MM:SS HH:MM:SS
+ \end{verbatim}
+
+
+where again the least-significant elements may be dropped and default
+to 00. This type of restriction defines a time range which is valid
+every day. The alternative is to specify a time range within the week,
+in the following form:
+
+\begin{verbatim}
+ trange DAY@HH:MM:SS DAY@HH:MM:SS
+\end{verbatim}
+
+where the value of DAY may take on any of the three letter day-of-week
+names (Sun, Mon, Tue, etc). This restriction specifies a start and end
+time within a week which is evaluated for each week.
+
+Below are several examples of valid time range restrictions
+
+\begin{verbatim}
+ trange 2005/01/01 2005/12/31   (only run during 2005!)
+ trange 18:00 00:00             (only run from 6pm until midnight)
+ trange 00:00 06:00             (only run from midnight until 6am)
+ trange Mon@08:00 Fri@17:00     (only run between Mon morning and Fri afternoon)
+ trange -exclude 12:00 13:00    (skip 1 hour from noon)
+\end{verbatim}
+
+Note that the current definition of trange does not include time zone
+information. This means that all times are relative to UT. This should
+be addressed by adding a timezone environment variable to PanTasks and
+by allowing the trange to define a timezone offset.
+
+It is also possible to restrict the total number of jobs which are
+spawned for a given task. This is done with the nmax command, which is
+given as part of the task definition. Once a task has constructed nmax
+jobs, it stops task evaluation. It is possible to redefine the value
+of nmax at any time by redefining the task. Any time the task is
+redefined, the new values for any task concept will override the
+existing values for the task concept.
+
+\subsubsection{Inter-Task and Inter-Job Communications}
+
+There are several ways in which the results of jobs may be used to
+influence other jobs. These include:
+
+\begin{itemize}
+\item external communications
+\item job exit status
+\item job stdout/stderr parsing 
+\end{itemize}
+
+It is always possible for the interprocess communication to be
+performed externally: all jobs may simply write results to an external
+data source which is queried as part of the task evaluation. PanTasks
+may interact with UNIX programs using Opihi system interaction
+functions. These interaction methods include: the backticks for
+setting Opihi variables:
+
+\begin{verbatim}
+ $variable = `UNIX Command`
+\end{verbatim}
+
+The exec command (which executes a UNIX command) and the backticks
+both receive the UNIX command exit status, setting the variable
+\code{$STATUS}. It is also possible to set a variable list to the
+output of a UNIX command:
+
+\begin{verbatim}
+ list var -x "UNIX Command"
+\end{verbatim}
+
+In this last case, the values \code{$var:0 - $var:N-1} are set to the
+value of the stdout lines from the UNIX command, and the value
+\code{$var:n} is set to the number of output lines.
+% $
+
+Fine-grained control over the job exit status is available with the
+task.exit macro command. This allows a task to define an exit macro
+which is performed for different exit status conditions. The argument
+to the task.exit command is the exit status value which triggers the
+macro. This may consist of any valid numeric exit status value
+(0-255). It may also have the value crash, in which case the macro is
+executed if the program exited as a result of a signal (ie,
+segmentation fault, etc). Finally, if may have the value default, in
+which case, the macro is run if no other macro describes the exit
+status.
+
+Jobs may transmit their results back to PanTasks for further
+evaluation through the standard output and standard error
+streams. Whenever a job exits, the complete stdout and stderr streams
+from the job are pushed onto the PanTasks queues \code{stdout} and
+\code{stderr}. The job exit macros may then parse these queues, moving
+the results into other PanTasks / Opihi data containers (queues,
+variables, vectors, whatever is appropriate). Note that currently, the
+output data is simply pushed onto these output queues. It is currently
+the responsibility of the PanTasks programmer to use or dispose of the
+data in these queues. This may change in the future: the queues may be
+flushed for each job completion.
+
+\subsubsection{PanTasks Monitoring Loop}
 
 \begin{figure}
 \begin{center}
-\resizebox{4.5in}{!}{\includegraphics{pics/Controller}}
-\caption{Schematic illustration of the Controller components}
-\label{fig:Controller}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.01.ps}
+\caption{\label{MonitorLoop} PanTasks Monitor Loop}
 \end{center}
 \end{figure}
 
+Figure~\ref{MonitorLoop} illustrates the operation of PanTasks.
+Currently, PanTasks is run as a stand-alone foreground process, not a
+server.  In this current operating mode, PanTasks accepts commands
+from the user (left side) via a GNU readline interface.  Readline
+provides a mechanism to schedule an background process which is
+executed on a regular basis, or in the interval after each keystroke.
+This background processing is represented as the loop on the right.
+In this loop, PanTasks checks on the status of pcontrol and on any
+locally spawned background jobs.  It also checks the list of tasks for
+tasks which are ready to be executed.  The tasks and the background
+jobs are kept in rotating queues.  Every time the loop is processed,
+PanTasks runs through a number of the background jobs and tasks,
+attempting to evaluate as many as possible, but stopping after a
+limited number of milliseconds.  This test stage cycles through the
+tasks and/or jobs in their queue, but stops if it examines all of the
+entries in the queue.  If this testing process runs out of time before
+the entire queue is searched, it leaves the sequence intact for the
+next pass.  The timeout is set to keep the keyboard small enough to
+keep the human interaction from being troublesome.
+
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.02.ps}
+\caption{\label{TaskLoop} PanTasks Task Check Loop}
+\end{center}
+\end{figure}
+
+Figure~\ref{TaskLoop} shows the interactions performed for each check
+of the list of tasks.  The task timers are examined to see if the
+specific task is ready to be executed.  If so, then the task exec
+macro is executed.  If this macro exit status is false, the loop goes
+to the next task.  If the task exec macro is true, the job command is
+constructed and the job is submitted to the correct location, either
+the controller (pcontrol) or to the queue of local, background jobs.
+
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.03.ps}
+\caption{\label{JobLoop} PanTasks Job Check Loop}
+\end{center}
+\end{figure}
+
+Figure~\ref{JobLoop} shows the interactions performed for each check
+of a single background job.  The job timer is checked to see if the
+job has timed out.  Next, the job run status is examined to see if the
+job has finished or not.  If the job has finished, the appropriate
+exit macro is executed and the job results are returned to the rest of
+the PanTasks data structures (ie, stderr and stdout queues are filled
+out).  Users should not define exit macros which perform long running
+jobs, or PanTasks will become quite sluggish to keyboard strokes.
+
+PanTasks also checks the current status of pcontrol on each loop.
+In this test, PanTasks requests from pcontrol a list of the jobs which
+have finished.  It then attempts to harvest the finished jobs one by
+one and place the results in the approrpiate locations.  This loop is
+also limited to a fixed amount of time; any pcontrol jobs which remain
+unharvested are left for the next pass by PanTasks.
+
+\subsubsection{Running the scheduler}
+
+Once a set of tasks has been defined, the scheduler can be
+started. The scheduler will run in the background, at regular
+intervals examining the collection of tasks and jobs. In these
+periods, the scheduler attempts to construct new jobs and checks on
+the status of jobs which may have finished, either locally or on the
+controller. To start the scheduler, give the command run. To stop the
+scheduler, given the command stop. The current status of the
+scheduler, controller, and any jobs which have been spawned are listed
+with the status command.
+
+It is also possible to kill or delete individual jobs by hand with the
+commands kill (jobID) or delete (jobID).  Other features
+
+\subsection{pcontrol : the PanTasks parallel controller}
+
 The IPP uses a group of computers to store and process images and to
-manipulate collections of detections.  These computers perform any of
-a large number of analysis stages or other processing tasks without
-significant interprocess communication.  It is necessary to have a
+manipulate collections of detections. These computers perform any of a
+large number of analysis stages or other processing tasks without
+significant interprocess communication. It is necessary to have a
 mechanism which initiates computing tasks on the different computers,
 which monitors the tasks as they are executed, which handles the
 output and the errors from these tasks, and which reacts to the
-failure of any of the computing nodes.  The system responsible for the
-tasks in the IPP is the IPP Controller.
-
-The IPP Controller interacts with the collection of computers under
-its management and with other subsystems in the IPP.  The IPP
-Controller receives a variety of inputs from other subsystems,
-described below, and initiates actions such as adding a new process to
-the queue of pending tasks.  The IPP Controller also provides
-information to other subsystems on demand about its processing history
-and current state.  Each physical computer may have multiple
-processors; since the IPP Controller is managing processing tasks, it
-treats each processor independently.  It is up to the system
-configuration if each computer needs to reserve one of its CPUs to
-manage background tasks or if the IPP Controller should attempt to
-send one task per CPU and let the operating system handle the I/O
-load.  The relationship between the different components of the
-Controller is illustrated in Figure~\ref{fig:Controller} and discussed
-below.
-
-\subsubsection{Nodes}
-
-The Controller maintains a table of available processing computers
-(`Nodes') and tracks the status of these Nodes.  Nodes managed by the
-IPP Controller are allowed to be in one of several states, and the IPP
-Controller must interact with it in an appropriate way for each of
-those states.  A Node may be {\tt alive}, {\tt dead} or {\tt off}.
-If the Node is {\tt alive}, it responds to commands from the IPP
-Controller and may be used for tasks subject to other constraints.  If
-it is {\tt dead}, the Node is not responsive and must not be used
-for executing tasks.  The IPP Controller must identify Nodes which
-have died (not responding) and occasionally test them to see if they
-are {\tt alive} again.  Nodes which are {\tt off} are not
-available for tasks and must not be tested.  Nodes may be set to
-the {\tt off} or {\tt dead} states by external subsystems; it is the
-responsibility of the IPP Controller to return a Node to the {\tt
-alive} state if possible.
-
-The IPP Controller must honor requests (normally from the users) to
-change the mode of any computing node on demand between {\tt off} and
-{\tt dead}.  This would normally be done after a Node has been
-rebooted and is released to the IPP Controller for its use.  It must
-also be able to change the list of allowed tasks as requested by
-external commands.
-
-Two example scenarios illustrate the transition between these states,
-and the basic concept of operations for the IPP Controller.  First,
-imagine a computer crashes.  At this point the IPP Controller should
-detect that the Node is no longer responsive and mark it as {\tt
-dead}.  It should occasionally try to re-establish communication with
-the Node, potentially with longer and longer delays between attempts.
-A human could be notified if the Node seems to remain {\tt dead} for a
-very long time.  In another scenario, a person needs to work on a
-Node.  They notify the IPP Controller that the machine is {\tt off},
-perhaps with a prior notification that the machine should be prepared
-to go off.  When work on the machine is complete, it should be placed
-in the {\tt dead} state.  Only when the person is done working and
-testing the machine, and tells the IPP Controller that the machine is
-now {\tt dead} can the IPP Controller attempt to re-start
-communications and re-new processing operations on that Node.
-
-\subsubsection{Node Agents}
-
-When the Controller starts, it attempts to launch a Node Agent on each
-of the available processing Nodes.  Nodes which are not responsive are
-marked as {\tt dead} so they may be re-tried.  A Node Agent runs on
-each of the individual nodes to execute the tasks as directed by the
-Controller.  The Node Agents communicate with the Controller via a
-socket connection.
-
-A Node Agent (which is only running on a Node in the {\tt alive}
-state) may be in one of four modes: {\tt idle}, {\tt busy}, {\tt
-done}, {\tt crash}.  A Node Agent which is {\tt busy} currently has a
-task assigned to it which is executing.  The IPP Controller may only
-assign one task to a Node at a time.  A Node Agent which is in the
-{\tt idle} state may have a task assigned to it.  When the Node Agent
-detects that a tasks has finished, it changes to either the {\tt done}
-or {\tt crash} states depending on the outcome of the process
-execution.  The IPP Controller must also respect a list of task
-restrictions which may require specific tasks to run on specific CPUs
-or exclude specific tasks from specific CPUs.
-
-A task being executed by the Node is run in the UNIX user space as a
-forked process.  The Node Agent must monitor the standard error and
-standard output of the executing task and save them in separate
-buffers.  If the process exits or dies, the Node Agent must detect
-this result and change state appropriately.  The Node Agent must
-respond to various commands from the Controller, as follows:
-
-\paragraph{Report status}
-
-The Node Agent returns its state ({\tt idle}, {\tt busy}, {\tt done},
-{\tt crash}) and the exit status of the current processing task, if
-available.  The reported exit state, if the process has completed
-without crashing, is the UNIX exit state reported by the task: 0--256
-with 0 indicating a successful completion.
-
-\paragraph{Report stdout}
-
-Send and flush the current stdout buffer.  The Node Agent will return
-the complete contents of the stdout buffer via a buffered write and
-flush the buffer when it is finished.  The Node Agent will not accept
-more data on the stdout buffer from the current processing task until
-the send is complete and the buffer is flushed.  The daemon must
-accept all of the buffer output.
-
-\paragraph{Report stderr}
-
-Identical to `report stdout', but for stderr.
-
-\paragraph{Kill task }
-
-The Node Agent should send a kill signal (\code{KILL} or \code{TERM})
-to the current processing task.  When the processing task has exited,
-the Node Agent should set its state to {\tt crash}.
-
-\paragraph{Clear task}
-
-The Node Agent should set its state {\tt idle}.  If a processing stage
-is currently running, it should be killed (\code{KILL} or \code{TERM})
-before the task is cleared.
-
-\paragraph{Start processing stage}
-
-The Node Agent forks a specified command.  The command should be a
-standard UNIX command without command line redirection or
-backgrounding.  The task is run with the same user ID as the Node
-Agent, which is also the same user ID as the Controller.
-
-\subsubsection{Tasks}
-
-The IPP Controller accepts tasks from other IPP subsystems.  The task
-requests include the specific command to be executed and are in the
-form of a UNIX command which could be performed on any of the
-computing nodes.  Any input or output data in the commands must be a
-valid resource regardless of the node on which the task is executed.
-Input and output data resources must be unique where necessary to
-avoid conflicts.  It is the responsibility of the task to wait for
-network lags (ie, NFS delays).  The IPP Controller gives each task a
-unique identifier, which is returned to the requesting entity.  The
-requestor may then use that ID to obtain status information on that
-task or to send control signals to the specific task.
-
-Task requests may specify a desired node for the task execution.  The
-IPP Controller attempts to honor the request if the node is {\tt
-alive}, but will execute it on another node if the requested one is
-{\tt dead} or {\tt off}.  Even if a node is {\tt alive}, the IPP
-Controller will choose another node if the specified task is not
-allowed on the requested node.  In all other cases, the IPP Controller
-waits until the currently executing processes, and processes with
-higher priority, are completed before executing the specified task on
-the requested node.
-
-Task requests may specify an urgency level.  The IPP Controller
-determines the priority of the task on the basis of both the urgency
-and the age of the request.  An executing task must be completed on a
-CPU before any new task is started on that CPU, regardless of
-priority.  The urgency levels range from 0 to 2.  Tasks with an
-urgency of 1 are scheduled whenever they reach the top of the stack.
-Tasks with an urgency of 2 are sent immediately to the top of the
-stack. Tasks assigned a priority of 0 are maintained in the queue and
-never executed.
-
-It may be useful for the Controller to distinguish between tasks
-dominated by I/O and tasks dominated by data processing.  It is
-possible that one of each of these types of tasks may be sent to the
-same node without significantly impacting the system performance.
-Alternatively, it may be necessary to limit a single machine with 2
-CPUs to only one of each of these types of tasks (i.e., one processor
-will be working on I/O while the other is working on processing).
-Such details will be studied by the IfA IPP Team.
-
-The IPP Controller monitors the output streams from the executing
-tasks and the exit status of the tasks.  Each task is associated with
-a log file, to which all output is written.  The status, including the
-exit status, of each task is maintained by the IPP Controller so that
-other subsystems may determine if specific tasks have started or
-completed.
-
-\subsubsection{Controller Interfaces}
-
-The IPP Controller must accept commands from other IPP subsystems.
-These commands include those which govern the processing of specified
-tasks, those which govern the behavior of specific computing nodes,
-and those which request information from the IPP Controller.  The IPP
-Controller must be able to halt the execution of a specified task,
-delete an unexecuted task from the task list, change the priority of
-tasks, and change the requested nodes for tasks.  The IPP Controller
-must also be able to stop the current execution of a task and push it
-to the end of the queue and also change its priority.
-
-The IPP Controller must respond to informational requests regarding the
-collection of machines and their states as well as the collection of
-tasks and their states.  The IPP Controller must monitor the execution
-times of the different tasks and provide summary statistics.  Finally,
-the IPP Controller must respond to three top-level commands: {\tt finish},
-{\tt stop} and {\tt abort}.  When {\tt finish} is requested, no more
-new tasks are accepted on the stack of task, and when all tasks in the
-stack have completed, the IPP Controller must exit.  When {\tt stop} is
-requested, the currently executing tasks must be completed at which
-point the IPP Controller must exit, but tasks remaining in the stack which
-have not been started are flushed.  When {\tt abort} is issued, the
-IPP Controller immediately kills all executing tasks and exits.
-
-The IPP Controller and the IPP Image Server have related needs for
-information from the combined storage-and-processing nodes regarding
-which nodes are available.  It is not yet clear if this information is
-best stored in a single location (either IPP Controller or IPP Image
-Server), which provides the information to other systems on demand, or
-if both systems should maintain the information.  Also, it may be
-necessary to distinguish nodes which are available for processing from
-those that are available to serve data as part of the IPP Image
-Server.
-
-The Controller maintains three tables of processing jobs: pending
-stages, active stages, and completed stages.  The pending stages are
-those which have not yet been performed.  The active stages are those
-currently being performed on one of the remote nodes.  The completed
-stages are those which have finished, either successfully or with an
-error state.  The Controller daemon monitors the collection of remote
-clients and sends them new pending stages when they become free.
-
-The IPP Controller provides a mechanism for users (either other
-programs or humans) to interact with it.  The user interface provides
-commands to check the current processing job queues, the tables of
-successful and failed jobs, to stop or delete jobs, etc.
-
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-
-\subsection{Scheduler}
-\label{sec:scheduler}
-
-\subsubsection{Corresponding Requirements}
-
-The Scheduler must meet the requirements specified in Section 3.4.5 of
-the Pan-STARRS PS-1 IPP SRS (PSDC-430-005).  The design must meet
-requirements 3.4.5.1 - 3.4.5.7.  In particular, the Task / Test
-division is chosen to prevent the Scheduler from blocking while an
-analysis process is performed.  Scheduling requirements will be met by
-defining appropriate Test periods for the different Tasks.
-
-\subsubsection{Overview}
-
-The IPP is responsible for a variety of analysis jobs: processing of
-the science images through several stages; routine assessment of the
-detrend (instrumental calibration) images used in processing the
-science images; construction of replacement detrend images when
-needed; generation of astrometric and photometric reference catalogs
-based on the collected dataset; and the performance of test analysis
-programs.  At any point, decisions need to be made about which of
-these tasks should be performed, based on an analysis of the contents
-of the metadata database, the requirements of the people monitoring
-the IPP, and the near-term observing plans.  The IPP Scheduler is the
-mechanism that assesses these various inputs to guide the decisions
-and initiate the actions.
-
-The IPP Scheduler acts as an interface between several components of
-the IPP and also between the IPP and external agents such as OTIS and
-the users who must monitor the behavior of the IPP.  The IPP Scheduler
-may be viewed as the central brain of the IPP.
-Figure~\ref{fig:Scheduler} illustrates the design of the IPP
-Scheduler.
-
-\subsubsection{Scheduler Tasks and Tests}
-
-The IPP Scheduler performs two types of actions.  'Tasks' are
-long-running programs which are executed by the Controller.  These are
-not only background tasks, but are distributed computing tasks.
-Examples of these include the science analysis tasks (eg, Phase 1, 2,
-3, 4), the Calibration construction tasks, and data copy tasks (such
-as copying images and metadata from the summit system).  'Tests' are
-short-running programs which are used to decide which tasks should be
-run.  Tests should be designed to return immediately ($< 100 ms$) and
-are not run in the background; the Scheduler will block until the test
-is complete.  The IPP Scheduler daemon, which runs continuously,
-performs tests (eg, queries of the IPP Metadata Database, queries of
-OTIS, checks of the IPP hardware status, etc).  Based on these tests,
-the daemon defines appropriate tasks and sends them to the Controller.
-When tasks are completed, their results may be used by the Scheduler
-to update the external systems (update the Metadata Database), or the
-tasks themselves may send their results directly to the Metadata
-Database or other subsystems.  Based on the successful completion (or
-not!) of the tasks, and the new state of entries in the Metadata
-Database, the Scheduler can define new tasks.
+failure of any of the computing nodes. The system responsible for the
+tasks in the IPP is pcontrol.
+
+\subsubsection{Host States}
 
 \begin{figure}
 \begin{center}
-\resizebox{6in}{!}{\includegraphics{pics/Scheduler}}
-\caption{ \label{fig:Scheduler} IPP Scheduler}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.04.ps}
+\caption{\label{HostStates} PanTasks Host States}
 \end{center}
 \end{figure}
 
-The IPP Scheduler sends tasks to the IPP Controller for execution.
-While the IPP Scheduler chooses the tasks to be performed, it is the
-IPP Controller's responsibility to manage the specific tasks executing
-on a given processing node.  This division of responsibilities allows
-the different functionalities of the IPP Scheduler and the IPP
-Controller to be isolated and encapsulated.  With this separation, the
-IPP Controller does not information about the details of the tasks it
-executes, while the IPP Scheduler does not need to monitor the
-computer hardware.
-
-Communication between the IPP Scheduler and the IPP Controller is
-bi-directional; the IPP Scheduler sends tasks to the IPP Controller,
-while the IPP Controller informs the IPP Scheduler of the outcome of
-those tasks.  For the PS-1 IPP, the IPP Scheduler and the IPP
-Controller are distinct, interacting software components.  The
-interface mechanisms are described in Section~\ref{sec:interfaces}.
-
-\subsubsection{Task Rules}
-
-The IPP Scheduler takes as input a collection of rules which define
-the dependency of tasks on certain tests.  The IPP Scheduler must
-choose between several types of analysis tasks based on those rules
-and on results of the tests.  The timescale on which different tasks
-(and their related tests) are executed may vary from 10s of seconds to
-hours, days, or even as long as a week.  The list of tasks which the
-IPP Scheduler must decide between, and the relevant timescale, follow:
+pcontrol maintains a table of available processing computers (hosts)
+and tracks their status. Hosts managed by pcontrol are allowed to be
+in one of several states: \code{off}, \code{down}, \code{idle},
+\code{busy}, and \code{done} (see Figure~\ref{HostStates}). There are
+also two virtual states: \code{new} and \code{delete}.  These states
+have the following meanings:
+
+If the host is \code{off}, it is known to pcontrol, but pcontrol does
+not have an active connection to the machine. Hosts which are
+\code{off} are not available for jobs, and pcontrol does not attempt to
+initiate a connection to them.  A pcontrol user may force a host to
+transition to the \code{off} state with the command host
+\code{off~(hostname)}. (Note that this command will set only one of
+the connections to the named host to \code{off}. If multiple
+connections to a machine have been defined, multiple \code{off}
+commands must be sent).
+
+When pcontrol is told to consider a machine on, the machine is moved
+from the \code{off} state to the \code{down} state. Pcontrol attempts
+to initiate a connection to the host. Connections are made by running
+a remote client on the host, using the specified connection
+method. The connection method may be ssh, rsh, or an equivalent remote
+shell connection. The choice is specified by the \code{COMMAND} Opihi
+variable. The remote connection starts a dedicated remote client which
+must accept the pcontrol client commands and respond
+appropriately. The provided remote client is called {\em pclient},
+though in principal other equivalent programs could be used by setting
+the Opihi variable \code{SHELL}.  This feature more generally allows a
+user to specify a path to the remote client, if it is not in the
+user's path.
+
+If the remote connection is successful, the connected host is moved by
+pcontrol from the \code{down} state to the \code{idle} state. If the
+connection is unsuccessful, pcontrol will try again after a certain
+period of time. If the connection continues to be unsuccessful, the
+retry period is doubled for each successiver connection attempt. If
+the user wants to force pcontrol to retry the connection to a machine
+(if, for example, the timeout is now very long, but the user knows the
+machine's ethernet cable has been re-inserted...), this can be
+achieved with the command host \code{retry (hostname)}. A host which
+is \code{down} is in the limbo state between \code{off} and
+\code{idle}.
+
+Once pcontrol has made a successful connection to the host, the host
+is in the \code{idle} state. At this point, it is ready to accept jobs
+from pcontrol for execution. Pcontrol periodically queries the hosts
+to check that they are still alive. If a host is discovered to be
+unresponsive, and particularly if the remote pipe connection has
+closed, then the machine is moved back to the \code{down} state.
+
+Hosts which are \code{idle} may accept a job from pcontrol. A job
+simply consists of a bare UNIX command, without redirection of
+standard input or standard output. The host will initiate the job, and
+pcontrol will place the host into the \code{busy} state. The remote
+client, pclient, runs the job in the background and will continue to
+accept input from pcontrol. pcontrol will continue to check the status
+of the host, and now also the status of the specific job. As before,
+if the connection breaks, pcontrol will migrate the host to the
+\code{down} state. Any job already initiated on a host which goes down
+will be returned to the pool of unexecuted jobs for later processing,
+so the job will not be lost.
+
+When the job exits, pclient tells pcontrol that the job is completed,
+and specifies the exit status. At this point, pcontrol will move the
+host from \code{busy} to \code{done} state. It will stay in this state
+until pcontrol can determine the ending conditions and reset the
+remote client. pcontrol requests the standard error and standard
+output from the job from pclient. pcontrol stores this data with its
+information about the completed job, and send a reset command to the
+remote client. Once these cleanup tasks are successfully completed,
+pcontrol will move the host to the \code{idle} state, ready for
+further jobs.
+
+Each physical computer may have multiple processors. pcontrol treats
+each processor independently. It is up to the system configuration if
+each computer needs to reserve one of its CPUs to manage background
+tasks or if pcontrol should attempt to send one task per CPU and let
+the operating system handle the I/O load. some of this behavior will
+probably be eventually more intelligent. For example, the commands
+which turn a host on or off should be able to do the same operation to
+all host connections for the same machine name.
+
+A machine may be completely removed from pcontrol's host tables with
+the command host delete (hostname).
+
+\subsubsection{Jobs}
+
+The pcontrol accepts new jobs with the command \code{job ...}, in
+which the ellipsis represents the command and arguments of a valid
+UNIX command. The commands are run under \code{sh}, and are executed
+in the user's home directory. Users should be wary of the conditions
+under which the remote jobs are run. If the nodes in question all
+cross-mount the same home directories, multiple jobs which interact
+with the same named file may produce unexpected results. The
+controller cannot enforce good behavior on the part of the remote
+jobs; it is the responsibility of the user to ensure that conflicts do
+not arise by, eg, always using unique output file names.
+
+Other issues may arise from the fact that pcontrol may be choosing any
+of the hosts to run the job. Typical failures arise if the user does
+not realize that specific jobs do not behave the same on all machines,
+or if a necessary resource (eg, some input data file) is only
+available or accessible from some of the hosts. It is also the
+responsibility of the task to wait for network lags (ie, NFS delays).
+
+pcontrol gives each task a unique internal identifier (Job ID)
+equivalent to the process ID used in UNIX. When a job is submitted to
+pcontrol, the command echoes back the Job ID. This ID may be used by
+other pcontrol commands to obtain information about or interact with
+the job.  For example, PanTasks uses the Job ID to examine specific
+jobs.
+
+A job may specify a specific host for the task execution. The host
+specified for a job may be required, or desired. In the first case,
+pcontrol, will only run the job on the specified host, waiting until
+it is available before attempting the job. In the second case,
+pcontrol will attempt to send the job to the specified host, but if
+the host is unavailable (how long? what conditions?), pcontrol will
+allow the job to be sent to an alternative host. pcontrol attempts to
+honor the requests for required and desired hosts, giving priority
+first to required-host jobs, then to the desired-host jobs, and
+finally to all other jobs. To specify a host for a job, the following
+commands are used:
+
+\begin{verbatim}
+ job -host (command and arguments...)
+ job +host (command and arguments...)
+\end{verbatim}
+
+
+The first case specifies a desired host, while the second specifies a
+required host. It is also possible to specify the special host name
+anyhost, which is equivalent to not specifying a host at all.
+
+Job priority / urgency levels are not implemented at this time.
+
+I/O vs CPU tasks are not currently distinguished by pcontrol
+
+pcontrol stores the stdout and stderr for each completed job. To
+retrieve these data from these streams, the user issues the commands
+stdout (JobID) and stderr (JobID). The result is a single line
+specifying the number of bytes to expect, followed by a dump of the
+buffers, followed by the prompt. It is the user's responsibility to
+relieve pcontrol of this data load by deleting jobs once they are no
+longer needed. Job deletion is performed with the command delete
+(JobID).
+
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.05.ps}
+\caption{\label{queues} pcontrol job states.  Transitions labeled Ux
+  are issued by the pcontrol user (including PanTasks).  Transitions
+  labeled Px are initiated by pcontrol.  Transitions labeled Tx are
+  the result of the job completion}
+\end{center}
+\end{figure}
+
+Jobs are moved between the following states by pcontrol (see
+Figure~\ref{JobStates}):
 \begin{itemize}
-\item moving data from the Summit pixel server ($\sim 30$ second timescales)
-\item running the science analysis stages ($\sim 30$ second timescales)
-\item testing the validity of the current detrend images ($\sim$
-  nightly)
-\item constructing new detrend images ($\sim$ weekly)
+\item pending: the job has not yet been executed.
+\item busy: the job is currently being executed.
+\item done: the job has completed, but the stdout/stderr has not been processed by pcontrol.
+\item exit: the job has completed with a valid exit status
+\item crash: the job has completed with a crash status (exit on signal). 
 \end{itemize}
-The scheduler may be viewed as a complex state machine.  The goal is
-to design the scheduler so that rules may be specified independently
-from the engine which parses the rules to determine which specific jobs
-to send to the controller.
-
-\subsubsection{User Interface}
-
-The IPP Scheduler shall possess a user interface which allows a human
-operator, or other processes, to monitor the current state of the
-Scheduler.  Users have the option to specify that a particular task or
-set of tasks is of higher or lower urgency (as defined in
-Section~\ref{sec:Controller}) than the norm, or to schedule a
-particular tasks on a different timescale from the basic rule.
-
-The IPP Scheduler defines the operating state of the IPP and shares
-the same set of states:
-\begin{itemize}
-\item active state
-\item interactive state
-\item paused state
-\end{itemize}
-When the IPP Scheduler is in the {\em active state}, it performs the
-most appropriate of all possible tasks at a particular time.  When the
-IPP Scheduler is in the {\em interactive state}, it performs only a
-specific requested action regardless of the outcome of the decision
-trees.  In addition, in the interactive state, the IPP Scheduler must
-only perform the requested actions and not attempt to perform the
-other normally-required actions.  The only exception to this exclusion
-is that, in the interactive state, data is still copied from the
-summit system.  An additional IPP state is the {\em paused state},
-intended for tests or maintenance, in which case the IPP Scheduler
-does not perform even the data copy tasks.  Every task is performed on
-demand by the user.  A user command sets the IPP Scheduler in one of
-these three states, {\em active}, {\em interactive}, and {\em paused}.
+
+\subsubsection{Miscellaneous Commands}
+
+It is possible to check the status of a single host or job with the
+user command \code{check}.
+
+pcontrol continuously examines the stack of jobs, adjusting their
+state as needed and extracting their output when it is ready. These
+checks are performed in the background, with pcontrol ready to accept
+further commands from the user in the foreground. These checks are
+performed after every keystroke, and also after an inactivity
+timeout. The interrupt interval defaults to 1 second, but may be
+adjusted with the pulse command, which takes as an argument, the
+number of microseconds for the timeout.
+
+The pcontrol system status may be examined with the command
+status. This provides a dump of the job stacks and the host stacks.
+
+It is possible to list the jobs currently in a specific stack,
+corresponding to the list of jobs with a given state. This is done
+with the command jobstack (stackname). The valid stack names are
+pending, busy, exit, crash, and done. The result is a list of all jobs
+on the specified stack. This is useful to determine quickly which jobs
+have exited or crashed.
+
+A specific job may be killed with the command \code{kill
+(JobID)}. This command is only valid for a job in the busy state. Any
+job in the pending, exit, or crash state may be deleted with the
+\code{delete (JobID)} command. This is necessary to free the memory
+associated with the job and its output streams.  PanTasks
+automatically performs these job harvesting functions.
+
+The command \code{verbose (mode)} turns the verbosity of the pcontrol
+operations on or off.
+
+The pcontrol and Nebulous have related needs for information from the
+combined storage-and-processing nodes regarding which nodes are
+available. Currently, the two systems independently examine the
+hardware to judge the availability.  It is not yet clear if this
+information is best stored in a single location (either pcontrol or
+Nebulous), which provides the information to other systems on demand.
+The current implementation allows the IPP system as a whole to
+distinguih nodes which are available for processing from those that
+are available to serve data as part of Nebulous.
+
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.06.ps}
+\caption{\label{PControlLoop} PControl Job Monitor Loop}
+\end{center}
+\end{figure}
+
+Figure~\ref{PControlLoop} illustrated the pcontrol job monitor loop.
+This is very similar to the loop in PanTasks.  A background process
+launched by readline during idle periods cycles through the list of
+hosts (children) and migrates jobs to and from them as needed.
+
+\subsubsection{pclient}
+
+pclient is the remote process monitor for pcontrol, the parallel
+process controller.
+
+The program pclient is used to support the remote jobs which are run
+on the remote hosts by pcontrol. The concept of pclient is to act as a
+buffer between the job running on the remote host and pcontrol. The
+pcontrol design uses (by default) ssh connections initiated by
+pcontrol to the remote hosts. These connections execute the remote
+program of pclient. The use of a remote login process lets the UNIX
+system take care of the user authentication issues. In this case, the
+recommended practice is to set up ssh to allow the connection to the
+remote host without additional authentication using the appropriate
+authorized keys.  The security is maintained by the security of ssh
+logins to the computers used by PanTask and pcontrol.  
+
+It is convenient to keep a continuous connection to the remote
+hosts. This avoids incurring the overhead of authentication for each
+command which is executed, while keeping a high-quality user
+authentication process in place.
+
+pclient acts as a buffer between pcontrol and the remote background
+process, allowing the continuous connection to remain viable without
+samping pcontrol with output from the jobs.  
+
+\begin{figure}
+\begin{center}
+\includegraphics[scale=0.85,angle=-90]{pics/pantasks.07.ps}
+\caption{\label{queues} PanTasks queues and MDDB tables}
+\end{center}
+\end{figure}
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
@@ -2507 +2795 @@
 object analysis in the other analysis stages.
 
-\subsection{AstroRef: Astrometric Reference Catalog creation}
-
-\tbd{needs to be fleshed out substantially}
-
-This processing stage shall use many observations over a given time
-period to fit a consistent global astrometric solution, resulting in a
-high quality and internally-consistent astrometric catalog that may be
-published.
-
-\subsection{PhotoRef: Photometric Reference Catalog creation}
-
-\tbd{needs to be fleshed out substantially}
-
-This processing stage shall use many observations over a given time
-period to fit a consistent global photometric solution, resulting in a
-high quality and internally-consistent photometric catalog that may be
-published.
+\section{IPPtools}
+
+Above, we discussed PanTasks, the IPP scheduler which determines the
+new jobs to run and distributes them to computers across the network.
+PanTasks is a general tool; by it self it does not define the specific
+analysis tasks that the IPP requires.  The previous few sections
+discussed in detail the analysis which is performed by the IPP
+analysis stages.  IPPtools is the collection of PanTasks scripts,
+Metadata Database interaction programs, and other tools used to
+definet the specific analysis stages of the IPP. 
+
+\tbd{this section needs to be fleshed out with a summary of the
+  ippTools functions.  The stand-alone IPPTools document gives a
+  detailed discussion of these issues}.
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
@@ -3968 +4253 @@
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
+\section{PanTasks Example : Pcopy.pro}
+
+Below is an example script for psched which demonstrates the
+scheduling system. This parallel-copying script implements the
+Pan-STARRS image copying system, which requests images from the summit
+and copies them to the appropriate computer. The first task in the
+script queries an external system for new image names with the
+function new.images. In the case of Pan-STARRS, this would be a
+request from OTIS, the observatory controlling system. The second task
+initiates the individual image copies, with separate CCDs being copied
+to separate computers. This script uses the concept of having specific
+machines assigned to specific CCDs (as Pan-STARRS intends to
+operate). The association is determined by calling the external
+function chip.host, providing the identifier of the chip in
+question. This returns an appropriate host. The copy.image function
+copies the file and also sends a message to the summit system to
+inform it that the image has been successfully copied.
+
+\begin{verbatim}
+verbose on
+ queueinit newImages
+ exec echo 0 > new.last
+ exec cp -f raw.list new.list
+ 
+ controller host add po01
+ controller host add po02
+ controller host add po03
+ controller host add po04
+ 
+ # identify the images ready for copy 
+ # new entries are added to queue newImages
+ # need to compare the new list with the ones already being processed
+ task          new.images
+   command      new.images
+   host         local
+ 
+   periods      -poll 1
+   periods      -exec 5
+   periods      -timeout 5
+ 
+   # success
+   task.exit    0
+     local i j Nstdout Nimages
+     # compare output with new.image queue
+     # keep only new entries
+     queuesize stdout -var Nstdout
+     for i 0 $Nstdout
+       queuepop stdout -var line
+       queuepush newImages -uniq -key 0 "$line"
+     end
+   end
+ 
+   # locked list
+   task.exit    1
+     echo       "new.images: exec failure"
+     $new.image.failure ++
+   end
+ 
+   # default exit status
+   task.exit    default
+     echo       "new.images: unknown exit status: $EXIT"
+     $new.image.failure ++
+   end
+ 
+   # operation times out?
+   task.exit    timeout
+     echo       "new.images: timeout"
+     $new.image.failure ++
+   end
+ end
+ 
+ # copy new images, sending job to desired host
+ task          copy.images
+   periods      -poll 0.2
+   periods      -exec 1
+   periods      -timeout 5
+ 
+   task.exec
+     queuesize  newImages -var N
+     if ($N == 0) break
+     # if ($network == 0) break
+     # if ($filesystem == 1) break
+     
+     queuepop newImages -var line
+     list tmp -split $line
+     $filename   = $tmp:0
+     $chip       = $tmp:1
+     $state      = $tmp:2
+     if ($state == new) 
+       # copy this image
+       queuepush newImages -replace -key 0 "$filename $chip run"
+     else
+       # ignore this image
+       queuepush newImages -replace -key 0 "$filename $chip $state"
+       break
+     end
+     # echo $chip
+     $host = `chip.host $chip`
+     # echo $host
+     host $host
+     # echo "starting copy for $filename on $host..."
+     command copy.image $filename $chip
+   end
+ 
+   # can I have access to argc,argv?
+ 
+   # success
+   task.exit    0
+     echo "done copy..."
+     queuepop stdout -var line
+     list tmp -split $line
+     $filename   = $tmp:0
+     $chip       = $tmp:1
+     exec mark.image $filename
+     queuepush newImages -replace -key 0 "$filename $chip copy"
+   end
+ 
+   # default exit status
+   task.exit    default
+     echo       "new.images: unknown exit status: $EXIT"
+     $new.image.failure ++
+   end
+ 
+   # operation times out?
+   task.exit    timeout
+     echo       "new.images: timeout"
+     $new.image.failure ++
+   end
+ end
+ 
+\end{verbatim}
+
 \input{glossary.tex}
 
@@ -3976 +4393 @@
 * output data products
 * DVO
-* PanTasks
 * ipptools / ippMonitor?
 * analysis stages, versions and iterations