Index: branches/eam_branches/ipp-20120905/Ohana/src/opihi/doc/pantasks.controller.txt
===================================================================
--- branches/eam_branches/ipp-20120905/Ohana/src/opihi/doc/pantasks.controller.txt	(revision 34495)
+++ branches/eam_branches/ipp-20120905/Ohana/src/opihi/doc/pantasks.controller.txt	(revision 34508)
@@ -181,7 +181,93 @@
    readline timeout.
   
-
 [1] : note that the command 'controller status' includes the job and
 host ID values in the form X.X.X.X where X is a hexidecimal number.
 Unfortunately, pcontrol does not understand this format for the IDs
 for the 'check' command.   
+
+--
+
+Pantasks / Controller Interactions
+
+Pantasks Threads & Controller Interactions
+
+Pantasks has a server/client mode and a stand-alone mode.  These share
+the bulk of code, but there are some minor difference.  In stand-alone
+mode, there are 3 active threads.  The main thread interprets the
+commands (accepted by readline); a second thread manages the tasks and
+jobs known to pantasks; the third thread manages interactions with the
+parallel controller.   In server/client mode, a fourth thread manages
+the communcations with the remote clients.  
+
+The task/job thread is responsible for monitoring the task rules and
+constructing the commands which corresponds to jobs when appropriate.
+Those jobs which are defined to be local are executed on the local
+machine, while remote jobs are sent to the thread which interacts with
+the parallel controller.
+
+The primary job of the controller thread is to monitor the status of
+jobs submitted to the controller and to harvest jobs which have
+finished.  A secondary job is to flush the stdout and stderr buffers
+of the pantasks / pcontrol connection.  
+
+Jobs are submitted to the controller directly by the Task/Job thread.
+Other operations, check as manual checks of the job status are
+performed by the main thread.
+
+Commands send to pcontrol:
+
+In addition to the user-level commands discussed above, the following
+messages are sent to the controller:
+
+* jobstack exit : the controller thread checks for the set of completed
+  jobs (which did not crash) by sending this command.  The response is
+  a list of jobs ready for harvest.
+
+* jobstack crash : the controller thread checks for the set of
+  complete jobs which crashed by sending this command.  The response
+  is a list of jobs ready for harvest.
+
+* delete : jobs which have completed and for which the stderr/stdout
+  have been received can be deleted from the controller.  For
+  'pcontrol', this is necessary to free up resources managing the
+  specific job.
+
+* check job : once the controller gets a list of jobs which have
+  exited or crashed, pantasks loops over those jobs, harvesting their
+  results.  this command is used to get the needed stats (size of
+  stderr buffer, size of stdout buffer, exit status, etc).  
+
+* job : Submit a job to the controller.  this command is used by the
+  function SubmitControllerJob to send a new job to the controller.
+  the function appends options such as +host, -nice as needed.  the
+  function expects to receive a job ID from the controller for future
+  interactions.
+
+* quit : shutdown the controller
+
+---
+
+pantasks & condor
+
+There are three classes of pcontrol operations which pantasks
+currently perform:
+
+* submit jobs to pcontrol
+* detect completion and harvest job output
+* manage the hosts used by pcontrol
+
+The communication between pantasks and pcontrol uses a pipe; pcontrol
+is run as a forked child of pantasks.  The communication is done via
+text blocks with fairly minimal hand-shaking.  
+
+If pantasks is integrated with condor, it is critical that it be able
+to perform the first two of these functions.  It is not required that
+pantasks manage the condor hosts.
+
+Condor client / server interaction could be performed via forked
+commands or we could use the SOAP interface.  I suspect that SOAP will
+be more effective for the high-rate job harvesting process.
+
+However jobs are submitted to condor, it is critical that condor be
+able to respect the limit on the number of mistargeted jobs currently
+active.