
Pantasks interacts with a parallel job manager.  The current version
is 'pcontrol', but we would like to allow interactions with condor as
well.  This file documents the pantasks / controller interactions and
the requirements for implementing a condor (or other) interface.

Some terminology:

* controller : the abstract concept of the software or system which
  manages parallel jobs for pantasks.

* pcontrol : the default (Ohana-native) controller implementation

Pantasks Controller Commands.  

The following commands are available within the pantasks shell to send
commands to the controller.  Within the pantasks shell, these are
invoked with "controller (command) [options]".  When running the
pcontrol shell on its own, these commands are called directly.

* check job (jobID)
* check host (hostID)

check on the status of a single job or host.  the jobID or hostID is
an integer value [1].  The return is a block of information giving the
status and some other infomation.  

For 'host', the return is of the form:

host (state)
STATUS 1

where (state) is one of IDLE, BUSY, RESP, DONE, DOWN, OFF.

For 'job', the return is of the form:

STATUS (status)
EXITST (Nexit)
STDOUT (Nbytes)
STDERR (Nbytes)
DTIME  (elapsed time)

where:

(status) is one of PENDING, BUSY, DONE, EXIT, CRASH
(Nexit) is the exit status of the command (ie, as if it were run on
	the UNIX command line).
(Nbytes) is the size of the standard out and standard error buffers
	 from the job
(elapsed time) is the number of seconds it took to run the command;
	 this is only set on exit.

* exit

Tell the controller to exit.  In pantasks, it is necessary to give this command in
the form 'controller exit TRUE'.

* host [options] : manipulate hosts managed by the controller.  The
following commands may be given:

  * host add (hostname) [-threads N] : add a new connection to the host
    (hostname).  The optional -threads N argument specifies the default
    value for this machine for the @MAX_THREADS@ directive.  A job sent
    to this machine with @MAX_THREADS@ in the command line will have
    that value replaced by "-threads N" for this machine, were N is the
    value specified in this host add command.
  
  * host on (hostname) : tell the controller to activate the specified
    host
  
  * host off (hostname) : tell the contoller to de-activate the
    specified host
  
  * host check (hostname) : check the status of the given host by
    hostname (returns 'host (hostname) is (status)' where (status) is
    one of the list given above for check host
  
  * host retry (hostname) : tell the controller to re-attempt a
    connection to the specified machine NOW (if the connection failed,
    pcontrol attempts to connect with an increasingly long timeout.  If
    the timeout is long, but the user knows the machine is now alive,
    they may desire to force a connection attempt sooner rather than
    waiting for the timeout to complete).
  
  * host delete (hostname) : remove the named host from the list of
    managed hosts.

Note that the controller manages connections to host machines by
name. Multiple connections are not generally tied together -- as far
as pantasks normally is aware, they are not related.  Thus, if a given
session has N connections to a given machine (host add was called N
times), then manipulation of the state of the machine may require N
calls of the same function.  (Note that commands which change a
machine state, such as 'on', 'off', 'delete' only affect machines in
the appropriate state.  eg, 'host on ipp050' is invalid if ipp050 is not
currently off.

* hoststack (stack) : list the hosts in the given 'stack'.  A
  collection of hosts in a given state is called a 'stack' of hosts.
  This command lists all of the hosts in one of the stacks.  Stack
  names are case-insensitive and may be one of the following:

  * IDLE : machines which are currently unoccupied with processing
  * BUSY : machines which are currently active
  * RESP : machines which are currently active and responding to
           another command
  * DONE : machines which have completed a job, but are not yet ready
    	   to accept a new job.
  * DOWN : machines which are currently unresponsive (pcontrol will
    	   try to reconnect after an interval)
  * OFF  : machines which are currently off (pcontrol will not try to reconnect)

The response to this command is a list of the machines (a series of
lines each with ID NAME on a single line).

* jobstack (stack) : list the jobs in the given 'stack'.  Like the
  hosts, a collection of jobs in a given state is called a 'stack' of
  jobs.  This command lists all of the jobs in one of the stacks.
  Stack names are case-insensitive and may be one of the following

  * PENDING : job is waiting for a host
  * BUSY    : job is running on a host
  * RESP    : job is running on a host, and responding to another command
  * DONE    : job has finished, but its completion state has not yet
    	      been assessed
  * EXIT    : job finished with a valid exit status (ie, no abort or segfault)
  * CRASH   : job aborted or segfaulted
  * KILL    : kill has been requested for the job

* machines : list the status of the hosts by unique machine name.
  This command is one of the few which interact with a single 


Note that 


[1] : note that the command 'controller status' includes the job and
host ID values in the form X.X.X.X where X is a hexidecimal number.
Unfortunately, pcontrol does not understand this format for the IDs
for the 'check' command.   
