
---

sched / pcontrol todo:

- sched: validate task hosts with controller
- pcontrol: host (name) -check should return state
- sched: start / stop controller connection?
- pcontrol: add command to check status of a job
- pcontrol: add command to dump stdout/stderr for a job

---

scheduler commands:

task (taskname)
 - define a new task
 - loads task-related commands from list / readline
 - commands are parsed on load
 - end with end (like if / for)

task.exit (value)
 - define a new task macro for this exit condition
   (value) may be an exit status (number)
   (value) may be 'timeout'
   (value) may be 'crash' ?
 - commands are parsed on execution (not on definition)

task.exec
 - define a task macro for exec condition
 - commands are parsed on execution (not on definition)

command (args) (args)
 - defines command associated with task
 - may be in task or in task.macro (exit/exec)
   (in task, command line is static; in task.macro, command line is expanded for each instance)

host (machine) [-required]
 - defines preferred host
 - may be in task or in task.macro (exit/exec)
   (in task, value is static; in task.macro, value is defined for each instance)

stderr (file / variable)
 - defines destination for stderr capture from task
 - written to destination at end of execution?

stdout (file / variable)
 - defines destination for stdout capture from task
 - written to destination at end of execution?

periods -poll 1
periods -exec 30
periods -timeout 2
 - defines relevant time-scale for the task

schedule
 - runs the scheduler loop, executing the various tasks

scheduler functions:

InitTasks ();
FindTask ();
CreateTask ();

---

local jobs vs controller jobs

if 'host' is 'local', task is run as background fork (ie, not on controller)
if 'host' is NULL or 'any', task is run on controller without specified host
if 'host' is (machine), task is run on controller with specified host

---

scheduler / controller / local interactions

scheduler can execute local jobs and jobs on the controller.
scheduler needs to initiate the connection to the controller, if the
controller is being used.  it also needs to fork each local command.

we need a command to define the controller hosts to be used.  if a
task or job defines a host which is not known, how is this caught?
does sched need to keep a list of valid hosts, or should the host be
passed down to the controller.  Also, the scheduler should not
initiate a connection to the controller unless it is requested.  is
this implicit when defining a host?  

- controller host foobar
 
  this should setup the controller interface (if it does not currently
  exist), and send the command 'host foobar'.  scheduler should not
  care if the connection is successful or not (we can know about hosts
  which are currently off).  


---
controller interaction:

controller accepts the following commands:

job [options] argv0 argv1 argv2 ...
  -host name : run job on specified host, or any other if not available
  +host name : run job on specified host, error if not available
  -timeout N : seconds before controller gives up on job (once started)
  -stdout name : redirect job stdout to file directly
  -stderr name : redirect job stderr to file directly
  * priority information?
  * returns JobID

status (JobID)

stdout (JobID)

stderr (JobID)

stop (JobID)
  
delete (JobID)

---

possible errors which the scheduler may encounter when executing a
job:

  - controller is not responding
  - controller says machine is DOWN
  - controller says command not found
  - controller has too many processes
  - controller takes to long to start job (scheduler timeout)
  - controller says job timed out
  - controller says job crashed
  - controller says job exited with status

---

notes:

 - watch for NFS lags / blocking.  if NFS has file visibility lags, we
   may need to add blocking as an option to the job (-block filename)

