
scheduler commands:

task (taskname)
 - define a new task
 - loads task-related commands from list / readline
 - commands are parsed on load
 - end with end (like if / for)

task.exit (value)
 - define a new task macro for this exit condition
   (value) may be an exit status (number)
   (value) may be 'timeout'
   (value) may be 'crash' ?
 - commands are parsed on execution (not on definition)

task.exec
 - define a task macro for exec condition
 - commands are parsed on execution (not on definition)

command (args) (args)
 - defines command associated with task
 - may be in task or in task.macro (exit/exec)
   (in task, command line is static; in task.macro, command line is expanded for each instance)

host (machine)
 - defines preferred host
 - may be in task or in task.macro (exit/exec)
   (in task, value is static; in task.macro, value is defined for each instance)

stderr (file / variable)
 - defines destination for stderr capture from task
 - written to destination at end of execution?

stdout (file / variable)
 - defines destination for stdout capture from task
 - written to destination at end of execution?

periods -poll 1
periods -exec 30
periods -timeout 2
 - defines relevant time-scale for the task

schedule
 - runs the scheduler loop, executing the various tasks

scheduler functions:

InitTasks ();
FindTask ();
CreateTask ();

---

local jobs vs controller jobs

if 'host' is 'local', task is run as background fork (ie, not on controller)
if 'host' is NULL or 'any', task is run on controller without specified host
if 'host' is (machine), task is run on controller with specified host

---

controller interaction:

controller accepts the following commands:

job [options] argv0 argv1 argv2 ...
  -host name : run job on specified host, or any other if not available
  +host name : run job on specified host, error if not available
  -timeout N : seconds before controller gives up on job (once started)
  -stdout name : redirect job stdout to file directly
  -stderr name : redirect job stderr to file directly
  * priority information?
  * returns JobID

status (JobID)

stdout (JobID)

stderr (JobID)

stop (JobID)
  
delete (JobID)

---

possible errors which the scheduler may encounter when executing a
job:

  - controller is not responding
  - controller says machine is DOWN
  - controller says command not found
  - controller has too many processes
  - controller takes to long to start job (scheduler timeout)
  - controller says job timed out
  - controller says job crashed
  - controller says job exited with status

---

notes:

 - watch for NFS lags / blocking.  if NFS has file visibility lags, we
   may need to add blocking as an option to the job (-block filename)

