IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Ignore:
Timestamp:
Jul 5, 2005, 7:12:22 AM (21 years ago)
Author:
eugene
Message:

tested / bugfixed sched/pcontrol I/F

File:
1 edited

Legend:

Unmodified
Added
Removed

  • trunk/Ohana/src/opihi/doc/scheduler.txt

    r4450 r4452  
     1
     22005.07.05
     3
     4At this point, scheduler / pcontrol / pclient all work in a basic way.
     5pclient is the most robust of the three, having the simplest
     6responsibility.  pcontrol is generally pretty good, though I need to
     7flesh out the user interface a bit and clean up the output warning / info
     8messages.  scheduler will need the most attention, though it is
     9already fairly reasonable.  I need to flesh out the user commands to
     10check on the controller status (basically, these need to replicate the
     11status commands available to the controller). 
     12
     13I also need to handle the case of timeout on the controller,
     14independently of timeout for a local job on the scheduler.  currently,
     15if a local job exceeds the timeout value, scheduler flags it.  but, it
     16does not make sense to use the same timeout value for a controller
     17job. I could pass the timeout to the controller when the job executes,
     18in which case it has the same meaning, essentially, for the controller
     19jobs as it does for the local jobs: once you start the function, it
     20needs to complete within NN seconds.  However, I think I still need to
     21have a scheduler concept of a job which the controller is unable to
     22complete.  It should be possible to prevent a job from sitting pending
     23on the controller forever.  What exactly you do if the controller is
     24unwilling to execute a job is another story (possible reasons:
     25controller overload, missing required host, missing any hosts,
     26something hung somewhere?). 
     27
     28The scheduler does not do a good job of shutting down the controller
     29when it (the scheduler) exits.  This works well for the
     30pcontrol/pclient interface, so the solution lies there. 
     31
     32I need to decide how to behave if the scheduler asks for a job with a
     33required host which the controller knows is currently down or
     34non-existent.  Several options could be used.  The controller could
     35simply hold the task until the scheduler notices it is not being
     36executed (after all, the controller does not know if the machine is
     37being serviced for a short time or a long time, but the scheduler
     38could know).  The controller could immediately return a failure noting
     39the current state of the machine (this would put the burden of
     40deciding that the machine should be available on the scheduler).  The
     41controller could try to execute the job a certain number of times, and
     42then it could report the failure to the scheduler.  This is not so
     43different from having a pending-timeout which the scheduler tracks
     44(moves the timeout check to the controller, essentially). 
     45
     46There was some odd behavior with 'exec echo $stdout >> foo'.  This
     47resulted in empty files 'foo'.  The following work fine, so something
     48is just weird:
     49exec echo foobar >> foo
     50output foo
     51echo $stdout
     52output stdout
     53
     54Various error conditions should be checked
     55
     56What do we do if a task requests a host which is not available to the
     57controller (ie, not defined)?  this is similar to the problem of
     58requesting a host which is down.  I think the controller should either
     59immediately refuse or accept in anticipation that such a host may
     60eventually be defined.
     61
     62I need to be careful about jobs sent to the controller and not
     63harvested before stopping the scheduler execution.
    164
    265---
    … …  
    568
    669- sched: validate task hosts with controller
    7 - pcontrol: host (name) -check should return state
    8 - sched: start / stop controller connection?
    9 - pcontrol: add command to check status of a job
    10 - pcontrol: add command to dump stdout/stderr for a job
    1170
    1271---
    … …  
    4099 - may be in task or in task.macro (exit/exec)
    41100   (in task, value is static; in task.macro, value is defined for each instance)
     101 - value of LOCAL runs job as local job (not on controller)
     102 - value of NONE runs job on controller without specifying host
    42103
    43104stderr (file / variable)
    … …  
    54115 - defines relevant time-scale for the task
    55116
    56 schedule
    57  - runs the scheduler loop, executing the various tasks
    58 
    59 scheduler functions:
    60 
    61 InitTasks ();
    62 FindTask ();
    63 CreateTask ();
     117run
     118stop
     119 - start or stop the scheduler loop, executing the various tasks
    64120
    65121---
    … …  
    67123local jobs vs controller jobs
    68124
    69 if 'host' is 'local', task is run as background fork (ie, not on controller)
    70 if 'host' is NULL or 'any', task is run on controller without specified host
    71 if 'host' is (machine), task is run on controller with specified host
    72 
    73 ---
    74 
    75 scheduler / controller / local interactions
    76 
    77 scheduler can execute local jobs and jobs on the controller.
    78 scheduler needs to initiate the connection to the controller, if the
    79 controller is being used.  it also needs to fork each local command.
    80 
    81 we need a command to define the controller hosts to be used.  if a
    82 task or job defines a host which is not known, how is this caught?
    83 does sched need to keep a list of valid hosts, or should the host be
    84 passed down to the controller.  Also, the scheduler should not
    85 initiate a connection to the controller unless it is requested.  is
    86 this implicit when defining a host? 
    87 
    88 - controller host foobar
    89  
    90   this should setup the controller interface (if it does not currently
    91   exist), and send the command 'host foobar'.  scheduler should not
    92   care if the connection is successful or not (we can know about hosts
    93   which are currently off). 
    94 
    95 
    96 ---
    97 controller interaction:
    98 
    99 controller accepts the following commands:
    100 
    101 job [options] argv0 argv1 argv2 ...
    102   -host name : run job on specified host, or any other if not available
    103   +host name : run job on specified host, error if not available
    104   -timeout N : seconds before controller gives up on job (once started)
    105   -stdout name : redirect job stdout to file directly
    106   -stderr name : redirect job stderr to file directly
    107   * priority information?
    108   * returns JobID
    109 
    110 status (JobID)
    111 
    112 stdout (JobID)
    113 
    114 stderr (JobID)
    115 
    116 stop (JobID)
    117  
    118 delete (JobID)
     125a local job is run as background fork (ie, not on controller)
     126a controller job is sent to the controller to run when it can
    119127
    120128---
    … …  
    127135  - controller says command not found
    128136  - controller has too many processes
    129   - controller takes to long to start job (scheduler timeout)
     137  - controller takes to long to start job (pending timeout)
    130138  - controller says job timed out
    131139  - controller says job crashed
    … …  
    138146 - watch for NFS lags / blocking.  if NFS has file visibility lags, we
    139147   may need to add blocking as an option to the job (-block filename)
    140 
Note: See TracChangeset for help on using the changeset viewer.