IPP Software Navigation Tools IPP Links Communication Pan-STARRS Links

Changeset 4452


Ignore:
Timestamp:
Jul 5, 2005, 7:12:22 AM (21 years ago)
Author:
eugene
Message:

tested / bugfixed sched/pcontrol I/F

Location:
trunk/Ohana/src/opihi
Files:
5 edited

Legend:

Unmodified
Added
Removed

  • trunk/Ohana/src/opihi/doc/pcontrol.txt

    r4450 r4452  
    9999  +host name : run job on specified host, error if not available (error when attempted, not when submitted)
    100100  -timeout N : seconds before controller gives up on job (once started)
     101  -stdout name : redirect job stdout to file directly
     102  -stderr name : redirect job stderr to file directly
    101103
    102104  * priority information?
    … …  
    137139currently down, while the transition idle -> down is immediately
    138140followed by an attempt by pcontrol to move the host from down -> idle.
     141This functionality can be used with non-automatic calling of
     142CheckSystem to test the pcontrol host interface operations.
    139143
    140144does it makes sense to kill all jobs on a host?  this would only have
    … …  
    154158---
    155159
    156 write message
    157 read until MESSAGE END
    158 
    159 
    160160Job States:
    161161
    … …  
    164164EXIT
    165165CRASH
    166 NEW
    167 DEL
     166NEW *
     167DEL *
     168
     169* - invisible states
    168170
    169171Job State Transitions:
    … …  
    185187DOWN
    186188OFF
    187 NEW
    188 DEL
     189NEW *
     190DEL *
     191
     192* - invisible states
    189193
    190194Host State Transitions:
    … …  
    214218StartHost - P
    215219CheckHost - P
     220
     221U - operation performed by the user
     222P - operation performed by the program

  • trunk/Ohana/src/opihi/doc/scheduler.txt

    r4450 r4452  
     1
     22005.07.05
     3
     4At this point, scheduler / pcontrol / pclient all work in a basic way.
     5pclient is the most robust of the three, having the simplest
     6responsibility.  pcontrol is generally pretty good, though I need to
     7flesh out the user interface a bit and clean up the output warning / info
     8messages.  scheduler will need the most attention, though it is
     9already fairly reasonable.  I need to flesh out the user commands to
     10check on the controller status (basically, these need to replicate the
     11status commands available to the controller). 
     12
     13I also need to handle the case of timeout on the controller,
     14independently of timeout for a local job on the scheduler.  currently,
     15if a local job exceeds the timeout value, scheduler flags it.  but, it
     16does not make sense to use the same timeout value for a controller
     17job. I could pass the timeout to the controller when the job executes,
     18in which case it has the same meaning, essentially, for the controller
     19jobs as it does for the local jobs: once you start the function, it
     20needs to complete within NN seconds.  However, I think I still need to
     21have a scheduler concept of a job which the controller is unable to
     22complete.  It should be possible to prevent a job from sitting pending
     23on the controller forever.  What exactly you do if the controller is
     24unwilling to execute a job is another story (possible reasons:
     25controller overload, missing required host, missing any hosts,
     26something hung somewhere?). 
     27
     28The scheduler does not do a good job of shutting down the controller
     29when it (the scheduler) exits.  This works well for the
     30pcontrol/pclient interface, so the solution lies there. 
     31
     32I need to decide how to behave if the scheduler asks for a job with a
     33required host which the controller knows is currently down or
     34non-existent.  Several options could be used.  The controller could
     35simply hold the task until the scheduler notices it is not being
     36executed (after all, the controller does not know if the machine is
     37being serviced for a short time or a long time, but the scheduler
     38could know).  The controller could immediately return a failure noting
     39the current state of the machine (this would put the burden of
     40deciding that the machine should be available on the scheduler).  The
     41controller could try to execute the job a certain number of times, and
     42then it could report the failure to the scheduler.  This is not so
     43different from having a pending-timeout which the scheduler tracks
     44(moves the timeout check to the controller, essentially). 
     45
     46There was some odd behavior with 'exec echo $stdout >> foo'.  This
     47resulted in empty files 'foo'.  The following work fine, so something
     48is just weird:
     49exec echo foobar >> foo
     50output foo
     51echo $stdout
     52output stdout
     53
     54Various error conditions should be checked
     55
     56What do we do if a task requests a host which is not available to the
     57controller (ie, not defined)?  this is similar to the problem of
     58requesting a host which is down.  I think the controller should either
     59immediately refuse or accept in anticipation that such a host may
     60eventually be defined.
     61
     62I need to be careful about jobs sent to the controller and not
     63harvested before stopping the scheduler execution.
    164
    265---
    … …  
    568
    669- sched: validate task hosts with controller
    7 - pcontrol: host (name) -check should return state
    8 - sched: start / stop controller connection?
    9 - pcontrol: add command to check status of a job
    10 - pcontrol: add command to dump stdout/stderr for a job
    1170
    1271---
    … …  
    4099 - may be in task or in task.macro (exit/exec)
    41100   (in task, value is static; in task.macro, value is defined for each instance)
     101 - value of LOCAL runs job as local job (not on controller)
     102 - value of NONE runs job on controller without specifying host
    42103
    43104stderr (file / variable)
    … …  
    54115 - defines relevant time-scale for the task
    55116
    56 schedule
    57  - runs the scheduler loop, executing the various tasks
    58 
    59 scheduler functions:
    60 
    61 InitTasks ();
    62 FindTask ();
    63 CreateTask ();
     117run
     118stop
     119 - start or stop the scheduler loop, executing the various tasks
    64120
    65121---
    … …  
    67123local jobs vs controller jobs
    68124
    69 if 'host' is 'local', task is run as background fork (ie, not on controller)
    70 if 'host' is NULL or 'any', task is run on controller without specified host
    71 if 'host' is (machine), task is run on controller with specified host
    72 
    73 ---
    74 
    75 scheduler / controller / local interactions
    76 
    77 scheduler can execute local jobs and jobs on the controller.
    78 scheduler needs to initiate the connection to the controller, if the
    79 controller is being used.  it also needs to fork each local command.
    80 
    81 we need a command to define the controller hosts to be used.  if a
    82 task or job defines a host which is not known, how is this caught?
    83 does sched need to keep a list of valid hosts, or should the host be
    84 passed down to the controller.  Also, the scheduler should not
    85 initiate a connection to the controller unless it is requested.  is
    86 this implicit when defining a host? 
    87 
    88 - controller host foobar
    89  
    90   this should setup the controller interface (if it does not currently
    91   exist), and send the command 'host foobar'.  scheduler should not
    92   care if the connection is successful or not (we can know about hosts
    93   which are currently off). 
    94 
    95 
    96 ---
    97 controller interaction:
    98 
    99 controller accepts the following commands:
    100 
    101 job [options] argv0 argv1 argv2 ...
    102   -host name : run job on specified host, or any other if not available
    103   +host name : run job on specified host, error if not available
    104   -timeout N : seconds before controller gives up on job (once started)
    105   -stdout name : redirect job stdout to file directly
    106   -stderr name : redirect job stderr to file directly
    107   * priority information?
    108   * returns JobID
    109 
    110 status (JobID)
    111 
    112 stdout (JobID)
    113 
    114 stderr (JobID)
    115 
    116 stop (JobID)
    117  
    118 delete (JobID)
     125a local job is run as background fork (ie, not on controller)
     126a controller job is sent to the controller to run when it can
    119127
    120128---
    … …  
    127135  - controller says command not found
    128136  - controller has too many processes
    129   - controller takes to long to start job (scheduler timeout)
     137  - controller takes to long to start job (pending timeout)
    130138  - controller says job timed out
    131139  - controller says job crashed
    … …  
    138146 - watch for NFS lags / blocking.  if NFS has file visibility lags, we
    139147   may need to add blocking as an option to the job (-block filename)
    140 

  • trunk/Ohana/src/opihi/pantasks/CheckJobs.c

    r4451 r4452  
    1212  while ((job = NextJob ()) != NULL) {
    1313
    14     /* check for timeout */
    15     if (GetTaskTimer(job[0].start) >= job[0].task[0].timeout_period) {
     14    /* check for timeout - only local jobs timeout here */
     15    if ((job[0].mode == JOB_LOCAL) && (GetTaskTimer(job[0].start) >= job[0].task[0].timeout_period)) {
    1616      fprintf (stderr, "timeout on %s\n", job[0].task[0].name);
    1717      /* run task[0].timeout macro, if it exists */
    … …  
    2929    status = CheckJob (job);
    3030    switch (status) {
     31      case JOB_PENDING:
     32        fprintf (stderr, "job %s (%d) pending\n", job[0].task[0].name, job[0].JobID);
     33        break;
     34
    3135      case JOB_BUSY:
    3236        fprintf (stderr, "job %s (%d) busy\n", job[0].task[0].name, job[0].JobID);
    … …  
    6569
    6670      default:
    67         fprintf (stderr, "unknown exit status\n");
     71        fprintf (stderr, "unknown exit status: %d\n", status);
    6872        /** do something more useful here ?? **/
    6973        break;

  • trunk/Ohana/src/opihi/pantasks/ControllerOps.c

    r4451 r4452  
    55int status = FALSE;
    66int stdin_cntl, stdout_cntl, stderr_cntl;
    7 
    8 /* we read Nbytes from the host, then watch for the prompt */
    9 int GetJobOutput (char *cmd, int pid, IOBuffer *buffer, int Nbytes) {
    10  
    11   int i, status, Nstart;
    12   char *line;
    13 
    14   /* flush any earlier messages */
    15   ReadtoIOBuffer (buffer, stdout_cntl);
    16   FlushIOBuffer (buffer);
    17   Nstart = buffer[0].Nbuffer;
    18 
    19   /* send command to get appropriate channel */
    20   ALLOCATE (line, char, MAX (1, strlen(cmd) + 15));
    21   sprintf (line, "%s %d\n", cmd, pid);
    22   status = write (stdin_cntl, line, strlen(line));
    23   free (line);
    24 
    25   /* is pipe still open? */
    26   if ((status == -1) && (errno == EPIPE)) return (CONTROLLER_DOWN);
    27 
    28   /* read at least Nbytes, then watch for CONTROLLER_PROMPT */
    29   line = NULL;
    30   status = -1;
    31   for (i = 0; (i < CONTROLLER_TIMEOUT) && (status == -1) && (line == NULL); i++) {
    32     status = ReadtoIOBuffer (buffer, stdout_cntl);
    33     if ((buffer[0].Nbuffer - Nstart) >= Nbytes) {
    34       line = memstr (buffer[0].buffer, CONTROLLER_PROMPT, buffer[0].Nbuffer);
    35     }
    36     if (status == -1) usleep (10000);
    37   }
    38   if (status ==  0) return (CONTROLLER_DOWN);
    39   if (status == -1) return (CONTROLLER_HUNG);
    40 
    41   fprintf (stderr, "message received (GetJobOutput : %s)\n", cmd); 
    42   /* drop extra bytes from pcontrol (not pclient:job) */
    43   buffer[0].Nbuffer = Nstart + Nbytes;
    44   if (buffer[0].Nalloc > buffer[0].Nbuffer) {
    45     bzero (buffer[0].buffer + buffer[0].Nbuffer, buffer[0].Nalloc - buffer[0].Nbuffer);
    46   }
    47   return (CONTROLLER_GOOD);
    48 }
    497
    508int CheckControllerJob (Job *job) {
    … …  
    12381}
    12482
     83/* we read Nbytes from the host, then watch for the prompt */
     84int GetJobOutput (char *cmd, int pid, IOBuffer *buffer, int Nbytes) {
     85 
     86  int i, status, Nstart;
     87  char *line;
     88
     89  /* flush any earlier messages */
     90  ReadtoIOBuffer (buffer, stdout_cntl);
     91  FlushIOBuffer (buffer);
     92  Nstart = buffer[0].Nbuffer;
     93
     94  /* send command to get appropriate channel */
     95  ALLOCATE (line, char, MAX (1, strlen(cmd) + 15));
     96  sprintf (line, "%s %d\n", cmd, pid);
     97  status = write (stdin_cntl, line, strlen(line));
     98  free (line);
     99
     100  /* is pipe still open? */
     101  if ((status == -1) && (errno == EPIPE)) return (CONTROLLER_DOWN);
     102
     103  /* read at least Nbytes, then watch for CONTROLLER_PROMPT */
     104  line = NULL;
     105  status = -1;
     106  for (i = 0; (i < CONTROLLER_TIMEOUT) && (status == -1) && (line == NULL); i++) {
     107    status = ReadtoIOBuffer (buffer, stdout_cntl);
     108    if ((buffer[0].Nbuffer - Nstart) >= Nbytes) {
     109      line = memstr (buffer[0].buffer, CONTROLLER_PROMPT, buffer[0].Nbuffer);
     110    }
     111    if (status == -1) usleep (10000);
     112  }
     113  if (status ==  0) return (CONTROLLER_DOWN);
     114  if (status == -1) return (CONTROLLER_HUNG);
     115
     116  fprintf (stderr, "message received (GetJobOutput : %s)\n", cmd); 
     117  /* drop extra bytes from pcontrol (not pclient:job) */
     118  buffer[0].Nbuffer = Nstart + Nbytes;
     119  if (buffer[0].Nalloc > buffer[0].Nbuffer) {
     120    bzero (buffer[0].buffer + buffer[0].Nbuffer, buffer[0].Nalloc - buffer[0].Nbuffer);
     121  }
     122  return (CONTROLLER_GOOD);
     123}
     124
    125125/* submitting a job to the controller automatically starts controller */
    126126int SubmitControllerJob (Job *job) {
    … …  
    165165  }
    166166
     167  fprintf (stderr, "sending command to controller: %s\n", cmd);
    167168  InitIOBuffer (&buffer, 0x100);
    168169  ControllerCommand (cmd, CONTROLLER_PROMPT, &buffer);
    169170  free (cmd);
    170171
     172  fprintf (stderr, "response from controller: %s\n", buffer.buffer);
     173
    171174  /* extract the job PID from the controller response */
    172   p = memstr (buffer.buffer, "STATUS", buffer.Nbuffer);
     175  p = memstr (buffer.buffer, "JobID", buffer.Nbuffer);
    173176  if (p == NULL) {
    174177    fprintf (stderr, "missing PID in pcontrol message : programming error\n");
    … …  
    298301  FlushIOBuffer (buffer);
    299302
     303  fprintf (stderr, "send: %s\n", cmd);
     304
    300305  /* send command to client (adding on \n) */
    301   /* fprintf (stderr, "send: %s (%d)\n", command, buffer[0].Nbuffer); */
    302306  ALLOCATE (line, char, MAX (1, strlen(cmd)));
    303307  sprintf (line, "%s\n", cmd);

  • trunk/Ohana/src/opihi/scripts/sched.pro

    r4451 r4452  
     1
     2controller host localhost
    13
    24task test
    … …  
    57  periods -exec 2.0
    68  periods -timeout 2.0
    7   host local
     9  # host local
     10  host localhost
    811  # local is default
    912
Note: See TracChangeset for help on using the changeset viewer.