Index: /trunk/Ohana/src/opihi/doc/pcontrol.txt
===================================================================
--- /trunk/Ohana/src/opihi/doc/pcontrol.txt	(revision 4451)
+++ /trunk/Ohana/src/opihi/doc/pcontrol.txt	(revision 4452)
@@ -99,4 +99,6 @@
   +host name : run job on specified host, error if not available (error when attempted, not when submitted)
   -timeout N : seconds before controller gives up on job (once started)
+  -stdout name : redirect job stdout to file directly
+  -stderr name : redirect job stderr to file directly
 
   * priority information?
@@ -137,4 +139,6 @@
 currently down, while the transition idle -> down is immediately
 followed by an attempt by pcontrol to move the host from down -> idle.
+This functionality can be used with non-automatic calling of
+CheckSystem to test the pcontrol host interface operations.
 
 does it makes sense to kill all jobs on a host?  this would only have
@@ -154,8 +158,4 @@
 ---
 
-write message
-read until MESSAGE END
-
-
 Job States:
 
@@ -164,6 +164,8 @@
 EXIT
 CRASH
-NEW
-DEL
+NEW *
+DEL *
+
+* - invisible states 
 
 Job State Transitions:
@@ -185,6 +187,8 @@
 DOWN
 OFF
-NEW
-DEL
+NEW *
+DEL *
+
+* - invisible states 
 
 Host State Transitions:
@@ -214,2 +218,5 @@
 StartHost - P
 CheckHost - P
+
+U - operation performed by the user
+P - operation performed by the program
Index: /trunk/Ohana/src/opihi/doc/scheduler.txt
===================================================================
--- /trunk/Ohana/src/opihi/doc/scheduler.txt	(revision 4451)
+++ /trunk/Ohana/src/opihi/doc/scheduler.txt	(revision 4452)
@@ -1,2 +1,65 @@
+
+2005.07.05
+
+At this point, scheduler / pcontrol / pclient all work in a basic way.
+pclient is the most robust of the three, having the simplest
+responsibility.  pcontrol is generally pretty good, though I need to
+flesh out the user interface a bit and clean up the output warning / info
+messages.  scheduler will need the most attention, though it is
+already fairly reasonable.  I need to flesh out the user commands to
+check on the controller status (basically, these need to replicate the
+status commands available to the controller).  
+
+I also need to handle the case of timeout on the controller,
+independently of timeout for a local job on the scheduler.  currently,
+if a local job exceeds the timeout value, scheduler flags it.  but, it
+does not make sense to use the same timeout value for a controller
+job. I could pass the timeout to the controller when the job executes,
+in which case it has the same meaning, essentially, for the controller
+jobs as it does for the local jobs: once you start the function, it
+needs to complete within NN seconds.  However, I think I still need to
+have a scheduler concept of a job which the controller is unable to
+complete.  It should be possible to prevent a job from sitting pending
+on the controller forever.  What exactly you do if the controller is
+unwilling to execute a job is another story (possible reasons:
+controller overload, missing required host, missing any hosts,
+something hung somewhere?).  
+
+The scheduler does not do a good job of shutting down the controller
+when it (the scheduler) exits.  This works well for the
+pcontrol/pclient interface, so the solution lies there.  
+
+I need to decide how to behave if the scheduler asks for a job with a
+required host which the controller knows is currently down or
+non-existent.  Several options could be used.  The controller could
+simply hold the task until the scheduler notices it is not being
+executed (after all, the controller does not know if the machine is
+being serviced for a short time or a long time, but the scheduler
+could know).  The controller could immediately return a failure noting
+the current state of the machine (this would put the burden of
+deciding that the machine should be available on the scheduler).  The
+controller could try to execute the job a certain number of times, and
+then it could report the failure to the scheduler.  This is not so
+different from having a pending-timeout which the scheduler tracks
+(moves the timeout check to the controller, essentially).  
+
+There was some odd behavior with 'exec echo $stdout >> foo'.  This
+resulted in empty files 'foo'.  The following work fine, so something
+is just weird:
+exec echo foobar >> foo
+output foo
+echo $stdout
+output stdout
+
+Various error conditions should be checked
+
+What do we do if a task requests a host which is not available to the
+controller (ie, not defined)?  this is similar to the problem of
+requesting a host which is down.  I think the controller should either
+immediately refuse or accept in anticipation that such a host may
+eventually be defined.
+
+I need to be careful about jobs sent to the controller and not
+harvested before stopping the scheduler execution.
 
 ---
@@ -5,8 +68,4 @@
 
 - sched: validate task hosts with controller
-- pcontrol: host (name) -check should return state
-- sched: start / stop controller connection?
-- pcontrol: add command to check status of a job
-- pcontrol: add command to dump stdout/stderr for a job
 
 ---
@@ -40,4 +99,6 @@
  - may be in task or in task.macro (exit/exec)
    (in task, value is static; in task.macro, value is defined for each instance)
+ - value of LOCAL runs job as local job (not on controller)
+ - value of NONE runs job on controller without specifying host
 
 stderr (file / variable)
@@ -54,12 +115,7 @@
  - defines relevant time-scale for the task
 
-schedule
- - runs the scheduler loop, executing the various tasks
-
-scheduler functions:
-
-InitTasks ();
-FindTask ();
-CreateTask ();
+run
+stop
+ - start or stop the scheduler loop, executing the various tasks
 
 ---
@@ -67,54 +123,6 @@
 local jobs vs controller jobs
 
-if 'host' is 'local', task is run as background fork (ie, not on controller)
-if 'host' is NULL or 'any', task is run on controller without specified host
-if 'host' is (machine), task is run on controller with specified host
-
----
-
-scheduler / controller / local interactions
-
-scheduler can execute local jobs and jobs on the controller.
-scheduler needs to initiate the connection to the controller, if the
-controller is being used.  it also needs to fork each local command.
-
-we need a command to define the controller hosts to be used.  if a
-task or job defines a host which is not known, how is this caught?
-does sched need to keep a list of valid hosts, or should the host be
-passed down to the controller.  Also, the scheduler should not
-initiate a connection to the controller unless it is requested.  is
-this implicit when defining a host?  
-
-- controller host foobar
- 
-  this should setup the controller interface (if it does not currently
-  exist), and send the command 'host foobar'.  scheduler should not
-  care if the connection is successful or not (we can know about hosts
-  which are currently off).  
-
-
----
-controller interaction:
-
-controller accepts the following commands:
-
-job [options] argv0 argv1 argv2 ...
-  -host name : run job on specified host, or any other if not available
-  +host name : run job on specified host, error if not available
-  -timeout N : seconds before controller gives up on job (once started)
-  -stdout name : redirect job stdout to file directly
-  -stderr name : redirect job stderr to file directly
-  * priority information?
-  * returns JobID
-
-status (JobID)
-
-stdout (JobID)
-
-stderr (JobID)
-
-stop (JobID)
-  
-delete (JobID)
+a local job is run as background fork (ie, not on controller)
+a controller job is sent to the controller to run when it can
 
 ---
@@ -127,5 +135,5 @@
   - controller says command not found
   - controller has too many processes
-  - controller takes to long to start job (scheduler timeout)
+  - controller takes to long to start job (pending timeout)
   - controller says job timed out
   - controller says job crashed
@@ -138,3 +146,2 @@
  - watch for NFS lags / blocking.  if NFS has file visibility lags, we
    may need to add blocking as an option to the job (-block filename)
-
Index: /trunk/Ohana/src/opihi/pantasks/CheckJobs.c
===================================================================
--- /trunk/Ohana/src/opihi/pantasks/CheckJobs.c	(revision 4451)
+++ /trunk/Ohana/src/opihi/pantasks/CheckJobs.c	(revision 4452)
@@ -12,6 +12,6 @@
   while ((job = NextJob ()) != NULL) {
 
-    /* check for timeout */
-    if (GetTaskTimer(job[0].start) >= job[0].task[0].timeout_period) {
+    /* check for timeout - only local jobs timeout here */
+    if ((job[0].mode == JOB_LOCAL) && (GetTaskTimer(job[0].start) >= job[0].task[0].timeout_period)) {
       fprintf (stderr, "timeout on %s\n", job[0].task[0].name);
       /* run task[0].timeout macro, if it exists */
@@ -29,4 +29,8 @@
     status = CheckJob (job);
     switch (status) {
+      case JOB_PENDING:
+	fprintf (stderr, "job %s (%d) pending\n", job[0].task[0].name, job[0].JobID);
+	break;
+
       case JOB_BUSY:
 	fprintf (stderr, "job %s (%d) busy\n", job[0].task[0].name, job[0].JobID);
@@ -65,5 +69,5 @@
 
       default:
-	fprintf (stderr, "unknown exit status\n");
+	fprintf (stderr, "unknown exit status: %d\n", status);
 	/** do something more useful here ?? **/
 	break;
Index: /trunk/Ohana/src/opihi/pantasks/ControllerOps.c
===================================================================
--- /trunk/Ohana/src/opihi/pantasks/ControllerOps.c	(revision 4451)
+++ /trunk/Ohana/src/opihi/pantasks/ControllerOps.c	(revision 4452)
@@ -5,46 +5,4 @@
 int status = FALSE;
 int stdin_cntl, stdout_cntl, stderr_cntl;
-
-/* we read Nbytes from the host, then watch for the prompt */ 
-int GetJobOutput (char *cmd, int pid, IOBuffer *buffer, int Nbytes) {
-  
-  int i, status, Nstart;
-  char *line;
-
-  /* flush any earlier messages */
-  ReadtoIOBuffer (buffer, stdout_cntl);
-  FlushIOBuffer (buffer);
-  Nstart = buffer[0].Nbuffer;
-
-  /* send command to get appropriate channel */
-  ALLOCATE (line, char, MAX (1, strlen(cmd) + 15));
-  sprintf (line, "%s %d\n", cmd, pid);
-  status = write (stdin_cntl, line, strlen(line));
-  free (line);
-
-  /* is pipe still open? */
-  if ((status == -1) && (errno == EPIPE)) return (CONTROLLER_DOWN);
-
-  /* read at least Nbytes, then watch for CONTROLLER_PROMPT */
-  line = NULL;
-  status = -1;
-  for (i = 0; (i < CONTROLLER_TIMEOUT) && (status == -1) && (line == NULL); i++) {
-    status = ReadtoIOBuffer (buffer, stdout_cntl);
-    if ((buffer[0].Nbuffer - Nstart) >= Nbytes) {
-      line = memstr (buffer[0].buffer, CONTROLLER_PROMPT, buffer[0].Nbuffer);
-    }
-    if (status == -1) usleep (10000);
-  }
-  if (status ==  0) return (CONTROLLER_DOWN);
-  if (status == -1) return (CONTROLLER_HUNG);
-
-  fprintf (stderr, "message received (GetJobOutput : %s)\n", cmd);  
-  /* drop extra bytes from pcontrol (not pclient:job) */
-  buffer[0].Nbuffer = Nstart + Nbytes;
-  if (buffer[0].Nalloc > buffer[0].Nbuffer) {
-    bzero (buffer[0].buffer + buffer[0].Nbuffer, buffer[0].Nalloc - buffer[0].Nbuffer);
-  }
-  return (CONTROLLER_GOOD);
-}
 
 int CheckControllerJob (Job *job) {
@@ -123,4 +81,46 @@
 }
 
+/* we read Nbytes from the host, then watch for the prompt */ 
+int GetJobOutput (char *cmd, int pid, IOBuffer *buffer, int Nbytes) {
+  
+  int i, status, Nstart;
+  char *line;
+
+  /* flush any earlier messages */
+  ReadtoIOBuffer (buffer, stdout_cntl);
+  FlushIOBuffer (buffer);
+  Nstart = buffer[0].Nbuffer;
+
+  /* send command to get appropriate channel */
+  ALLOCATE (line, char, MAX (1, strlen(cmd) + 15));
+  sprintf (line, "%s %d\n", cmd, pid);
+  status = write (stdin_cntl, line, strlen(line));
+  free (line);
+
+  /* is pipe still open? */
+  if ((status == -1) && (errno == EPIPE)) return (CONTROLLER_DOWN);
+
+  /* read at least Nbytes, then watch for CONTROLLER_PROMPT */
+  line = NULL;
+  status = -1;
+  for (i = 0; (i < CONTROLLER_TIMEOUT) && (status == -1) && (line == NULL); i++) {
+    status = ReadtoIOBuffer (buffer, stdout_cntl);
+    if ((buffer[0].Nbuffer - Nstart) >= Nbytes) {
+      line = memstr (buffer[0].buffer, CONTROLLER_PROMPT, buffer[0].Nbuffer);
+    }
+    if (status == -1) usleep (10000);
+  }
+  if (status ==  0) return (CONTROLLER_DOWN);
+  if (status == -1) return (CONTROLLER_HUNG);
+
+  fprintf (stderr, "message received (GetJobOutput : %s)\n", cmd);  
+  /* drop extra bytes from pcontrol (not pclient:job) */
+  buffer[0].Nbuffer = Nstart + Nbytes;
+  if (buffer[0].Nalloc > buffer[0].Nbuffer) {
+    bzero (buffer[0].buffer + buffer[0].Nbuffer, buffer[0].Nalloc - buffer[0].Nbuffer);
+  }
+  return (CONTROLLER_GOOD);
+}
+
 /* submitting a job to the controller automatically starts controller */
 int SubmitControllerJob (Job *job) {
@@ -165,10 +165,13 @@
   }
 
+  fprintf (stderr, "sending command to controller: %s\n", cmd);
   InitIOBuffer (&buffer, 0x100);
   ControllerCommand (cmd, CONTROLLER_PROMPT, &buffer);
   free (cmd);
 
+  fprintf (stderr, "response from controller: %s\n", buffer.buffer);
+
   /* extract the job PID from the controller response */
-  p = memstr (buffer.buffer, "STATUS", buffer.Nbuffer);
+  p = memstr (buffer.buffer, "JobID", buffer.Nbuffer);
   if (p == NULL) {
     fprintf (stderr, "missing PID in pcontrol message : programming error\n");
@@ -298,6 +301,7 @@
   FlushIOBuffer (buffer);
 
+  fprintf (stderr, "send: %s\n", cmd);
+
   /* send command to client (adding on \n) */
-  /* fprintf (stderr, "send: %s (%d)\n", command, buffer[0].Nbuffer); */
   ALLOCATE (line, char, MAX (1, strlen(cmd)));
   sprintf (line, "%s\n", cmd);
Index: /trunk/Ohana/src/opihi/scripts/sched.pro
===================================================================
--- /trunk/Ohana/src/opihi/scripts/sched.pro	(revision 4451)
+++ /trunk/Ohana/src/opihi/scripts/sched.pro	(revision 4452)
@@ -1,2 +1,4 @@
+
+controller host localhost
 
 task test
@@ -5,5 +7,6 @@
   periods -exec 2.0
   periods -timeout 2.0
-  host local
+  # host local
+  host localhost
   # local is default