Changeset 4732
- Timestamp:
- Aug 8, 2005, 10:17:52 PM (21 years ago)
- Location:
- trunk/Ohana/doc/www/html/IPP-subsystems
- Files:
-
- 1 added
- 3 edited
-
pclient/index.htm (modified) (1 diff)
-
pcontrol/index.htm (modified) (2 diffs)
-
psched/index.htm (modified) (3 diffs)
-
sequence.idx (added)
Legend:
- Unmodified
- Added
- Removed
-
trunk/Ohana/doc/www/html/IPP-subsystems/pclient/index.htm
r4727 r4732 3 3 <meta name=page content=pclient summary> 4 4 5 pclient is the remote process monitor 5 #pclient/# is the remote process monitor for pcontrol, the parallel 6 process controller. 7 8 <h3>Overview</h3> 9 10 <p> 11 The program #pclient/# is used to support the remote jobs which are 12 run on the remote hosts by #pcontrol/#. The concept of pclient is to 13 act as a buffer between the job running on the remote host and 14 pcontrol. The pcontrol design uses (by default) ssh connections 15 initiated by pcontrol to the remote hosts. These connections execute 16 the remote program of pclient. The use of a remote login process lets 17 the UNIX system take care of the user authentication issues. In this 18 case, the recommended practice is to set up ssh to allow the 19 connection to the remote host without additional authentication using 20 the appropriate authorized keys (see <a href=ssh-issues>this 21 article<a> on ssh issues). 22 23 <p> 24 It is convenient to keep a continuous connection to the remote hosts. 25 This avoids incurring the overhead of authentication for each command 26 which is executed, while keeping a high-quality user authentication 27 process in place. 28 29 <p> 30 pclient acts as a buffer between pcontrol and the remote background 31 process, allowing the continuous connection to remain viable without 32 samping pcontrol with output from the jobs. 33 34 <h3>Commands</h3> 35 36 <p> 37 pclient has a very limited command set, as follows: 38 <ul> 39 <li> job (command) : start the job (UNIX command) in the background. 40 <li> check : return the current job status 41 <li> status : return the current job status (?) 42 <li> stdout : dump the stdout stream accumulated from the job 43 back to the calling program. 44 <li> stderr : dump the stderr stream accumulated from the job 45 back to the calling program. 46 <li> reset : kill (if needed) the job and reset to accept 47 another job. 48 </ul> 49 -
trunk/Ohana/doc/www/html/IPP-subsystems/pcontrol/index.htm
r4727 r4732 1 1 <meta name=file content=index> 2 <meta name=title content=PCONTROL 3 <meta name=page content=pcontrol 2 <meta name=title content=PCONTROL.SUMMARY> 3 <meta name=page content=pcontrol.summary> 4 4 5 5 <tt>pcontrol</tt> is the IPP parallel process controller. 6 6 7 <h 2>Overview</h2>7 <h3>Overview</h3> 8 8 9 9 <p> … … 18 18 tasks in the IPP is <tt>pcontrol</tt>. 19 19 20 <p> 21 <tt>pcontrol</tt> interacts with the collection of computers under its 22 management and with other subsystems in the IPP. The IPP Controller 23 receives a variety of inputs from other subsystems, described below, 24 and initiates actions such as adding a new process to the queue of 25 pending tasks. <tt>pcontrol</tt> also provides information to other 26 subsystems on demand about its processing history and current state. 27 Each physical computer may have multiple processors; since 28 <tt>pcontrol</tt> is managing processing tasks, it treats each 29 processor independently. It is up to the system configuration if each 30 computer needs to reserve one of its CPUs to manage background tasks 31 or if <tt>pcontrol</tt> should attempt to send one task per CPU and 32 let the operating system handle the I/O load. 33 34 <h2>hosts</h2> 35 36 The Controller maintains a table of available processing computers 20 <h3>Host States</h3> 21 22 <p> 23 <tt>pcontrol</tt> maintains a table of available processing computers 37 24 (<em>hosts</em>) and tracks their status. Hosts managed by 38 <tt>pcontrol</tt> are allowed to be in one of several states, and 39 <tt>pcontrol</tt> must interact with it in an appropriate way for each 40 of those states. A Node may be {\tt alive}, {\tt dead} or {\tt off}. 41 If the Node is {\tt alive}, it responds to commands from the IPP 42 Controller and may be used for tasks subject to other constraints. If 43 it is {\tt dead}, the Node is not responsive and must not be used for 44 executing tasks. <tt>pcontrol</tt> must identify Nodes which have 45 died (not responding) and occasionally test them to see if they are 46 {\tt alive} again. Nodes which are {\tt off} are not available for 47 tasks and must not be tested. Nodes may be set to the {\tt off} or 48 {\tt dead} states by external subsystems; it is the responsibility of 49 <tt>pcontrol</tt> to return a Node to the {\tt alive} state if 50 possible. 51 52 <tt>pcontrol</tt> must honor requests (normally from the users) to 53 change the mode of any computing node on demand between {\tt off} and 54 {\tt dead}. This would normally be done after a Node has been 55 rebooted and is released to <tt>pcontrol</tt> for its use. It 56 must also be able to change the list of allowed tasks as requested by 57 external commands. 58 59 Two example scenarios illustrate the transition between these states, 60 and the basic concept of operations for <tt>pcontrol</tt>. First, 61 imagine a computer crashes. At this point <tt>pcontrol</tt> should 62 detect that the Node is no longer responsive and mark it as {\tt 63 dead}. It should occasionally try to re-establish communication with 64 the Node, potentially with longer and longer delays between attempts. 65 A human could be notified if the Node seems to remain {\tt dead} for a 66 very long time. In another scenario, a person needs to work on a 67 Node. They notify <tt>pcontrol</tt> that the machine is {\tt off}, 68 perhaps with a prior notification that the machine should be prepared 69 to go off. When work on the machine is complete, it should be placed 70 in the {\tt dead} state. Only when the person is done working and 71 testing the machine, and tells <tt>pcontrol</tt> that the machine is 72 now {\tt dead} can <tt>pcontrol</tt> attempt to re-start 73 communications and re-new processing operations on that Node. 74 75 <h2>pclient</h2> 76 77 When the Controller starts, it attempts to launch a Node Agent on each 78 of the available processing Nodes. Nodes which are not responsive are 79 marked as {\tt dead} so they may be re-tried. A Node Agent runs on 80 each of the individual nodes to execute the tasks as directed by the 81 Controller. The Node Agents communicate with the Controller via a 82 socket connection. 83 84 A Node Agent (which is only running on a Node in the {\tt alive} 85 state) may be in one of four modes: {\tt idle}, {\tt busy}, {\tt 86 done}, {\tt crash}. A Node Agent which is {\tt busy} currently has a 87 task assigned to it which is executing. The <tt>pcontrol</tt> may only 88 assign one task to a Node at a time. A Node Agent which is in the 89 {\tt idle} state may have a task assigned to it. When the Node Agent 90 detects that a tasks has finished, it changes to either the {\tt done} 91 or {\tt crash} states depending on the outcome of the process 92 execution. The <tt>pcontrol</tt> must also respect a list of task 93 restrictions which may require specific tasks to run on specific CPUs 94 or exclude specific tasks from specific CPUs. 95 96 A task being executed by the Node is run in the UNIX user space as a 97 forked process. The Node Agent must monitor the standard error and 98 standard output of the executing task and save them in separate 99 buffers. If the process exits or dies, the Node Agent must detect 100 this result and change state appropriately. The Node Agent must 101 respond to various commands from the Controller, as follows: 102 103 \paragraph{Report status} 104 105 The Node Agent returns its state ({\tt idle}, {\tt busy}, {\tt done}, 106 {\tt crash}) and the exit status of the current processing task, if 107 available. The reported exit state, if the process has completed 108 without crashing, is the UNIX exit state reported by the task: 0--256 109 with 0 indicating a successful completion. 110 111 \paragraph{Report stdout} 112 113 Send and flush the current stdout buffer. The Node Agent will return 114 the complete contents of the stdout buffer via a buffered write and 115 flush the buffer when it is finished. The Node Agent will not accept 116 more data on the stdout buffer from the current processing task until 117 the send is complete and the buffer is flushed. The daemon must 118 accept all of the buffer output. 119 120 \paragraph{Report stderr} 121 122 Identical to `report stdout', but for stderr. 123 124 \paragraph{Kill task } 125 126 The Node Agent should send a kill signal (\code{KILL} or \code{TERM}) 127 to the current processing task. When the processing task has exited, 128 the Node Agent should set its state to {\tt crash}. 129 130 \paragraph{Clear task} 131 132 The Node Agent should set its state {\tt idle}. If a processing stage 133 is currently running, it should be killed (\code{KILL} or \code{TERM}) 134 before the task is cleared. 135 136 \paragraph{Start processing stage} 137 138 The Node Agent forks a specified command. The command should be a 139 standard UNIX command without command line redirection or 140 backgrounding. The task is run with the same user ID as the Node 141 Agent, which is also the same user ID as the Controller. 142 143 \subsubsection{Tasks} 144 145 The <tt>pcontrol</tt> accepts tasks from other IPP subsystems. The task 146 requests include the specific command to be executed and are in the 147 form of a UNIX command which could be performed on any of the 148 computing nodes. Any input or output data in the commands must be a 149 valid resource regardless of the node on which the task is executed. 150 Input and output data resources must be unique where necessary to 151 avoid conflicts. It is the responsibility of the task to wait for 152 network lags (ie, NFS delays). The <tt>pcontrol</tt> gives each task a 153 unique identifier, which is returned to the requesting entity. The 154 requestor may then use that ID to obtain status information on that 155 task or to send control signals to the specific task. 156 157 Task requests may specify a desired node for the task execution. The 158 <tt>pcontrol</tt> attempts to honor the request if the node is {\tt 159 alive}, but will execute it on another node if the requested one is 160 {\tt dead} or {\tt off}. Even if a node is {\tt alive}, the IPP 161 Controller will choose another node if the specified task is not 162 allowed on the requested node. In all other cases, the <tt>pcontrol</tt> 163 waits until the currently executing processes, and processes with 164 higher priority, are completed before executing the specified task on 165 the requested node. 166 167 Task requests may specify an urgency level. The <tt>pcontrol</tt> 168 determines the priority of the task on the basis of both the urgency 169 and the age of the request. An executing task must be completed on a 170 CPU before any new task is started on that CPU, regardless of 171 priority. The urgency levels range from 0 to 2. Tasks with an 172 urgency of 1 are scheduled whenever they reach the top of the stack. 173 Tasks with an urgency of 2 are sent immediately to the top of the 174 stack. Tasks assigned a priority of 0 are maintained in the queue and 175 never executed. 176 177 It may be useful for the Controller to distinguish between tasks 178 dominated by I/O and tasks dominated by data processing. It is 179 possible that one of each of these types of tasks may be sent to the 180 same node without significantly impacting the system performance. 181 Alternatively, it may be necessary to limit a single machine with 2 182 CPUs to only one of each of these types of tasks (i.e., one processor 183 will be working on I/O while the other is working on processing). 184 Such details will be studied by the IfA IPP Team. 185 186 The <tt>pcontrol</tt> monitors the output streams from the executing 187 tasks and the exit status of the tasks. Each task is associated with 188 a log file, to which all output is written. The status, including the 189 exit status, of each task is maintained by the <tt>pcontrol</tt> so that 190 other subsystems may determine if specific tasks have started or 191 completed. 192 193 \subsubsection{Controller Interfaces} 194 195 The <tt>pcontrol</tt> must accept commands from other IPP subsystems. 196 These commands include those which govern the processing of specified 197 tasks, those which govern the behavior of specific computing nodes, 198 and those which request information from the <tt>pcontrol</tt>. The IPP 199 Controller must be able to halt the execution of a specified task, 200 delete an unexecuted task from the task list, change the priority of 201 tasks, and change the requested nodes for tasks. The <tt>pcontrol</tt> 202 must also be able to stop the current execution of a task and push it 203 to the end of the queue and also change its priority. 204 205 The <tt>pcontrol</tt> must respond to informational requests regarding the 206 collection of machines and their states as well as the collection of 207 tasks and their states. The <tt>pcontrol</tt> must monitor the execution 208 times of the different tasks and provide summary statistics. Finally, 209 the <tt>pcontrol</tt> must respond to three top-level commands: {\tt finish}, 210 {\tt stop} and {\tt abort}. When {\tt finish} is requested, no more 211 new tasks are accepted on the stack of task, and when all tasks in the 212 stack have completed, the <tt>pcontrol</tt> must exit. When {\tt stop} is 213 requested, the currently executing tasks must be completed at which 214 point the <tt>pcontrol</tt> must exit, but tasks remaining in the stack which 215 have not been started are flushed. When {\tt abort} is issued, the 216 <tt>pcontrol</tt> immediately kills all executing tasks and exits. 217 25 <tt>pcontrol</tt> are allowed to be in one of several states: 26 <tt>off</tt>, <tt>down</tt>, <tt>idle</tt>, <tt>busy</tt>, and 27 <tt>done</tt>. These states have the following meanings: 28 29 <p> 30 If the host is <tt>off</tt>, it is known to pcontrol, but pcontrol 31 does not have an active connection to the machine. Hosts which are 32 <tt>off</tt> are not available for jobs, and pcontrol does not attempt 33 to initiate a connection to them. 34 35 <p> 36 When pcontrol is told to consider a machine on, the machine is moved 37 from the <tt>off</tt> state to the <tt>down</tt> state. Pcontrol 38 attempts to initiate a connection to the host. Connections are made 39 by running a remote client on the host, using the specified connection 40 method. The connection method may be <tt>ssh</tt>, <tt>rsh</tt>, or 41 an equivalent remote shell connection. The choice is specified by the 42 COMMAND Opihi variable. The remote connection starts a dedicated 43 remote client which must accept the pcontrol client commands and 44 respond appropriately. The provided remote client is called 45 <tt>pclient</tt>, though in principal other equivalent programs could 46 be used by setting the Opihi variable SHELL (this feature more 47 generally allows a user to specify a path to the remote client, if it 48 is not in the user's path). A pcontrol user may force a host to 49 transition to the <tt>off</tt> state with the command <tt>host off 50 (hostname)</tt>. (<em> Note that this command will set only one of 51 the connections to the named host to <tt>off</tt>. If multiple 52 connections to a machine have been defined, multiple <tt>off</tt> 53 commands must be sent</em>). 54 55 <p> 56 If the remote connection is successful, the connected host is moved by 57 pcontrol from the <tt>down</tt> state to the <tt>idle</tt> state. If 58 the connection is unsuccessful, pcontrol will try again after a 59 certain period of time. If the connection continues to be 60 unsuccessful, the retry period is doubled for each successiver 61 connection attempt. If the user wants to force pcontrol to retry the 62 connection to a machine (if, for example, the timeout is now very 63 long, but the user knows the machine's ethernet cable has been 64 re-inserted...), this can be achieved with the command <tt>host retry 65 (hostname)</tt>. A host which is <tt>down</tt> is in the limbo state 66 between <tt>off</tt> and <tt>idle</tt>. 67 68 <p> 69 Once pcontrol has made a successful connection to the host, the host 70 is in the <tt>idle</tt> state. At this point, it is ready to accept 71 jobs from pcontrol for execution. Pcontrol repeatedly queries the 72 hosts to check that they are still alive. If a host is discovered to 73 be unresponsive, and particularly if the remote pipe connection has 74 closed, then the machine is moved back to the <tt>down</tt> state. 75 76 <p> 77 Hosts which are <tt>idle</tt> may accept a job from pcontrol. A job simply 78 consists of a bare UNIX command, without redirection of standard input 79 or standard output. The host will initiate the job, and pcontrol will 80 place the host into the <tt>busy</tt> state. The remote client, pclient, 81 runs the job in the background and will continue to accept input from 82 pcontrol. pcontrol will continue to check the status of the host, and 83 now also the status of the specific job. As before, if the connection 84 breaks, pcontrol will migrate the host to the <tt>down</tt> state. Any job 85 already initiated on a host which goes down will be returned for later 86 processing, so the job will not be lost. 87 88 <p> 89 When the job exits, pclient tells pcontrol that the job is completed, 90 and specifies the exit status. At this point, pcontrol will move the 91 host from <tt>busy</tt> to <tt>done</tt> state. It will stay in this 92 state until pcontrol can determine the ending conditions and reset the 93 remote client. pcontrol requests the standard error and standard 94 output from the job from pclient. pcontrol stores this data with its 95 information about the completed job, and send a reset command to the 96 remote client. Once these cleanup tasks are successfully completed, 97 pcontrol will move the host to the <tt>idle</tt> state, ready for 98 further jobs. 99 100 <p> 101 Each physical computer may have multiple processors. 102 <tt>pcontrol</tt> treats each processor independently. It is up to 103 the system configuration if each computer needs to reserve one of its 104 CPUs to manage background tasks or if <tt>pcontrol</tt> should attempt 105 to send one task per CPU and let the operating system handle the I/O 106 load. <em>some of this behavior will probably be eventually more 107 intelligent. For example, the commands which turn a host on or off 108 should be able to do the same operation to all host connections for 109 the same machine name.</em> 110 111 <p> 112 A machine may be completely removed from pcontrol's host tables with 113 the command <tt>host delete (hostname)</tt>. 114 115 <h3>Jobs</h3> 116 117 <p> 118 The <tt>pcontrol</tt> accepts new jobs with the command <tt>job 119 ...</tt>, in which the ellipsis represents the command and arguments 120 of a valid UNIX command. The commands are run under <tt>sh</tt>, and 121 are executed in the user's home directory. (<em>If it is desired, we 122 can easily add a command to tell pclient to perform <tt>cd</tt></em>). 123 Users should be wary of the conditions under which the remote jobs are 124 run. If the nodes in question all cross-mount the same home 125 directories, multiple jobs which interact with the same named file may 126 produce unexpected results. The controller cannot enforce good 127 behavior on the part of the remote jobs; it is the responsibility of 128 the user to ensure that conflicts do not arise by, eg, always using 129 unique output file names. 130 131 <p> 132 Other issues may arise from the fact that pcontrol may be choosing any 133 of the hosts to run the job. Typical failures arise if the user does 134 not realize that specific jobs do not behave the same on all machines, 135 or if a necessary resource (eg, some input data file) is only 136 available or accessible from some of the hosts. It is the 137 responsibility of the task to wait for network lags (ie, NFS delays). 138 139 <p> 140 <tt>pcontrol</tt> gives each task a unique internal identifier (Job 141 ID) equivalent to the process ID used in UNIX. When a job is 142 submitted to pcontrol, the command echoes back the Job ID. This ID 143 may be used by other pcontrol commands to obtain information about or 144 interact with the job. 145 146 <p> 147 A job may specify a specific host for the task execution. The host 148 specified for a job may be <b>required</b>, or <b>desired</b>. In the 149 first case, pcontrol, will only run the job on the specified host, 150 waiting until it is available before attempting the job. In the 151 second case, pcontrol will attempt to send the job to the specified 152 host, but if the host is unavailable (<em>how long? what 153 conditions?</em>), pcontrol will allow the job to be sent to an 154 alternative host. <tt>pcontrol</tt> attempts to honor the requests 155 for required and desired hosts, giving priority first to required-host 156 jobs, then to the desired-host jobs, and finally to all other jobs. 157 To specify a host for a job, the following commands are used: 158 159 <pre> 160 job -host (command and arguments...) 161 job +host (command and arguments...) 162 </pre> 163 164 The first case specifies a desired host, while the second specifies a 165 required host. It is also possible to specify the special host name 166 <tt>anyhost</tt>, which is equivalent to not specifying a host at all. 167 168 <p> 169 <em>Job priority / urgency levels are not implemented at this time.</em> 170 171 <p> 172 <em>I/O vs CPU tasks are not currently distinguished by pcontrol</em> 173 174 <p> 175 <tt>pcontrol</tt> stores the stdout and stderr for each completed job. 176 To retrieve these data from these streams, the user issues the 177 commands <tt>stdout (JobID)</tt> and <tt>stderr (JobID)</tt>. The 178 result is a single line specifying the number of bytes to expect, 179 followed by a dump of the buffers, followed by the prompt. It is the 180 user's responsibility to relieve pcontrol of this data load by 181 deleting jobs once they are no longer needed. Job deletion is 182 performed with the command <tt>delete (JobID)</tt>. 183 184 <p> 185 Jobs are moved between the following states by pcontrol: 186 <ul> 187 <li> pending: the job has not yet been executed. 188 <li> busy: the job is currently being executed. 189 <li> done: the job has completed, but the stdout/stderr has not been processed by 190 pcontrol. 191 <li> exit: the job has completed with a valid exit status 192 <li>crash: the job has completed with a crash status (exit on signal). 193 </ul> 194 195 <h3>Miscellaneous Commands</h3> 196 197 <p> 198 It is possible to check the status of a single host or job with the 199 user command <tt>check</tt>. 200 201 <p> 202 pcontrol continuously examines the stack of jobs, adjusting their 203 state as needed and extracting their output when it is ready. These 204 checks are performed in the background, with pcontrol ready to accept 205 further commands from the user in the foreground. These checks are 206 performed after every keystroke, and also after an inactivity timeout. 207 The interrupt interval defaults to 1 second, but may be adjusted with 208 the <tt>pulse</tt> command, which takes as an argument, the number of 209 microseconds for the timeout. 210 211 <p> 212 the pcontrol system status may be examined with the command 213 <tt>status</tt>. This provides a dump of the job stacks and the host 214 stacks. 215 216 <p> 217 It is possible to list the jobs currently in a specific stack, 218 corresponding to the list of jobs with a given state. This is done 219 with the command <tt>jobstack (stackname)</tt>. The valid stack names are 220 pending, busy, exit, crash, and done. The result is a list of all 221 jobs on the specified stack. This is useful to determine quickly 222 which jobs have exited or crashed. 223 224 <p> 225 A specific job may be killed with the command <tt>kill (JobID)</tt>. 226 This command is only valid for a job in the <tt>busy</tt> state. Any 227 job in the <tt>pending</tt>, <tt>exit</tt>, or <tt>crash</tt> state 228 may be deleted with the <tt>delete (JobID)</tt> command. This is 229 necessary to free the memory associated with the job and its output 230 streams. 231 232 <p> 233 The command <tt>verbose (mode)</tt> turns the verbosity of the 234 pcontrol operations on or off. 235 236 <p> 218 237 The <tt>pcontrol</tt> and the IPP Image Server have related needs for 219 238 information from the combined storage-and-processing nodes regarding 220 239 which nodes are available. It is not yet clear if this information is 221 best stored in a single location (either <tt>pcontrol</tt> or IPP Image 222 Server), which provides the information to other systems on demand, or 223 if both systems should maintain the information. Also, it may be 224 necessary to distinguish nodes which are available for processing from 225 those that are available to serve data as part of the IPP Image 226 Server. 227 228 The Controller maintains three tables of processing jobs: pending 229 stages, active stages, and completed stages. The pending stages are 230 those which have not yet been performed. The active stages are those 231 currently being performed on one of the remote nodes. The completed 232 stages are those which have finished, either successfully or with an 233 error state. The Controller daemon monitors the collection of remote 234 clients and sends them new pending stages when they become free. 235 236 The <tt>pcontrol</tt> provides a mechanism for users (either other 237 programs or humans) to interact with it. The user interface provides 238 commands to check the current processing job queues, the tables of 239 successful and failed jobs, to stop or delete jobs, etc. 240 240 best stored in a single location (either <tt>pcontrol</tt> or IPP 241 Image Server), which provides the information to other systems on 242 demand, or if both systems should maintain the information. Also, it 243 may be necessary to distinguish nodes which are available for 244 processing from those that are available to serve data as part of the 245 IPP Image Server. 246 -
trunk/Ohana/doc/www/html/IPP-subsystems/psched/index.htm
r4727 r4732 1 1 2 This article describes the concept, design, and operation of3 <tt>psched</tt>, the Pan-STARRS IPP task scheduler. 4 5 <h 2> Basic Concept </h2>6 7 8 9 10 <tt>psched</tt> uses a set of rules to define UNIX commands, and11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 <tt>psched</tt> allows the easy definition of a totally state-based37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 watching for the multiple inputs to be available. <tt>psched</tt>52 53 54 <tt>psched</tt> action consist of initiating UNIX commands, where55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 2 This article describes <tt>psched</tt>, the Pan-STARRS IPP task 3 scheduler. 4 5 <h3>Overview</h3> 6 7 <p> 8 The purpose of <tt>psched</tt> is to manage the automatic construction 9 and execution of inter-related (often repetative) operations. 10 <tt>Psched</tt> uses a set of rules to define UNIX commands, and 11 their corresponding command-line arguments, to be performed on some 12 regular, repeated basis. The utility of <tt>psched</tt> is that it 13 can easily define an analysis system which is completely 14 state-based, as opposed to an event-driven system. 15 16 <p> 17 Consider, for example, a telescope which obtains a collection of 18 images over the course of a night. Every minute or two, it takes an 19 image and writes the image to some disk. An event-driven analysis 20 system would involve having the telescope initiate a process at the 21 end of the exposure. This process would perform an analysis, write 22 some output, then send trigger another process. This type of 23 operation works very well for a simple set up with reliable 24 hardware. Such a system becomes more difficult to maintain when 25 hardware failures occur or when multiple systems need to interact 26 with each other. When failures occur, the triggering information 27 (the events) is easily lost, thus some mechanisms are needed to 28 detect these failures and either re-send the trigger or send an 29 alternative failure-mode trigger. Or, if two systems need to 30 interact, one or the other system must block for results from the 31 first. Stopping and restarting such an analysis system is very 32 delicate since the appropriate triggers must be set up some how, eg 33 by noticing which images have not succeeded and restarting them at 34 the appropriate stage. All of these types of methods of handling 35 complexity and failures are essentially state-based rules. 36 <tt>Psched</tt> allows the easy definition of a totally state-based 37 analysis system. 38 39 <p> 40 In a state-based system, some mechanism examines the state of the 41 system and decides which actions to perform based on the current 42 state. In the illustration above, the mechanism could examine the 43 images available (either by examining the disk or by examining the 44 state of a data table) and decide to perform an operation based on 45 what images are available. This makes it very easy to handle 46 complexity and errors. If an analysis fails, the state either is 47 not successfully updated or the error state is recorded, both 48 situations being easy to detect and easy to handle. Restarting the 49 system simply involves starting the state-monitoring mechanism. 50 Combining results from multiple input sources simply involves 51 watching for the multiple inputs to be available. <tt>Psched</tt> 52 provides a mechanism to define state monitors, and to define the 53 actions which are performed when those states occur. 54 <tt>Psched</tt> action consist of initiating UNIX commands, where 55 the arguments of those commands may depend on the results of the 56 state tests. 57 58 <h3> Tasks vs Jobs </h3> 59 60 <p> 61 The primary function of <tt>psched</tt> is to repeatedly perform 62 <b>tasks</b>, and execute <b>jobs</b> on the basis of those tasks. 63 A task consists of a set of rules which describe system state tests 64 to perform on a regular time scale. Based on the results of those 65 state tests, the task will then choose whether or not to construct a 66 job. The task also defines actions to perform upon the completion 67 of a job, based upon the output and exit status of the job. A task 68 thus defines the repeat period. It may optionally define valid or 69 invalid time ranges (eg, Mon-Fri or 10:00-17:00, etc). The task may 70 also specify that the job be run locally (ie, in the background on 71 the same computer as psched) or remotely by the parallel process 72 controller (<tt>pcontrol</tt>). A job may even be restricted to a 73 specific computer managed by <tt>pcontrol</tt>. 74 75 An example of a simple tasks is given below. 76 76 77 77 <pre> … … 93 93 </pre> 94 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 95 <p> 96 This task does not perform any system state tests; it is simply 97 constructs a new job every 5.0 seconds. The job in this case is 98 always the same: <tt> ls /data/foo </tt>. When the job finished, 99 if the job exit status is 0 (normal UNIX success status), the 100 resulting output is printed to the screen. If the job returns an 101 exit status of 1 (a failure), the failure queue receives a single 102 entry. Although they are not defined in this case, it is also 103 possible to specify the action to be taken if the job crashes (does 104 not exit normally) or if it times out (runs beyond the specified 105 timeout period). 106 107 A slightly more complex task which performs a state test and 108 constructs a command based on that test is shown below 109 109 110 110 <pre> … … 134 134 </pre> 135 135 136 The <tt>task.exec</tt> macro is executed by psched every 5.0 137 seconds. This macro executes a (hypothetical user-defined) UNIX 138 command (<tt>next.file</tt>) which examines the system state, return 139 either a filename or the word "none". If the result of this test is 140 "none", the task does nothing: no job is constructed. Otherwise, a 141 job is constructed using the name of the file returned by the state 142 test. Successful jobs have the filename added to the 'copied' 143 queue, while failed jobs add the filename to the 'failure' queue. 144 145 <h3> Parallel vs Local Job Processing </h3> 146 147 <h3> Task Restrictions </h3> 148 149 <h3> Inter-Task and Inter-Job Communications </h3> 150 151 <h2> psched Design </h2> 152 153 <h3> The Opihi Shell </h3> 154 155 <h3> Task List </h3> 156 157 <h3> Job List </h3> 158 159 <h3> pcontrol Interface </h3> 160 161 <h3> 136 The <tt>task.exec</tt> macro is executed by psched every 5.0 137 seconds. This macro executes a (hypothetical user-defined) UNIX 138 command (<tt>next.file</tt>) which examines the system state, return 139 either a filename or the word "none". If the result of this test is 140 "none", the task does nothing: no job is constructed. Otherwise, a 141 job is constructed using the name of the file returned by the state 142 test. Successful jobs have the filename added to the 'copied' 143 queue, while failed jobs add the filename to the 'failure' queue. 144 145 <h3> Parallel vs Local Job Processing </h3> 146 147 Job which are generated by psched tasks may either be run locally 148 (forked in the background on the same machine as psched) or run on 149 the IPP parallel process controller, <tt>pcontrol</tt>. The default 150 is for the job to be run locally. If a job should be run on the 151 parallel controller, this can be specified by including the command 152 <tt>host (hostname)</tt> in the definition of a task. If the value 153 of (hostname) is 'anyhost', then pcontrol may select any of its host 154 computers to run the job according to its own rules. If the value 155 of (hostname) is one of the computers managed by pcontrol, then that 156 machine will be selected for the job, if it is available. This 157 amounts to a preference to use that machine, but pcontrol is allowed 158 to substitute a different machine if it chooses. If the 159 <tt>host</tt> command is given the option <tt>-required</tt>, then 160 pcontrol is forced to use the named host, even if the machine is 161 down, unknown, or otherwise unavailable. If the machine is not 162 available, pcontrol will simply hold onto the job until the machine 163 is available or the job is deleted. Note that psched may delete 164 jobs from pcontrol if they remain pending for too long (see 165 <tt>period -timeout</tt>). 166 167 <p> 168 It is possible to interact directly with the parallel processor to 169 examine the current status, halt the parallel processor, etc. 170 Commands to the parallel processor are defined under the 171 <tt>controller</tt> command. The following controller commands are 172 available: 173 174 <ul> 175 <li> <tt>controller host (command) (hostname)</tt>: Manage the 176 parallel controller collection of hosts. This command can be used 177 to <tt>add</tt> a new host, the <tt>delete</tt> one of the existing 178 hosts, to turn a host <tt>on</tt> or <tt>off</tt>, and to 179 <tt>check</tt> the status of a host 180 <ul> 181 <li> <tt>controller host add (hostname)</tt>: add a new host. 182 <li> <tt>controller host delete (hostname)</tt>: delete a host. 183 <li> <tt>controller host on (hostname)</tt>: tell pcontrol that the host is on. 184 <li> <tt>controller host off (hostname)</tt>: tell pcontrol that the host is off. 185 <li> <tt>controller host retry (hostname)</tt>: tell pcontrol to retry the host connection. 186 <li> <tt>controller host check (hostname)</tt>: check the current status of a host. 187 </ul> 188 <li><tt>controller exit</tt>: stop controller execution. 189 <li><tt>controller status</tt>: report controller current status. 190 <li><tt>controller check</tt>: check job or host status. 191 <li><tt>controller output</tt>: print accumulated messages from the controller. 192 </ul> 193 194 It is also possible to specify a host for a task which has not been 195 identified to the controller. If such a host is required, the 196 controller will simply keep the associated jobs in the pending state 197 until such a machine exists. See the <a href=../pcontrol>pcontrol</a> 198 documentation for further discussion of the controller manipuation of 199 jobs and hosts. 200 201 <h3> Task Restrictions </h3> 202 203 Tasks may have restrictions on when they create jobs and how 204 frequently they create jobs. The task command <tt>trange</tt> is 205 used to specify a valid or invalid time range for a task. A valid 206 time range limits the task evaluation to that time period. An 207 invalid time range excludes task evaluation from the time period. 208 Any number of time range restrictions may be defined, and the union 209 of all restrictions will define if a job may be created. By 210 default, the time range is an inclusive time range: the task is 211 evaluated only if the current time falls within the specified time 212 range. Alternatively, if the <tt>-exclude</tt> flag is given, the 213 time range is exclusive, in which case the task is <em>not</em> 214 evaluated if the current time falls within this range. 215 216 <p> 217 The time range may be given as a range of absolute dates as follows: 218 219 <pre> 220 trange YYYY/MM/DD,HH:MM:SS YYYY/MM/DD,HH:MM:SS 221 </pre> 222 223 where the two dates specify the start and end of the time range. In 224 either of these date representations, the least-significant elements 225 of the date and time may be dropped, defaulting to 00 (in the case 226 of hours, minutes, and seconds) or 01 (in the case of day and 227 months). Rather than specifying an end date, it is also valid to 228 specify a time interval from the starting date. The time interval 229 is specified as a number followed by a unit indicated by a single 230 letter: d (days), h (hours), m (minutes), s (seconds). 231 232 <p> 233 The time range may also be specified as a repeated period of time, 234 either as a time of day or a day and time of week. In the first 235 case, the time range is specified as follows: 236 237 <pre> 238 trange HH:MM:SS HH:MM:SS 239 </pre> 240 241 where again the least-significant elements may be dropped and 242 default to 00. This type of restriction defines a time range which 243 is valid every day. The alternative is to specify a time range 244 within the week, in the following form: 245 246 <pre> 247 trange DAY@HH:MM:SS DAY@HH:MM:SS 248 </pre> 249 250 where the value of DAY may take on any of the three letter 251 day-of-week names (Sun, Mon, Tue, etc). This restriction specifies 252 a start and end time within a week which is evaluated for each 253 week. 254 255 <p> 256 Below are several examples of valid time range restrictions 257 258 <pre> 259 trange 2005/01/01 2005/12/31 (only run during 2005!) 260 trange 18:00 00:00 (only run from 6pm until midnight) 261 trange 00:00 06:00 (only run from midnight until 6am) 262 trange Mon@08:00 Fri@17:00 (only run between Mon morning and Fri afternoon) 263 trange -exclude 12:00 13:00 (skip 1 hour from noon) 264 </pre> 265 266 <em>Note that the current definition of trange does not include time 267 zone information. This means that <b>all</b> times are relative to 268 UT. This should be addressed by adding a timezone environment 269 variable to psched and by allowing the trange to define a timezone 270 offset.</em> 271 272 <p> 273 It is also possible to restrict the total number of jobs which are 274 spawned for a given task. This is done with the <tt>nmax</tt> 275 command, which is given as part of the task definition. Once a task 276 has constructed nmax jobs, it stops task evaluation. It is possible 277 to redefine the value of nmax at any time by redefining the task. 278 Any time the task is redefined, the new values for any task concept 279 will override the existing values for the task concept. 280 281 <h3> Inter-Task and Inter-Job Communications </h3> 282 283 There are several ways in which the results of jobs may be used to 284 influence other jobs. These include: 285 <ul> 286 <li> external communications 287 <li> job exit status 288 <li> job stdout parsing 289 </ul> 290 291 <p> 292 It is always possible for the interprocess communication to be 293 performed externally: all jobs may simply write results to an 294 external data source which is queried as part of the task 295 evaluation. Psched may interact with UNIX programs using Opihi 296 system interaction functions. These interaction methods include: 297 the backticks for setting Opihi variables: 298 299 <pre> 300 $variable = `UNIX Command` 301 </pre> 302 303 The exec command (which executes a UNIX command) and the backticks 304 both receive the UNIX command exit status, setting the variable 305 $STATUS. It is also possible to set a variable list to the output 306 of a UNIX command: 307 308 <pre> 309 list var -x "UNIX Command" 310 </pre> 311 312 In this last case, the values $var:0 - $var:N-1 are set to the value 313 of the stdout lines from the UNIX command, and the value $var:n is 314 set to the number of output lines. 315 316 <p> 317 Fine-grained control over the job exit status is available with the 318 <tt>task.exit</tt> macro command. This allows a task to define an 319 exit macro which is performed for different exit status conditions. 320 The argument to the <tt>task.exit</tt> command is the exit status 321 value which triggers the macro. This may consist of any valid 322 numeric exit status value (0-255). It may also have the value 323 <tt>crash</tt>, in which case the macro is executed if the program 324 exited as a result of a signal (ie, segmentation fault, etc). 325 Finally, if may have the value <tt>default</tt>, in which case, the 326 macro is run if no other macro describes the exit status. 327 328 <p> 329 Jobs may transmit their results back to psched for further 330 evaluation through the standard output and standard error streams. 331 Whenever a job exits, the complete stdout and stderr streams from 332 the job are pushed onto the psched queues <tt>stdout</tt> and 333 <tt>stderr</tt>. The job exit macros may then parse these queues, 334 moving the results into other psched / Opihi data containers 335 (queues, variables, vectors, whatever is appropriate). <em>Note 336 that currently, the output data is simply pushed onto these output 337 queues. It is currently the responsibility of the psched programmer 338 to use or dispose of the data in these queues. This may change in 339 the future: the queues may be flushed for each job completion.</em> 340 341 <h3>Running the scheduler</h3> 342 343 Once a set of tasks has been defined, the scheduler can be started. 344 The scheduler will run in the background, at regular intervals 345 examining the collection of tasks and jobs. In these periods, the 346 scheduler attempts to construct new jobs and checks on the status of 347 jobs which may have finished, either locally or on the controller. 348 To start the scheduler, give the command <tt>run</tt>. To stop the 349 scheduler, given the command <tt>stop</tt>. The current status of 350 the scheduler, controller, and any jobs which have been spawned are 351 listed with the <tt>status</tt> command. 352 353 <p> 354 It is also possible to kill or delete individual jobs by hand with 355 the commands <tt>kill (jobID)</tt> or <tt>delete (jobID)</tt>. 356 357 <h3>Other features</h3> 358 359 The command <tt>verbose (mode)</tt> turns the verbosity of the 360 scheduler operations on or off. 361 362 It is possible to change the rate at which the scheduler checks the 363 task and job lists with the command <tt>pulse (usec)</tt), which 364 takes as an argument, the number of microseconds between timeouts.
Note:
See TracChangeset
for help on using the changeset viewer.
