Changeset 4732 for trunk/Ohana/doc/www/html/IPP-subsystems/psched/index.htm
- Timestamp:
- Aug 8, 2005, 10:17:52 PM (21 years ago)
- File:
-
- 1 edited
Legend:
- Unmodified
- Added
- Removed
-
trunk/Ohana/doc/www/html/IPP-subsystems/psched/index.htm
r4727 r4732 1 1 2 This article describes the concept, design, and operation of3 <tt>psched</tt>, the Pan-STARRS IPP task scheduler. 4 5 <h 2> Basic Concept </h2>6 7 8 9 10 <tt>psched</tt> uses a set of rules to define UNIX commands, and11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 <tt>psched</tt> allows the easy definition of a totally state-based37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 watching for the multiple inputs to be available. <tt>psched</tt>52 53 54 <tt>psched</tt> action consist of initiating UNIX commands, where55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 2 This article describes <tt>psched</tt>, the Pan-STARRS IPP task 3 scheduler. 4 5 <h3>Overview</h3> 6 7 <p> 8 The purpose of <tt>psched</tt> is to manage the automatic construction 9 and execution of inter-related (often repetative) operations. 10 <tt>Psched</tt> uses a set of rules to define UNIX commands, and 11 their corresponding command-line arguments, to be performed on some 12 regular, repeated basis. The utility of <tt>psched</tt> is that it 13 can easily define an analysis system which is completely 14 state-based, as opposed to an event-driven system. 15 16 <p> 17 Consider, for example, a telescope which obtains a collection of 18 images over the course of a night. Every minute or two, it takes an 19 image and writes the image to some disk. An event-driven analysis 20 system would involve having the telescope initiate a process at the 21 end of the exposure. This process would perform an analysis, write 22 some output, then send trigger another process. This type of 23 operation works very well for a simple set up with reliable 24 hardware. Such a system becomes more difficult to maintain when 25 hardware failures occur or when multiple systems need to interact 26 with each other. When failures occur, the triggering information 27 (the events) is easily lost, thus some mechanisms are needed to 28 detect these failures and either re-send the trigger or send an 29 alternative failure-mode trigger. Or, if two systems need to 30 interact, one or the other system must block for results from the 31 first. Stopping and restarting such an analysis system is very 32 delicate since the appropriate triggers must be set up some how, eg 33 by noticing which images have not succeeded and restarting them at 34 the appropriate stage. All of these types of methods of handling 35 complexity and failures are essentially state-based rules. 36 <tt>Psched</tt> allows the easy definition of a totally state-based 37 analysis system. 38 39 <p> 40 In a state-based system, some mechanism examines the state of the 41 system and decides which actions to perform based on the current 42 state. In the illustration above, the mechanism could examine the 43 images available (either by examining the disk or by examining the 44 state of a data table) and decide to perform an operation based on 45 what images are available. This makes it very easy to handle 46 complexity and errors. If an analysis fails, the state either is 47 not successfully updated or the error state is recorded, both 48 situations being easy to detect and easy to handle. Restarting the 49 system simply involves starting the state-monitoring mechanism. 50 Combining results from multiple input sources simply involves 51 watching for the multiple inputs to be available. <tt>Psched</tt> 52 provides a mechanism to define state monitors, and to define the 53 actions which are performed when those states occur. 54 <tt>Psched</tt> action consist of initiating UNIX commands, where 55 the arguments of those commands may depend on the results of the 56 state tests. 57 58 <h3> Tasks vs Jobs </h3> 59 60 <p> 61 The primary function of <tt>psched</tt> is to repeatedly perform 62 <b>tasks</b>, and execute <b>jobs</b> on the basis of those tasks. 63 A task consists of a set of rules which describe system state tests 64 to perform on a regular time scale. Based on the results of those 65 state tests, the task will then choose whether or not to construct a 66 job. The task also defines actions to perform upon the completion 67 of a job, based upon the output and exit status of the job. A task 68 thus defines the repeat period. It may optionally define valid or 69 invalid time ranges (eg, Mon-Fri or 10:00-17:00, etc). The task may 70 also specify that the job be run locally (ie, in the background on 71 the same computer as psched) or remotely by the parallel process 72 controller (<tt>pcontrol</tt>). A job may even be restricted to a 73 specific computer managed by <tt>pcontrol</tt>. 74 75 An example of a simple tasks is given below. 76 76 77 77 <pre> … … 93 93 </pre> 94 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 95 <p> 96 This task does not perform any system state tests; it is simply 97 constructs a new job every 5.0 seconds. The job in this case is 98 always the same: <tt> ls /data/foo </tt>. When the job finished, 99 if the job exit status is 0 (normal UNIX success status), the 100 resulting output is printed to the screen. If the job returns an 101 exit status of 1 (a failure), the failure queue receives a single 102 entry. Although they are not defined in this case, it is also 103 possible to specify the action to be taken if the job crashes (does 104 not exit normally) or if it times out (runs beyond the specified 105 timeout period). 106 107 A slightly more complex task which performs a state test and 108 constructs a command based on that test is shown below 109 109 110 110 <pre> … … 134 134 </pre> 135 135 136 The <tt>task.exec</tt> macro is executed by psched every 5.0 137 seconds. This macro executes a (hypothetical user-defined) UNIX 138 command (<tt>next.file</tt>) which examines the system state, return 139 either a filename or the word "none". If the result of this test is 140 "none", the task does nothing: no job is constructed. Otherwise, a 141 job is constructed using the name of the file returned by the state 142 test. Successful jobs have the filename added to the 'copied' 143 queue, while failed jobs add the filename to the 'failure' queue. 144 145 <h3> Parallel vs Local Job Processing </h3> 146 147 <h3> Task Restrictions </h3> 148 149 <h3> Inter-Task and Inter-Job Communications </h3> 150 151 <h2> psched Design </h2> 152 153 <h3> The Opihi Shell </h3> 154 155 <h3> Task List </h3> 156 157 <h3> Job List </h3> 158 159 <h3> pcontrol Interface </h3> 160 161 <h3> 136 The <tt>task.exec</tt> macro is executed by psched every 5.0 137 seconds. This macro executes a (hypothetical user-defined) UNIX 138 command (<tt>next.file</tt>) which examines the system state, return 139 either a filename or the word "none". If the result of this test is 140 "none", the task does nothing: no job is constructed. Otherwise, a 141 job is constructed using the name of the file returned by the state 142 test. Successful jobs have the filename added to the 'copied' 143 queue, while failed jobs add the filename to the 'failure' queue. 144 145 <h3> Parallel vs Local Job Processing </h3> 146 147 Job which are generated by psched tasks may either be run locally 148 (forked in the background on the same machine as psched) or run on 149 the IPP parallel process controller, <tt>pcontrol</tt>. The default 150 is for the job to be run locally. If a job should be run on the 151 parallel controller, this can be specified by including the command 152 <tt>host (hostname)</tt> in the definition of a task. If the value 153 of (hostname) is 'anyhost', then pcontrol may select any of its host 154 computers to run the job according to its own rules. If the value 155 of (hostname) is one of the computers managed by pcontrol, then that 156 machine will be selected for the job, if it is available. This 157 amounts to a preference to use that machine, but pcontrol is allowed 158 to substitute a different machine if it chooses. If the 159 <tt>host</tt> command is given the option <tt>-required</tt>, then 160 pcontrol is forced to use the named host, even if the machine is 161 down, unknown, or otherwise unavailable. If the machine is not 162 available, pcontrol will simply hold onto the job until the machine 163 is available or the job is deleted. Note that psched may delete 164 jobs from pcontrol if they remain pending for too long (see 165 <tt>period -timeout</tt>). 166 167 <p> 168 It is possible to interact directly with the parallel processor to 169 examine the current status, halt the parallel processor, etc. 170 Commands to the parallel processor are defined under the 171 <tt>controller</tt> command. The following controller commands are 172 available: 173 174 <ul> 175 <li> <tt>controller host (command) (hostname)</tt>: Manage the 176 parallel controller collection of hosts. This command can be used 177 to <tt>add</tt> a new host, the <tt>delete</tt> one of the existing 178 hosts, to turn a host <tt>on</tt> or <tt>off</tt>, and to 179 <tt>check</tt> the status of a host 180 <ul> 181 <li> <tt>controller host add (hostname)</tt>: add a new host. 182 <li> <tt>controller host delete (hostname)</tt>: delete a host. 183 <li> <tt>controller host on (hostname)</tt>: tell pcontrol that the host is on. 184 <li> <tt>controller host off (hostname)</tt>: tell pcontrol that the host is off. 185 <li> <tt>controller host retry (hostname)</tt>: tell pcontrol to retry the host connection. 186 <li> <tt>controller host check (hostname)</tt>: check the current status of a host. 187 </ul> 188 <li><tt>controller exit</tt>: stop controller execution. 189 <li><tt>controller status</tt>: report controller current status. 190 <li><tt>controller check</tt>: check job or host status. 191 <li><tt>controller output</tt>: print accumulated messages from the controller. 192 </ul> 193 194 It is also possible to specify a host for a task which has not been 195 identified to the controller. If such a host is required, the 196 controller will simply keep the associated jobs in the pending state 197 until such a machine exists. See the <a href=../pcontrol>pcontrol</a> 198 documentation for further discussion of the controller manipuation of 199 jobs and hosts. 200 201 <h3> Task Restrictions </h3> 202 203 Tasks may have restrictions on when they create jobs and how 204 frequently they create jobs. The task command <tt>trange</tt> is 205 used to specify a valid or invalid time range for a task. A valid 206 time range limits the task evaluation to that time period. An 207 invalid time range excludes task evaluation from the time period. 208 Any number of time range restrictions may be defined, and the union 209 of all restrictions will define if a job may be created. By 210 default, the time range is an inclusive time range: the task is 211 evaluated only if the current time falls within the specified time 212 range. Alternatively, if the <tt>-exclude</tt> flag is given, the 213 time range is exclusive, in which case the task is <em>not</em> 214 evaluated if the current time falls within this range. 215 216 <p> 217 The time range may be given as a range of absolute dates as follows: 218 219 <pre> 220 trange YYYY/MM/DD,HH:MM:SS YYYY/MM/DD,HH:MM:SS 221 </pre> 222 223 where the two dates specify the start and end of the time range. In 224 either of these date representations, the least-significant elements 225 of the date and time may be dropped, defaulting to 00 (in the case 226 of hours, minutes, and seconds) or 01 (in the case of day and 227 months). Rather than specifying an end date, it is also valid to 228 specify a time interval from the starting date. The time interval 229 is specified as a number followed by a unit indicated by a single 230 letter: d (days), h (hours), m (minutes), s (seconds). 231 232 <p> 233 The time range may also be specified as a repeated period of time, 234 either as a time of day or a day and time of week. In the first 235 case, the time range is specified as follows: 236 237 <pre> 238 trange HH:MM:SS HH:MM:SS 239 </pre> 240 241 where again the least-significant elements may be dropped and 242 default to 00. This type of restriction defines a time range which 243 is valid every day. The alternative is to specify a time range 244 within the week, in the following form: 245 246 <pre> 247 trange DAY@HH:MM:SS DAY@HH:MM:SS 248 </pre> 249 250 where the value of DAY may take on any of the three letter 251 day-of-week names (Sun, Mon, Tue, etc). This restriction specifies 252 a start and end time within a week which is evaluated for each 253 week. 254 255 <p> 256 Below are several examples of valid time range restrictions 257 258 <pre> 259 trange 2005/01/01 2005/12/31 (only run during 2005!) 260 trange 18:00 00:00 (only run from 6pm until midnight) 261 trange 00:00 06:00 (only run from midnight until 6am) 262 trange Mon@08:00 Fri@17:00 (only run between Mon morning and Fri afternoon) 263 trange -exclude 12:00 13:00 (skip 1 hour from noon) 264 </pre> 265 266 <em>Note that the current definition of trange does not include time 267 zone information. This means that <b>all</b> times are relative to 268 UT. This should be addressed by adding a timezone environment 269 variable to psched and by allowing the trange to define a timezone 270 offset.</em> 271 272 <p> 273 It is also possible to restrict the total number of jobs which are 274 spawned for a given task. This is done with the <tt>nmax</tt> 275 command, which is given as part of the task definition. Once a task 276 has constructed nmax jobs, it stops task evaluation. It is possible 277 to redefine the value of nmax at any time by redefining the task. 278 Any time the task is redefined, the new values for any task concept 279 will override the existing values for the task concept. 280 281 <h3> Inter-Task and Inter-Job Communications </h3> 282 283 There are several ways in which the results of jobs may be used to 284 influence other jobs. These include: 285 <ul> 286 <li> external communications 287 <li> job exit status 288 <li> job stdout parsing 289 </ul> 290 291 <p> 292 It is always possible for the interprocess communication to be 293 performed externally: all jobs may simply write results to an 294 external data source which is queried as part of the task 295 evaluation. Psched may interact with UNIX programs using Opihi 296 system interaction functions. These interaction methods include: 297 the backticks for setting Opihi variables: 298 299 <pre> 300 $variable = `UNIX Command` 301 </pre> 302 303 The exec command (which executes a UNIX command) and the backticks 304 both receive the UNIX command exit status, setting the variable 305 $STATUS. It is also possible to set a variable list to the output 306 of a UNIX command: 307 308 <pre> 309 list var -x "UNIX Command" 310 </pre> 311 312 In this last case, the values $var:0 - $var:N-1 are set to the value 313 of the stdout lines from the UNIX command, and the value $var:n is 314 set to the number of output lines. 315 316 <p> 317 Fine-grained control over the job exit status is available with the 318 <tt>task.exit</tt> macro command. This allows a task to define an 319 exit macro which is performed for different exit status conditions. 320 The argument to the <tt>task.exit</tt> command is the exit status 321 value which triggers the macro. This may consist of any valid 322 numeric exit status value (0-255). It may also have the value 323 <tt>crash</tt>, in which case the macro is executed if the program 324 exited as a result of a signal (ie, segmentation fault, etc). 325 Finally, if may have the value <tt>default</tt>, in which case, the 326 macro is run if no other macro describes the exit status. 327 328 <p> 329 Jobs may transmit their results back to psched for further 330 evaluation through the standard output and standard error streams. 331 Whenever a job exits, the complete stdout and stderr streams from 332 the job are pushed onto the psched queues <tt>stdout</tt> and 333 <tt>stderr</tt>. The job exit macros may then parse these queues, 334 moving the results into other psched / Opihi data containers 335 (queues, variables, vectors, whatever is appropriate). <em>Note 336 that currently, the output data is simply pushed onto these output 337 queues. It is currently the responsibility of the psched programmer 338 to use or dispose of the data in these queues. This may change in 339 the future: the queues may be flushed for each job completion.</em> 340 341 <h3>Running the scheduler</h3> 342 343 Once a set of tasks has been defined, the scheduler can be started. 344 The scheduler will run in the background, at regular intervals 345 examining the collection of tasks and jobs. In these periods, the 346 scheduler attempts to construct new jobs and checks on the status of 347 jobs which may have finished, either locally or on the controller. 348 To start the scheduler, give the command <tt>run</tt>. To stop the 349 scheduler, given the command <tt>stop</tt>. The current status of 350 the scheduler, controller, and any jobs which have been spawned are 351 listed with the <tt>status</tt> command. 352 353 <p> 354 It is also possible to kill or delete individual jobs by hand with 355 the commands <tt>kill (jobID)</tt> or <tt>delete (jobID)</tt>. 356 357 <h3>Other features</h3> 358 359 The command <tt>verbose (mode)</tt> turns the verbosity of the 360 scheduler operations on or off. 361 362 It is possible to change the rate at which the scheduler checks the 363 task and job lists with the command <tt>pulse (usec)</tt), which 364 takes as an argument, the number of microseconds between timeouts.
Note:
See TracChangeset
for help on using the changeset viewer.
