Changeset 6016 for trunk/doc/ipptools/ipptools.tex
- Timestamp:
- Jan 16, 2006, 8:21:43 AM (21 years ago)
- File:
-
- 1 edited
-
trunk/doc/ipptools/ipptools.tex (modified) (13 diffs)
Legend:
- Unmodified
- Added
- Removed
-
trunk/doc/ipptools/ipptools.tex
r6014 r6016 1 1 \documentclass[panstarrs,spec]{panstarrs} 2 2 3 \title{P Stask \& Metadata DB interactions} % put in your title4 \subtitle{Job Flow in IPP}3 \title{PanTasks \& the IPP Analysis Stages} 4 \subtitle{Job Relationships and Data Flow} 5 5 \author{Eugene Magnier} 6 6 \audience{IPP} 7 %\shorttitle{PSPhot}7 \shorttitle{PanTasks for IPP} 8 8 \group{Pan-STARRS IPP} 9 9 \project{Pan-STARRS IPP} … … 12 12 \docnumber{PSDC-xxx-xxx} 13 13 14 \newcommand\ugriz{$u^\prime g^\prime r^\prime i^\prime z^\prime$}15 \newcommand\grizy{$g r i z y$}16 17 14 \begin{document} 18 15 \maketitle … … 24 21 \section{Overview} 25 22 26 This document illustrates the interrelationships between IPP tasks and 27 the Metadata Database (MDDB). The Metadata Database is used to store 28 the current IPP analysis state, as well as result processing data 29 points. The tasks needed to define each of the IPP analysis stages 30 (Phase 1-4, detrend creation, etc) are illustrated in this document 31 and the relevant MDDB tables are listed. The collection of diagrams 32 shows the IPP tasks and the Metadata Database tables needed to manage 33 the flow of data through the system. 34 35 \note{The flat-field correction analysis stage is not yet defined in 36 this document} 37 38 \note{AP DB interfaces are mostly ignored} 39 40 \section{PStask queues and metadata tables : persistent vs ephemeral state} 23 This document defines the tasks used by PanTasks to control the data 24 analysis stages of the IPP. The document examines the specific tasks, 25 the related Metadata Database tables, and the pipeline management 26 commands used to connect these components together. The Metadata 27 Database is used to store the current IPP analysis state, as well as 28 result processing data points. The tasks needed to define each of the 29 IPP analysis stages (Phase 1-4, detrend creation, etc) are illustrated 30 in this document and the relevant MDDB tables are listed. The 31 collection of diagrams shows the IPP tasks and the Metadata Database 32 tables needed to manage the flow of data through the system. This 33 document does not discuss in depth the interactions of various 34 analysis programs with either DVO (the IPP Astrometric and Photometric 35 Object database) or Nebulous (the IPP large data file management 36 tool). 37 38 \section{PanTasks Summary} 39 40 PanTasks is the IPP tool which manages the sequencing of data analysis 41 steps and, with the related tool `PControl', distributes the data 42 processing across a cluster of computers. This document is not a 43 user's guide to PanTasks. We will briefly discuss the capabilities of 44 PanTasks to give the reader a basic working knowledge; for further 45 usage details, please see the PanTasks user's guide. 46 47 The purpose of PanTasks is to manage the automatic construction and 48 execution of inter-related (often repetative) operations. PanTasks 49 uses a set of rules to define UNIX commands, and their corresponding 50 command-line arguments, to be performed on some regular, repeated 51 basis. The utility of PanTasks is that it can easily define an 52 analysis system which is completely state-based, as opposed to an 53 event-driven system. 54 55 The two basic units of PanTasks operation are the 'task' and the 56 'job'. A 'job' is simply a command the user would execute on a 57 command line; it consists of a command along with optional command 58 line arguments. A 'task' is a generic description of a type of job in 59 which the details of any specific piece of data are omitted. The task 60 defines the UNIX command which corresponds to the job, and it provides 61 rules for determining the identity of data for the job. The task also 62 defines tests which are used to decide if the job may be executed. 63 Finally, it defines a polling frequency in which PanTasks should 64 attempt to construct a new job for the task. 65 66 For example, we may want to regularly copy files from one location to 67 another. Perhaps the name of the next available file is available in 68 a database table, and can be retrieved with the command 'nextFile'. 69 Perhaps we wish to check for new files every 60 seconds. Thus, the 70 task is to copy files, while a specific job of this task might be of 71 the form {\tt cp newfile newpath}. With PanTasks, we would define a 72 copy task somewhat like the following: 73 74 \begin{verbatim} 75 task copyfile 76 periods -exec 60.0 77 78 task.exec 79 $file = `nextFile` 80 if ($file == "none") 81 break 82 end 83 command cp $file $newpath 84 end 85 86 task.exit 0 87 queuepush copied $file 88 end 89 90 task.exit 1 91 queuepush failure $file 92 end 93 end 94 \end{verbatim} 95 96 In this simple example, the task is attempted every 60 seconds. If 97 there is no new file (output of 'nextFile' is 'none'), then no job 98 results from this task. If there is a new file, the copy command is 99 performed. This example is deceptively simple because one could 100 easily imagine writing a stand-along program which performs the same 101 thing. The important advantages of using PanTasks for this type of 102 operation are: 103 \begin{itemize} 104 \item the output from one job can be used to spawn a number of other tasks. 105 \item the success or failure state of each job can be used to spawn 106 other tasks. 107 \item the details of the job and data test are kept separated from the 108 rules which connect tasks together. 109 \item the relationships between tasks are kept together in a single 110 location. 111 \end{itemize} 112 113 In addition to these organizational advantages, PanTasks also provides 114 a direct connection to the tool for monitoring parallel jobs, 115 pcontrol. Thus the jobs spawned by PanTasks can be defined to run in 116 the background locally or on any of the computers in the parallel 117 processing cluster. 118 119 \section{Persistent vs Ephemeral State in Pantasks} 41 120 42 121 \begin{figure} 43 122 \begin{center} 44 \includegraphics[scale=0.85]{pics/p stasks.01.ps}45 \caption{\label{queues} P Stask queues and MDDB tables}123 \includegraphics[scale=0.85]{pics/pantasks.01.ps} 124 \caption{\label{queues} Pantask queues and MDDB tables} 46 125 \end{center} 47 126 \end{figure} 48 127 49 There are some subtleties in the interaction between IPP task 50 scheduler, PStasks, the metadata tables which store the system state, 51 and the jobs which are immediately being performed. There is a choice 52 to be made between rigorously maintaining the system state in the 53 Metadata DB at all times or in keeping an intermediate set of state 54 tables. Keeping the exact system state in the Metadata DB tables 55 would require more queries to/from the database and may introduce 56 additional latencies which are undesirable. This is because any 57 attempt by the scheduler to initiate a new job would require the 58 scheduler to mark the corresponding data item in the Metadata DB (the 59 item which acts as the trigger) with a `pending' state, and then mark 60 it again as `done' when the job actually completes. This also has the 61 drawback that, if the scheduler crashes, some initial process would be 62 required on start up to find all Metadata DB items which are in the 63 `pending' state (examining all possible items which can be in such a 64 state) and reset them to the `new' state. 65 66 We implement an alternative in which the scheduler maintains an 67 internal, ephemeral stack of the pending jobs, and only updates the 68 system state entries in the Metadata DB when jobs are actually 69 completed. In this scenario, as far as the Metadata DB tables are 70 concerned, data items transition only between a `new' and a `done' 71 state. Any jobs which are pending when the system crashes or the 72 power is lost are simply dropped, and will be automatically 73 re-constructed when the system restarts. In this paradigm, no 74 intermediate operation state is saved, and no partially completed job 75 can be recovered. Since the IPP is defined in terms of a fine 76 granularity, with jobs lasting no more than 30 - 120 seconds, this 77 model will not have a large impact. 128 The IPP, a fairly complex analysis system, uses PanTasks to select 129 jobs, distribute them to the cluster, and harvest the results. It 130 uses the Metadata Database to record the results of a given analysis 131 step, and to determine which jobs must be performed when. 132 133 There are some subtleties in the interaction between PanTasks, the 134 Metadata Database tables which store the system state, and the jobs 135 which are currently being performed. There is a choice to be made 136 between rigorously maintaining the system state in the Metadata DB at 137 all times or keeping an intermediate set of state tables. Keeping the 138 exact system state in the Metadata DB tables would require many extra 139 queries to/from the database and may introduce additional latencies 140 which are undesirable. This is because any attempt by PanTasks to 141 initiate a new job would require PanTasks to mark the corresponding 142 data item in the Metadata DB (the item which acts as the trigger) with 143 a `pending' state, and then mark it again as `done' when the job 144 actually completes. This also has the drawback that, if the system 145 crashes (eg, hardware failure), some initial process would be required 146 on start up to find all Metadata DB items which are in the `pending' 147 state (examining all possible items which can be in such a state) and 148 reset them to the `new' state. 149 150 We implement an alternative in which PanTasks maintains an internal, 151 ephemeral stack of the pending jobs, and only updates the system state 152 entries in the Metadata DB when jobs are actually completed. In this 153 scenario, as far as the Metadata DB tables are concerned, data items 154 transition only between a `new' and a `done' state. Any jobs which 155 are pending when the system crashes or the power is lost are simply 156 dropped, and will be automatically re-constructed when the system 157 restarts. In this paradigm, no intermediate operation state is saved, 158 and no partially completed job can be recovered. Since the IPP is 159 defined in terms of a fine granularity, with jobs lasting no more than 160 30 - 120 seconds, crashes under this model will not have a large 161 impact on the data processing. 78 162 79 163 Figure~\ref{queues} illustrates this ephemeral vs persistent state 80 information and the inter ation between the metadata tables and81 P Stasks. The left-hand portion of the diagram illustrates the82 recommended interaction between the metadata database tables and the83 schedulerinternal queues. Some table in the metadata database164 information and the interrelation between the metadata tables and 165 PanTasks. The left-hand portion of the diagram illustrates the 166 recommended interaction between the metadata database tables and 167 PanTasks' internal queues. Some table in the metadata database 84 168 defines a list of data items which are to be processed by some 85 analysis job. The scheduleruses a two-step approach to define the86 analysis jobs based on this list. First, one scheduler task queries87 the MDDB for a list of pending items, adds the returned items to an 88 internal scheduler queue. The process of adding the elements to the 89 queue is defined so that only unique items are added: already existing 90 items are skipped. The entries in the queue consist of the data items 91 ofinterest and an internal temporary state. At first, this would be92 `pending'. A second scheduler tasks pops `pending' entries one-by-one93 from this internal queue, submits a job based on the entry, and sets 94 t he temporary state in the internal queue to `running'. The internal95 state is needed to prevent the scheduler from re-submitting a job for96 the same data item before the first job is done or assessed. Since97 the job make take an arbitrary amount of time, the scheduler requires 98 a mechanism to remember which data items it has already submitted. 99 When the job eventually completes, the metadata database table is 100 updated noting the completion. This may be done either by the job 101 itself or by the scheduler. In addition, the state of the entry in 102 the queue can be set to either `done' or the entry can be simply103 removed from the queue.169 analysis job. PanTasks uses a two-step approach to define the 170 analysis jobs based on this list. First, one task queries the MDDB 171 for a list of pending items, adds the returned items to an internal 172 PanTasks queue. The process of adding the elements to the queue is 173 defined so that only unique items are added: already existing items 174 are skipped. The entries in the queue consist of the data items of 175 interest and an internal temporary state. At first, this would be 176 `pending'. A second tasks pops `pending' entries one-by-one from this 177 internal queue, submits a job based on the entry, and sets the 178 temporary state in the internal queue to `running'. The internal 179 state is needed to prevent PanTasks from re-submitting a job for the 180 same data item before the first job is done or assessed. Since the 181 job make take an arbitrary amount of time, PanTasks requires a 182 mechanism to remember which data items it has already submitted. When 183 the job eventually completes, the metadata database table is updated 184 noting the completion. This may be done either by the job itself or 185 by PanTasks as part of the job exit rules. In addition, the state of 186 the entry in the queue can be set to either `done' or the entry can be 187 simply removed from the queue. 104 188 105 189 The purpose of this interaction is to maintain the temporary state 106 information within non-persistent elements of the scheduler rather 107 than using the metadata database tables to store this information. 108 This concept has two advantages. First, the scheduler internal queues 109 are in memory and relatively small, thus interfacing with them is 110 quite fast for the scheduler -- this should reduce the system latency. 111 Second, by keeping this information non-persistent, the system 112 responds correctly to stopping and restarting the scheduler. Any jobs 113 which have not been completed will not be marked in the database, and 114 will be restarted naturally by the scheduler. The alternative, of 115 writing a temporary state marker in the database would require a 116 restarted scheduler to initially clean all database tables of these 117 temporary state markers. 190 information within non-persistent elements of PanTasks rather than 191 using the metadata database tables to store this information. This 192 concept has two advantages. First, PanTasks internal queues are in 193 memory and relatively small, thus interfacing with them is quite fast 194 for PanTasks -- this should reduce the system latency. Second, by 195 keeping this information non-persistent, the system responds correctly 196 to stopping and restarting PanTasks. Any jobs which have not been 197 completed will not be marked in the database, and will be restarted 198 naturally by PanTasks. The alternative, of writing a temporary state 199 marker in the database would require PanTasks, on startup, to 200 initially clean all database tables of these temporary state markers. 118 201 119 202 The right-hand portion of the diagram illustrates this process using … … 130 213 image file has been copied.) 131 214 132 In the rest of this document, the use of P Stask internal queues to215 In the rest of this document, the use of Pantask internal queues to 133 216 manage the temporary data states is glossed over and assumed part of 134 217 the tasks defined in the process. … … 138 221 \begin{figure} 139 222 \begin{center} 140 \includegraphics[scale=0.85]{pics/p stasks.02.ps}223 \includegraphics[scale=0.85]{pics/pantasks.02.ps} 141 224 \caption{\label{pcopy} Summit Copy Tasks} 142 225 \end{center} … … 216 299 \begin{figure} 217 300 \begin{center} 218 \includegraphics[scale=0.85]{pics/p stasks.03.ps}301 \includegraphics[scale=0.85]{pics/pantasks.03.ps} 219 302 \caption{\label{phase0} Phase 0 Tasks} 220 303 \end{center} … … 257 340 \begin{figure} 258 341 \begin{center} 259 \includegraphics[scale=0.85]{pics/p stasks.04.ps}342 \includegraphics[scale=0.85]{pics/pantasks.04.ps} 260 343 \caption{\label{phase1} Phase 1 Tasks} 261 344 \end{center} … … 296 379 \begin{figure} 297 380 \begin{center} 298 \includegraphics[scale=0.85]{pics/p stasks.05.ps}381 \includegraphics[scale=0.85]{pics/pantasks.05.ps} 299 382 \caption{\label{phase2} Phase 2 Tasks} 300 383 \end{center} … … 344 427 \begin{figure} 345 428 \begin{center} 346 \includegraphics[scale=0.85]{pics/p stasks.06.ps}429 \includegraphics[scale=0.85]{pics/pantasks.06.ps} 347 430 \caption{\label{phase3} Phase 3 Tasks} 348 431 \end{center} … … 379 462 \begin{figure} 380 463 \begin{center} 381 \includegraphics[scale=0.85]{pics/p stasks.07.ps}464 \includegraphics[scale=0.85]{pics/pantasks.07.ps} 382 465 \caption{\label{phase4} Phase 4 Tasks} 383 466 \end{center} … … 455 538 \begin{figure} 456 539 \begin{center} 457 \includegraphics[scale=0.85]{pics/p stasks.08.ps}540 \includegraphics[scale=0.85]{pics/pantasks.08.ps} 458 541 \caption{\label{detrend} Detrend Creation Tasks} 459 542 \end{center} … … 530 613 flat-field analysis, applying an earlier flat can result in a very 531 614 good first-pass rejection of poor input images. The logic to make 532 this leap must be part of the scheduler, since each of the individual615 this leap must be part of PanTasks, since each of the individual 533 616 blocks represent complete processing jobs. 534 617 … … 1320 1403 \end{table} 1321 1404 1405 \section{Summit Copy Tasks} 1406 1407 \begin{verbatim} 1408 # identify the images ready for copy 1409 # new entries are added to queue newImages 1410 # need to compare the new list with the ones already being processed 1411 task new.images 1412 command new.images 1413 host local 1414 1415 periods -poll 1 1416 periods -exec 1 1417 periods -timeout 30 1418 1419 # success 1420 task.exit 0 1421 local i j Nstdout Nimages 1422 # compare output with new.image queue 1423 # keep only new entries 1424 queuesize stdout -var Nstdout 1425 for i 0 $Nstdout 1426 queuepop stdout -var line 1427 queuepush newImages -uniq -key 0 "$line" 1428 end 1429 end 1430 1431 # locked list 1432 task.exit 1 1433 echo "new.images: exec failure" 1434 $new.image.failure ++ 1435 end 1436 1437 # default exit status 1438 task.exit default 1439 echo "new.images: unknown exit status: $EXIT" 1440 $new.image.failure ++ 1441 end 1442 1443 # operation times out? 1444 task.exit timeout 1445 echo "new.images: timeout" 1446 $new.image.failure ++ 1447 end 1448 end 1449 1450 # copy new images, sending job to desired host 1451 task copy.images 1452 periods -poll 0.5 1453 periods -exec 0.1 1454 periods -timeout 5 1455 1456 task.exec 1457 queuesize newImages -var N 1458 if ($N == 0) break 1459 if ($network == 0) break 1460 1461 queuepop newImages -var line 1462 list tmp -split $line 1463 $filename = $tmp:0 1464 $chip = $tmp:1 1465 $state = $tmp:2 1466 if ($state == new) 1467 # copy this image 1468 queuepush newImages -replace -key 0 "$filename $chip run" 1469 else 1470 # ignore this image 1471 queuepush newImages -replace -key 0 "$filename $chip $state" 1472 break 1473 end 1474 $host = `chip.host $chip` 1475 host $host 1476 command copy.image $filename $chip 1477 end 1478 1479 # success 1480 task.exit 0 1481 # echo "done copy..." 1482 queuepop stdout -var line 1483 list tmp -split $line 1484 $filename = $tmp:0 1485 $chip = $tmp:1 1486 exec mark.image $filename 1487 queuepush newImages -replace -key 0 "$filename $chip copy" 1488 end 1489 1490 # default exit status 1491 task.exit default 1492 echo "new.images: unknown exit status: $EXIT" 1493 $new.image.failure ++ 1494 end 1495 1496 # operation timed out? 1497 task.exit timeout 1498 echo "new.images: timeout" 1499 $new.image.failure ++ 1500 end 1501 end 1502 \end{verbatim} 1503 1322 1504 \end{document}
Note:
See TracChangeset
for help on using the changeset viewer.
