Index: trunk/doc/ipptools/ipptools.tex =================================================================== --- trunk/doc/ipptools/ipptools.tex (revision 6014) +++ trunk/doc/ipptools/ipptools.tex (revision 6016) @@ -1,9 +1,9 @@ \documentclass[panstarrs,spec]{panstarrs} -\title{PStask \& Metadata DB interactions} % put in your title -\subtitle{Job Flow in IPP} +\title{PanTasks \& the IPP Analysis Stages} +\subtitle{Job Relationships and Data Flow} \author{Eugene Magnier} \audience{IPP} -%\shorttitle{PSPhot} +\shorttitle{PanTasks for IPP} \group{Pan-STARRS IPP} \project{Pan-STARRS IPP} @@ -12,7 +12,4 @@ \docnumber{PSDC-xxx-xxx} -\newcommand\ugriz{$u^\prime g^\prime r^\prime i^\prime z^\prime$} -\newcommand\grizy{$g r i z y$} - \begin{document} \maketitle @@ -24,96 +21,182 @@ \section{Overview} -This document illustrates the interrelationships between IPP tasks and -the Metadata Database (MDDB). The Metadata Database is used to store -the current IPP analysis state, as well as result processing data -points. The tasks needed to define each of the IPP analysis stages -(Phase 1-4, detrend creation, etc) are illustrated in this document -and the relevant MDDB tables are listed. The collection of diagrams -shows the IPP tasks and the Metadata Database tables needed to manage -the flow of data through the system. - -\note{The flat-field correction analysis stage is not yet defined in - this document} - -\note{AP DB interfaces are mostly ignored} - -\section{PStask queues and metadata tables : persistent vs ephemeral state} +This document defines the tasks used by PanTasks to control the data +analysis stages of the IPP. The document examines the specific tasks, +the related Metadata Database tables, and the pipeline management +commands used to connect these components together. The Metadata +Database is used to store the current IPP analysis state, as well as +result processing data points. The tasks needed to define each of the +IPP analysis stages (Phase 1-4, detrend creation, etc) are illustrated +in this document and the relevant MDDB tables are listed. The +collection of diagrams shows the IPP tasks and the Metadata Database +tables needed to manage the flow of data through the system. This +document does not discuss in depth the interactions of various +analysis programs with either DVO (the IPP Astrometric and Photometric +Object database) or Nebulous (the IPP large data file management +tool). + +\section{PanTasks Summary} + +PanTasks is the IPP tool which manages the sequencing of data analysis +steps and, with the related tool `PControl', distributes the data +processing across a cluster of computers. This document is not a +user's guide to PanTasks. We will briefly discuss the capabilities of +PanTasks to give the reader a basic working knowledge; for further +usage details, please see the PanTasks user's guide. + +The purpose of PanTasks is to manage the automatic construction and +execution of inter-related (often repetative) operations. PanTasks +uses a set of rules to define UNIX commands, and their corresponding +command-line arguments, to be performed on some regular, repeated +basis. The utility of PanTasks is that it can easily define an +analysis system which is completely state-based, as opposed to an +event-driven system. + +The two basic units of PanTasks operation are the 'task' and the +'job'. A 'job' is simply a command the user would execute on a +command line; it consists of a command along with optional command +line arguments. A 'task' is a generic description of a type of job in +which the details of any specific piece of data are omitted. The task +defines the UNIX command which corresponds to the job, and it provides +rules for determining the identity of data for the job. The task also +defines tests which are used to decide if the job may be executed. +Finally, it defines a polling frequency in which PanTasks should +attempt to construct a new job for the task. + +For example, we may want to regularly copy files from one location to +another. Perhaps the name of the next available file is available in +a database table, and can be retrieved with the command 'nextFile'. +Perhaps we wish to check for new files every 60 seconds. Thus, the +task is to copy files, while a specific job of this task might be of +the form {\tt cp newfile newpath}. With PanTasks, we would define a +copy task somewhat like the following: + +\begin{verbatim} + task copyfile + periods -exec 60.0 + + task.exec + $file = `nextFile` + if ($file == "none") + break + end + command cp $file $newpath + end + + task.exit 0 + queuepush copied $file + end + + task.exit 1 + queuepush failure $file + end + end +\end{verbatim} + +In this simple example, the task is attempted every 60 seconds. If +there is no new file (output of 'nextFile' is 'none'), then no job +results from this task. If there is a new file, the copy command is +performed. This example is deceptively simple because one could +easily imagine writing a stand-along program which performs the same +thing. The important advantages of using PanTasks for this type of +operation are: +\begin{itemize} +\item the output from one job can be used to spawn a number of other tasks. +\item the success or failure state of each job can be used to spawn + other tasks. +\item the details of the job and data test are kept separated from the + rules which connect tasks together. +\item the relationships between tasks are kept together in a single + location. +\end{itemize} + +In addition to these organizational advantages, PanTasks also provides +a direct connection to the tool for monitoring parallel jobs, +pcontrol. Thus the jobs spawned by PanTasks can be defined to run in +the background locally or on any of the computers in the parallel +processing cluster. + +\section{Persistent vs Ephemeral State in Pantasks} \begin{figure} \begin{center} -\includegraphics[scale=0.85]{pics/pstasks.01.ps} -\caption{\label{queues} PStask queues and MDDB tables} +\includegraphics[scale=0.85]{pics/pantasks.01.ps} +\caption{\label{queues} Pantask queues and MDDB tables} \end{center} \end{figure} -There are some subtleties in the interaction between IPP task -scheduler, PStasks, the metadata tables which store the system state, -and the jobs which are immediately being performed. There is a choice -to be made between rigorously maintaining the system state in the -Metadata DB at all times or in keeping an intermediate set of state -tables. Keeping the exact system state in the Metadata DB tables -would require more queries to/from the database and may introduce -additional latencies which are undesirable. This is because any -attempt by the scheduler to initiate a new job would require the -scheduler to mark the corresponding data item in the Metadata DB (the -item which acts as the trigger) with a `pending' state, and then mark -it again as `done' when the job actually completes. This also has the -drawback that, if the scheduler crashes, some initial process would be -required on start up to find all Metadata DB items which are in the -`pending' state (examining all possible items which can be in such a -state) and reset them to the `new' state. - -We implement an alternative in which the scheduler maintains an -internal, ephemeral stack of the pending jobs, and only updates the -system state entries in the Metadata DB when jobs are actually -completed. In this scenario, as far as the Metadata DB tables are -concerned, data items transition only between a `new' and a `done' -state. Any jobs which are pending when the system crashes or the -power is lost are simply dropped, and will be automatically -re-constructed when the system restarts. In this paradigm, no -intermediate operation state is saved, and no partially completed job -can be recovered. Since the IPP is defined in terms of a fine -granularity, with jobs lasting no more than 30 - 120 seconds, this -model will not have a large impact. +The IPP, a fairly complex analysis system, uses PanTasks to select +jobs, distribute them to the cluster, and harvest the results. It +uses the Metadata Database to record the results of a given analysis +step, and to determine which jobs must be performed when. + +There are some subtleties in the interaction between PanTasks, the +Metadata Database tables which store the system state, and the jobs +which are currently being performed. There is a choice to be made +between rigorously maintaining the system state in the Metadata DB at +all times or keeping an intermediate set of state tables. Keeping the +exact system state in the Metadata DB tables would require many extra +queries to/from the database and may introduce additional latencies +which are undesirable. This is because any attempt by PanTasks to +initiate a new job would require PanTasks to mark the corresponding +data item in the Metadata DB (the item which acts as the trigger) with +a `pending' state, and then mark it again as `done' when the job +actually completes. This also has the drawback that, if the system +crashes (eg, hardware failure), some initial process would be required +on start up to find all Metadata DB items which are in the `pending' +state (examining all possible items which can be in such a state) and +reset them to the `new' state. + +We implement an alternative in which PanTasks maintains an internal, +ephemeral stack of the pending jobs, and only updates the system state +entries in the Metadata DB when jobs are actually completed. In this +scenario, as far as the Metadata DB tables are concerned, data items +transition only between a `new' and a `done' state. Any jobs which +are pending when the system crashes or the power is lost are simply +dropped, and will be automatically re-constructed when the system +restarts. In this paradigm, no intermediate operation state is saved, +and no partially completed job can be recovered. Since the IPP is +defined in terms of a fine granularity, with jobs lasting no more than +30 - 120 seconds, crashes under this model will not have a large +impact on the data processing. Figure~\ref{queues} illustrates this ephemeral vs persistent state -information and the interation between the metadata tables and -PStasks. The left-hand portion of the diagram illustrates the -recommended interaction between the metadata database tables and the -scheduler internal queues. Some table in the metadata database +information and the interrelation between the metadata tables and +PanTasks. The left-hand portion of the diagram illustrates the +recommended interaction between the metadata database tables and +PanTasks' internal queues. Some table in the metadata database defines a list of data items which are to be processed by some -analysis job. The scheduler uses a two-step approach to define the -analysis jobs based on this list. First, one scheduler task queries -the MDDB for a list of pending items, adds the returned items to an -internal scheduler queue. The process of adding the elements to the -queue is defined so that only unique items are added: already existing -items are skipped. The entries in the queue consist of the data items -of interest and an internal temporary state. At first, this would be -`pending'. A second scheduler tasks pops `pending' entries one-by-one -from this internal queue, submits a job based on the entry, and sets -the temporary state in the internal queue to `running'. The internal -state is needed to prevent the scheduler from re-submitting a job for -the same data item before the first job is done or assessed. Since -the job make take an arbitrary amount of time, the scheduler requires -a mechanism to remember which data items it has already submitted. -When the job eventually completes, the metadata database table is -updated noting the completion. This may be done either by the job -itself or by the scheduler. In addition, the state of the entry in -the queue can be set to either `done' or the entry can be simply -removed from the queue. +analysis job. PanTasks uses a two-step approach to define the +analysis jobs based on this list. First, one task queries the MDDB +for a list of pending items, adds the returned items to an internal +PanTasks queue. The process of adding the elements to the queue is +defined so that only unique items are added: already existing items +are skipped. The entries in the queue consist of the data items of +interest and an internal temporary state. At first, this would be +`pending'. A second tasks pops `pending' entries one-by-one from this +internal queue, submits a job based on the entry, and sets the +temporary state in the internal queue to `running'. The internal +state is needed to prevent PanTasks from re-submitting a job for the +same data item before the first job is done or assessed. Since the +job make take an arbitrary amount of time, PanTasks requires a +mechanism to remember which data items it has already submitted. When +the job eventually completes, the metadata database table is updated +noting the completion. This may be done either by the job itself or +by PanTasks as part of the job exit rules. In addition, the state of +the entry in the queue can be set to either `done' or the entry can be +simply removed from the queue. The purpose of this interaction is to maintain the temporary state -information within non-persistent elements of the scheduler rather -than using the metadata database tables to store this information. -This concept has two advantages. First, the scheduler internal queues -are in memory and relatively small, thus interfacing with them is -quite fast for the scheduler -- this should reduce the system latency. -Second, by keeping this information non-persistent, the system -responds correctly to stopping and restarting the scheduler. Any jobs -which have not been completed will not be marked in the database, and -will be restarted naturally by the scheduler. The alternative, of -writing a temporary state marker in the database would require a -restarted scheduler to initially clean all database tables of these -temporary state markers. +information within non-persistent elements of PanTasks rather than +using the metadata database tables to store this information. This +concept has two advantages. First, PanTasks internal queues are in +memory and relatively small, thus interfacing with them is quite fast +for PanTasks -- this should reduce the system latency. Second, by +keeping this information non-persistent, the system responds correctly +to stopping and restarting PanTasks. Any jobs which have not been +completed will not be marked in the database, and will be restarted +naturally by PanTasks. The alternative, of writing a temporary state +marker in the database would require PanTasks, on startup, to +initially clean all database tables of these temporary state markers. The right-hand portion of the diagram illustrates this process using @@ -130,5 +213,5 @@ image file has been copied.) -In the rest of this document, the use of PStask internal queues to +In the rest of this document, the use of Pantask internal queues to manage the temporary data states is glossed over and assumed part of the tasks defined in the process. @@ -138,5 +221,5 @@ \begin{figure} \begin{center} -\includegraphics[scale=0.85]{pics/pstasks.02.ps} +\includegraphics[scale=0.85]{pics/pantasks.02.ps} \caption{\label{pcopy} Summit Copy Tasks} \end{center} @@ -216,5 +299,5 @@ \begin{figure} \begin{center} -\includegraphics[scale=0.85]{pics/pstasks.03.ps} +\includegraphics[scale=0.85]{pics/pantasks.03.ps} \caption{\label{phase0} Phase 0 Tasks} \end{center} @@ -257,5 +340,5 @@ \begin{figure} \begin{center} -\includegraphics[scale=0.85]{pics/pstasks.04.ps} +\includegraphics[scale=0.85]{pics/pantasks.04.ps} \caption{\label{phase1} Phase 1 Tasks} \end{center} @@ -296,5 +379,5 @@ \begin{figure} \begin{center} -\includegraphics[scale=0.85]{pics/pstasks.05.ps} +\includegraphics[scale=0.85]{pics/pantasks.05.ps} \caption{\label{phase2} Phase 2 Tasks} \end{center} @@ -344,5 +427,5 @@ \begin{figure} \begin{center} -\includegraphics[scale=0.85]{pics/pstasks.06.ps} +\includegraphics[scale=0.85]{pics/pantasks.06.ps} \caption{\label{phase3} Phase 3 Tasks} \end{center} @@ -379,5 +462,5 @@ \begin{figure} \begin{center} -\includegraphics[scale=0.85]{pics/pstasks.07.ps} +\includegraphics[scale=0.85]{pics/pantasks.07.ps} \caption{\label{phase4} Phase 4 Tasks} \end{center} @@ -455,5 +538,5 @@ \begin{figure} \begin{center} -\includegraphics[scale=0.85]{pics/pstasks.08.ps} +\includegraphics[scale=0.85]{pics/pantasks.08.ps} \caption{\label{detrend} Detrend Creation Tasks} \end{center} @@ -530,5 +613,5 @@ flat-field analysis, applying an earlier flat can result in a very good first-pass rejection of poor input images. The logic to make -this leap must be part of the scheduler, since each of the individual +this leap must be part of PanTasks, since each of the individual blocks represent complete processing jobs. @@ -1320,3 +1403,102 @@ \end{table} +\section{Summit Copy Tasks} + +\begin{verbatim} +# identify the images ready for copy +# new entries are added to queue newImages +# need to compare the new list with the ones already being processed +task new.images + command new.images + host local + + periods -poll 1 + periods -exec 1 + periods -timeout 30 + + # success + task.exit 0 + local i j Nstdout Nimages + # compare output with new.image queue + # keep only new entries + queuesize stdout -var Nstdout + for i 0 $Nstdout + queuepop stdout -var line + queuepush newImages -uniq -key 0 "$line" + end + end + + # locked list + task.exit 1 + echo "new.images: exec failure" + $new.image.failure ++ + end + + # default exit status + task.exit default + echo "new.images: unknown exit status: $EXIT" + $new.image.failure ++ + end + + # operation times out? + task.exit timeout + echo "new.images: timeout" + $new.image.failure ++ + end +end + +# copy new images, sending job to desired host +task copy.images + periods -poll 0.5 + periods -exec 0.1 + periods -timeout 5 + + task.exec + queuesize newImages -var N + if ($N == 0) break + if ($network == 0) break + + queuepop newImages -var line + list tmp -split $line + $filename = $tmp:0 + $chip = $tmp:1 + $state = $tmp:2 + if ($state == new) + # copy this image + queuepush newImages -replace -key 0 "$filename $chip run" + else + # ignore this image + queuepush newImages -replace -key 0 "$filename $chip $state" + break + end + $host = `chip.host $chip` + host $host + command copy.image $filename $chip + end + + # success + task.exit 0 + # echo "done copy..." + queuepop stdout -var line + list tmp -split $line + $filename = $tmp:0 + $chip = $tmp:1 + exec mark.image $filename + queuepush newImages -replace -key 0 "$filename $chip copy" + end + + # default exit status + task.exit default + echo "new.images: unknown exit status: $EXIT" + $new.image.failure ++ + end + + # operation timed out? + task.exit timeout + echo "new.images: timeout" + $new.image.failure ++ + end +end +\end{verbatim} + \end{document}