Index: /trunk/doc/design/ippSDRS.tex =================================================================== --- /trunk/doc/design/ippSDRS.tex (revision 2171) +++ /trunk/doc/design/ippSDRS.tex (revision 2172) @@ -1,3 +1,3 @@ -%%% $Id: ippSDRS.tex,v 1.7 2004-10-19 01:35:26 eugene Exp $ +%%% $Id: ippSDRS.tex,v 1.8 2004-10-19 03:28:04 eugene Exp $ \documentclass[panstarrs]{panstarrs} @@ -433,7 +433,9 @@ \begin{figure} -\resizebox{6in}{!}{\includegraphics{pics/ImageServer}} +\begin{center} +\resizebox{4.5in}{!}{\includegraphics{pics/ImageServer}} \caption{The components of the IPP Image Server.} \label{fig:ImageServer} +\end{center} \end{figure} @@ -730,34 +732,17 @@ \subsubsection{AP Database Tables} -The AP Database divides the sky into a regions, which are in turn -sub-divided into regions, in a hierarchical series. The regions are -used to subdivide the tables of images, objects, and detections. -These three tables are the three largest in terms of both data volume -and number of rows. Since nearly all interactions with the AP -Database performed by the IPP are limited in spatial coverage, -subdividing the tables allows a specific interaction to search only a -small subset of the data. The table of images is the smallest of the -three; the table of detections is likely to be the largest. As a -result, the images tables will be subdivided at a shallow hierarchical -level, while the objects and detections are subdivided on deeper (more -finely sampled) levels. The region table defines the sky regions and -specifies if the region corresponds to an image table, and object -table, and/or a detection table. It also specified which regions in -the next level of the hierarchy are contained by the region, and which -parent region it belongs to. In addition to improving the spatial -access to the image, object, and detection data, the region table -allows for the multiple computers to serve the database tables. The -region file specifies the machine which stores the specific table. +Table~\ref{APDBTables} lists the tables used by the AP Database. The +contents of these tables are outlined in +Appendix~\ref{APDBTableContents}. Below, we discuss how these tables +are used by the AP Database software. Three of the tables are not +simple tables but instead are divided into many subtables, each of +which represents a portion of the sky. These subtables may also be +distributed across different computers to distribute the processing +load. The table of Images lists all of the images which provided the data in the AP Database. In general, these images correspond to the Chips. -\tbd{how does the AP Database know about the relationship between a -collection of chips?}. This table includes sufficient astrometric -parameters to represent the coordinates of the detections to a -sufficient accuracy: \tbr{3rd order polynomial across the chip?}. -\tbr{does the AP Database know about FPA, Chip, Distortion Model, etc? -I think it probably needs to if it is going to solve for distortion -models. however, this operation may be a combination of AP DB -interaction and MD DB interaction.} +This table includes sufficient astrometric parameters to represent the +coordinates of the detections to a sufficient accuracy. The Images in the image table group are stored in the Image table @@ -791,4 +776,27 @@ non-detection statistics. +The table of regions is used to subdivide the tables of images, +objects, and detections. The AP Database divides the sky into a +hierarchy of regions (portions of the sky) each of which is in turn +sub-divided into smaller portions. These three tables are the three +largest in terms of both data volume and number of rows. Since nearly +all interactions with the AP Database performed by the IPP are limited +in spatial coverage, subdividing the tables allows a specific +interaction to search only a small subset of the data. The table of +images is the smallest of the three; the table of detections is likely +to be the largest. As a result, the images tables will be subdivided +at a shallow hierarchical level, while the objects and detections are +subdivided on deeper (more finely sampled) levels. The region table +defines the sky regions and specifies if the region corresponds to an +image table, and object table, and/or a detection table. It also +specified which regions in the next level of the hierarchy are +contained by the region, and which parent region it belongs to. In +addition to improving the spatial access to the image, object, and +detection data, the region table allows for the multiple computers to +serve the database tables. The region file specifies the machine +which stores the specific table. Figure~\ref{ABDBRegions} illustrates +this subdivision of the sky and the association between different +levels of the hierarchy with different subtables. + The Filters table identifies all of the physical filters (specific, named pieces of glass) known to the system. A related table, @@ -797,4 +805,43 @@ it may be a derived photometry system. \tbd{distinguish between reference, average, and detection photcodes}. + +\begin{table} +\begin{center} +\caption{AP Database Tables\label{APDBTables}} +\begin{tabular}{ll} +\hline +\hline +{\bf Table Name} & {\bf Description} \\ +\hline +Region Table & spatial distribution of tables \\ +Images & The images that have objects in the DB. \\ +Image Overlaps & Image regions which are touched by specific images. \\ +Objects & The objects --- average properties of multiple detections of the same object. \\ +Average Magnitudes & Average photometry in multiple filters \\ +Detections & Detections of sources in an image. \\ +Non-Detections & Non-detections of objects in an image. \\ +Filters & Filters understood by the system. \\ +Photcodes & Transformations between different photometric systems \\ +Database Machines & computers used to store the tables \\ +\hline +\end{tabular} +\end{center} +\end{table} + +\begin{figure} +\begin{center} +\resizebox{6in}{!}{\includegraphics{pics/APDBRegions}} +\caption{AP DB Regions and Image / Object tables} +\label{fig:APDBRegions} +\end{center} +\end{figure} + +\begin{figure} +\begin{center} +\resizebox{4.5in}{!}{\includegraphics{pics/APDB}} +\caption{AP DB components} +\label{fig:APDBRegions} +\end{center} +\end{figure} \subsubsection{AP Database servers} @@ -847,30 +894,28 @@ \end{table} -\begin{table} -\begin{center} -\caption{AP Database Tables\label{APDBTables}} -\begin{tabular}{ll} -\hline -\hline -{\bf Table Name} & {\bf Description} \\ -\hline -Region Table & spatial distribution of tables \\ -Images & The images that have objects in the DB. \\ -Image Overlaps & Image regions which are touched by specific images. \\ -Objects & The objects --- average properties of multiple detections of the same object. \\ -Average Magnitudes & Average photometry in multiple filters \\ -Detections & Detections of sources in an image. \\ -Non-Detections & Non-detections of objects in an image. \\ -Filters & Filters understood by the system. \\ -Photcodes & Transformations between different photometric systems \\ -Database Machines & computers used to store the tables \\ -\hline -\end{tabular} -\end{center} -\end{table} +\subsubsection{Notes} + +how does the AP Database know about the relationship between a +collection of chips? + +what is astrometry representation in image table? 3rd order polynomial +across the chip? + +does the AP Database know about FPA, Chip, Distortion Model, etc? I +think it probably needs to if it is going to solve for distortion +models. however, this operation may be a combination of AP DB +interaction and MD DB interaction. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% \subsection{Controller} + +\begin{figure} +\begin{center} +\resizebox{4.5in}{!}{\includegraphics{pics/Controller}} +\caption{Schematic illustration of the Controller components} +\label{fig:Controller} +\end{center} +\end{figure} The IPP uses a group of computers to store and process images and to @@ -897,4 +942,6 @@ kernel handle the I/O load. +\subsubsection{Controller Nodes} + Computers managed by the IPP Controller are allowed to be in one of several states, and the IPP Controller must interact with it in an @@ -909,15 +956,25 @@ Computers may be set to the {\tt off} or {\tt dead} states by external subsystems; it is the responsibility of the IPP Controller to return a -computer to the {\tt alive} state if possible. An example scenario: a -computer crashes. At this point the IPP Controller should detect that -the computer is no longer responsive and mark it {\tt dead}. It -should occasionally try to re-establish communication with the -computer, potentially with longer and longer delays between attempts. -A human could be notified if the computer seems to remain {\tt dead} -for a very long time. In another circumstance, a person needs to work -on a computer. They should have the ability to notify the IPP -Controller that the machine is off, perhaps with a prior notification -that the machine should be prepared to go off. Only when the person -is done working and testing the machine, and tells the IPP Controller +computer to the {\tt alive} state if possible. + +The IPP Controller must honor requests (normally from the users) to +change the mode of any computing node on demand between {\tt off} and +{\tt dead}. This would normally be done after a computer has been +rebooted and is release to the IPP Controller for its use. It must +also be able to change the list of allowed tasks as requested by +external commands. + +Two example scenarios illustrate the transition between these states. +First, imagine a computer crashes. At this point the IPP Controller +should detect that the computer is no longer responsive and mark it +{\tt dead}. It should occasionally try to re-establish communication +with the computer, potentially with longer and longer delays between +attempts. A human could be notified if the computer seems to remain +{\tt dead} for a very long time. In another scenario, a person needs +to work on a computer. They notify the IPP Controller that the +machine is off, perhaps with a prior notification that the machine +should be prepared to go off. When work on the machine is complete, +it should be placed in the {\tt dead} state. Only when the person is +done working and testing the machine, and tells the IPP Controller that the machine is now {\tt dead} can the IPP Controller attempt to re-start communications and processing on that computer. @@ -931,4 +988,76 @@ tasks to run on specific CPUs or exclude specific tasks from specific CPUs. + +The Controller maintains a table of processing nodes available to it +and the status of these Nodes. When the Controller starts, it +attempts to launch a Node Agent on each of the available processing +nodes. Modes which are not responsive are placed into an inactive +state and retried occasionally. + +\subsubsection{Controller Node Agents} + +A Node Agent runs on each of the individual nodes to perform the tasks +as directed by the Controller. The Node Agents communicate with the +Controller via a socket connection. + +A processing stage is executed in the UNIX user space, and is run as a +fork by the Node Agent. The Node Agent must monitor the standard +error and standard output of the processing stage and save them in +separate buffers. If the process dies, the Node Agent must detect the +crash. The Node Agent must respond to various commands from the +Controller, as follows: + +\paragraph{Report status} + +The Node Agent returns the state of the Node (idle, busy, done), the +state of the current processing stage (`none', `busy', `crash', +`done'), and the exit status of the current processing stage, if +available. + +The four possible states of the Node indicate that the client has no +current processing stage (`idle'), that it has a processing stage +which is still running (`busy'), or that it has a processing stage +which has completed. The last two states indicate if the current +processing stage has crashed (`crash'), or if the current processing +stage has exited gracefully (`done'). The reported exit state, if the +process has completed without crashing, is the UNIX exit state +reported by the processing stage: 0--256 with 0 indicating a +successful completion. + +\paragraph{Report stdout} + +Send and flush the current stdout buffer. The Node Agent will return +the complete contents of the stdout buffer via a buffered write and +flush the buffer when it is finished. The Node Agent will not accept +more data on the stdout buffer from the current processing stage until +the send is complete and the buffer is flushed. The daemon must +accept all of the buffer output. + +\paragraph{Report stderr} + +Identical to `report stdout', but for stderr. + +\paragraph{Kill processing stage} + +The Node Agent should send a kill signal to the current processing +stage. When the processing stage has exited, the Node Agent should +set the processing stage status to `crash' and the Node status to +`done'. + +\paragraph{Clear processing stage} + +The Node Agent should set the current processing stage state to `none' +and the Node state to `idle'. If a processing stage is currently +running, it should be killed (signal 9 or 15) before the processing +stage is cleared. + +\paragraph{Start processing stage} + +The Node Agent forks a specified command. The command should be a +standard UNIX command without command line redirection or +backgrounding. For this reason, the Node Agent must provide a layer +of security, for example, by employing SSL authentication. + +\subsubsection{Tasks} The IPP Controller accepts tasks from other IPP subsystems. The task @@ -960,4 +1089,13 @@ are maintained in the queue and never executed. +It may be useful for the Controller to distinguish between tasks +dominated by I/O and tasks dominated by data processing. It is +possible that one of each of these types of tasks may be sent to the +same node without significantly impacting the system performance. +Alternatively, it may be necessary to limit a single machine with 2 +CPUs to only one of each of these types of tasks (i.e., one processor +will be working on I/O while the other is working on processing). +Such details will be studied by the IfA IPP Team. + The IPP Controller monitors the output streams from the executing tasks and the exit status of the tasks. Each task is associated with @@ -966,4 +1104,6 @@ other subsystems may determine if specific tasks have started or completed. + +\subsubsection{External Interfaces} The IPP Controller must accept commands from other IPP subsystems. @@ -976,11 +1116,4 @@ must also be able to stop the current execution of a task and push it to the end of the queue and also change its priority. - -The IPP Controller must honor requests (normally from the users) to -change the mode of any computing node on demand between {\tt off} and -{\tt dead}. This would normally be done after a computer has been -rebooted and is release to the IPP Controller for its use. It must -also be able to change the list of allowed tasks as requested by -external commands. The IPP Controller must respond to informational requests regarding the @@ -1007,20 +1140,5 @@ Server. -It may be useful for the Controller to distinguish between tasks -dominated by I/O and tasks dominated by data processing. It is -possible that one of each of these types of tasks may be sent to the -same node without significantly impacting the system performance. -Alternatively, it may be necessary to limit a single machine with 2 -CPUs to only one of each of these types of tasks (i.e., one processor -will be working on I/O while the other is working on processing). -Such details will be studied by the IfA IPP Team. - -The Controller maintains a table of processing nodes available to it -and the status of these Nodes. When the Controller starts, it -attempts to launch a Node Agent on each of the available processing -nodes. Modes which are not responsive are placed into an inactive -state and retried occasionally. - -The Controller also maintains three tables of processing jobs: pending +The Controller maintains three tables of processing jobs: pending stages, active stages, and completed stages. The pending stages are those which have not yet been performed. The active stages are those @@ -1030,88 +1148,8 @@ clients and sends them new pending stages when they become free. -\subsubsection{Node Agents} - -A Node Agent runs on each of the individual nodes to perform the tasks -as directed by the Controller. The Node Agents communicate with the -Controller via a socket connection. - -A processing stage is executed in the UNIX user space, and is run as a -fork by the Node Agent. The Node Agent must monitor the standard -error and standard output of the processing stage and save them in -separate buffers. If the process dies, the Node Agent must detect the -crash. The Node Agent must respond to various commands from the -Controller, as follows: - -\paragraph{Report status} - -The Node Agent returns the state of the Node (idle, busy, done), the -state of the current processing stage\footnote{Note that a processing -stage is considered ``current'' until it is cleared with {\em clear -processing stage} --- even if it has crashed or completed.} (`none', -`busy', `crash', `done'), and the exit status of the current -processing stage (`none', 0--256). - -The three states of the Node indicate that the client has no current -processing stage (`idle'), that it has a processing stage which is -still running (`busy'), or that it has a processing stage which has -completed. - -The processing stage states indicate the there is no current -processing stage (`none'), that the current processing stage is -running (`busy'), that the current processing stage has crashed -(`crash'), or that the current processing stage has exited gracefully -(`done'). The exit state is the exit state reported by the processing -stage (0--256 with 0 indicating a successful completion) or is an -indication that there is no current processing stage (`none'). - -\paragraph{Report stdout} - -Send and flush the current stdout buffer. The Node Agent will return -the complete contents of the stdout buffer via a buffered write and -flush the buffer when it is finished. The Node Agent will not accept -more data on the stdout buffer from the current processing stage until -the send is complete and the buffer is flushed. The daemon must -accept all of the buffer output. - -\paragraph{Report stderr} - -Identical to `report stdout', but for stderr. - -\paragraph{Kill processing stage} - -The Node Agent should send a kill signal to the current processing -stage. When the processing stage has exited, the Node Agent should -set the processing stage status to `crash' and the Node status to -`done'. - -\paragraph{Clear processing stage} - -The Node Agent should set the current processing stage state to `none' -and the Node state to `idle'. If a processing stage is currently -running, it should be killed before the processing stage is cleared. - -\paragraph{Start processing stage} - -The Node Agent forks a specified command. The command should be a -standard UNIX command without command line redirection or -backgrounding. For this reason, the Node Agent must provide a layer -of security, for example, by employing SSL authentication. - -\subsubsection{Controller User Interface} - The IPP Controller provides a mechanism for users (either other programs or humans) to interact with it. The user interface provides commands to check the current processing job queues, the tables of successful and failed jobs, to stop or delete jobs, etc. - -\subsubsection{Notes} - -can a process send a message back to the controller before process is -complete? messages via controller? - -does the controller or the image server decide if a machine is offline -or both? - -I/O tasks vs CPU tasks? %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -1622,5 +1660,5 @@ \begin{figure} \begin{center} -\resizebox{8cm}{!}{\includegraphics{pics/phase2}} +\resizebox{6in}{!}{\includegraphics{pics/phase2}} \caption{ \label{phase2} Phase 2 dataflow} \end{center} @@ -1675,5 +1713,5 @@ \begin{figure} \begin{center} -\resizebox{8cm}{!}{\includegraphics{pics/phase3}} +\resizebox{4.5in}{!}{\includegraphics{pics/phase3}} \caption{ \label{phase3} Phase 3 dataflow} \end{center} @@ -1867,5 +1905,5 @@ \begin{figure} \begin{center} -\resizebox{8cm}{!}{\includegraphics{pics/phase4}} +\resizebox{6in}{!}{\includegraphics{pics/phase4}} \caption{ \label{phase4} Phase 4 dataflow} \end{center} @@ -2260,4 +2298,5 @@ \subsection{Image Server Database Table Contents} +\ref{ImageServerTableContents} \begin{table} @@ -2315,4 +2354,5 @@ \subsection{Metadata Database Table Contents} +\ref{MetadataTableContents} Tables \tbd{NN} -- \tbd{NN} list the basic contents of each of the @@ -2655,4 +2695,9 @@ \end{table} \clearpage + +\subsection{AP Database Table Contents} +\ref{APDBTableContents} + + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%