Index: /trunk/doc/release.2015/Makefile.Common =================================================================== --- /trunk/doc/release.2015/Makefile.Common (revision 40021) +++ /trunk/doc/release.2015/Makefile.Common (revision 40022) @@ -14,12 +14,11 @@ %.pdf: %.tex - $(MY_LATEX) $*.tex - -# $(MY_LATEX) $*.tex; if [ $? -eq 1 ]; then rm $*.pdf; exit 1; fi - -junk: - if [ $(DO_BIBTEX) -eq 1 ]; then $(BIBTEX) $*; fi - $(MY_LATEX) $*.tex; if [ $? -eq 1 ]; then rm $*.pdf; exit 1; fi - if [ $(DO_BIBTEX) -eq 1 ]; then $(MY_LATEX) $*.tex; fi +# need to remove the output pdf file if the latex fails or we cannot re-run + $(MY_LATEX) $*.tex || (rm $*.pdf; exit 1) + if [ $(DO_BIBTEX) -eq 1 ]; then $(BIBTEX) $* || (echo fail; exit 1); fi +# need to re-run latex to get the reference right + $(MY_LATEX) $*.tex || (rm $*.pdf; exit 1) +# if we are running bibtex, we need to re-run latex AGAIN to get the reference right + if [ $(DO_BIBTEX) -eq 1 ]; then $(MY_LATEX) $*.tex || (rm $*.pdf; exit 1); fi # thumbpdf --modes=dvips $*.pdf # $(MY_LATEX) $*.tex Index: /trunk/doc/release.2015/ps1.datasystem/datasystem.tex =================================================================== --- /trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 40021) +++ /trunk/doc/release.2015/ps1.datasystem/datasystem.tex (revision 40022) @@ -1082,12 +1082,12 @@ context. -One of the main purposes of the DVO system is to define the -relationship between individual detections of an astronomical object -and the definition of that object. New detections are generally added -to the database in a group associated with, for example, an image from -the GPC1 camera. As new detections are loaded, they are compared to -the objects already stored in the database. If an object is already -found in the database within the match radius of \note{one arcsecond}, the new detection is -associated to that object. If more than one object exists within the +One of the main purposes of DVO is to define the relationship between +individual detections of an astronomical object and the definition of +that object. New detections are generally added to the database in a +group associated with, for example, an image from the GPC1 camera. As +new detections are loaded, they are compared to the objects already +stored in the database. If an object is already found in the database +within the match radius of \note{one arcsecond}, the new detection is +assigned to that object. If more than one object exists within the database, the detection is associated with the closest object. @@ -1439,6 +1439,9 @@ \section{Operations and Automation} + \subsection{Pantasks and Parallel Processing} \label{sec:pantasks} + +\subsubsection{Pantasks} Sections~\ref{sec:subsystesm} \& \ref{sec:postprocessing} describe the @@ -1565,5 +1568,5 @@ prevent them from running for some reason. -\note{discuss pcontrol} +\subsubsection{pantasks passes jobs to pcontrol} Jobs which are generated by \ippprog{pantasks} may be run locally on @@ -1598,4 +1601,6 @@ them to the list of jobs to execute. It also accepts from pantasks the names of computers on which it is allowed to run those jobs. + +\subsubsection{pcontrol passes jobs to pclient} When pcontrol is provided with the name of a computer, it will attempt @@ -1638,12 +1643,15 @@ behavior depending on the exit code from the job. -\note{discuss pclient} - -\begin{verbatim} -* pantasks_client vs pantasks_server -* opihi scripting language -* task options (show example?) - -\end{verbatim} +The \ippprog{pantasks} program can be run as a stand-alone program +which presents an opihi shell interface to the user when it is +started. This mode is useful for testing as all errors are reported +back to the opihi shell. However, when the user exits the shell, the +pantasks instance exits, shutting down pcontrol and all remote client +connections. In standard operations, pantasks is run in a client +server mode. The server runs continuously in the background and +multiple users may connect via the \ippprog{pantasks_client} program. +Users can the send commands to the server to load scripts, add +parallel hosts, check status, and start or stop the pantasks +operations. \begin{figure} @@ -1655,129 +1663,182 @@ periods -exec 5.0 npending 1 - stdout NULL - stderr /data/local/example.task + stdout /data/local/example.task.output + stderr /data/local/example.task.errors end \end{verbatim} \caption{\label{fig:simple.static.task} Example of a simple static - task in the opihi-based scripting language used by pantasks. } + task in the opihi-based scripting language used by pantasks. In + this example, pantasks would run a single instance of the command + \code{ls /tmp} every 5 seconds, sending the stdout and stderr to + the listed files. } \end{center} \end{figure} -\note{this section needs to be re-written : pclient vs pcontrol vs pantasks} - -The actual processing of data is managed by the \ippprog{pantasks} -program. This program launches a series of connections to other -computers in the IPP compute cluster, and launches instances of the -\ippprog{pclient} program that wait for jobs to be submitted. These -jobs are generated from \ippmisc{ippTasks} command scripts that each -contain one set of tasks, written in the opihi language shared with -DVO. These task scripts are generally sorted by the IPP stage. The -tasks listed in these scripts are mainly written in the form of a pair -of \ippmisc{load} and \ippmisc{run} tasks. - -The \ippmisc{load} task for a particular stage queries the processing -database via an appropriate \ippmisc{ippTool} (see section \ref{sec:ipptools} below) for a list of jobs that are waiting to be run. -This task is executed on the host running the \ippprog{pantasks} -server, and only one of each type of \ippmisc{load} task is permitted to -run simultaneously, preventing race conditions. To manage the -processing, the primary database tables for each stage contain a -\ippdbcolumn{label} field. This field groups data into categories, -and the appropriate list of \ippdbcolumn{label}s are supplied to the -\ippprog{pantasks} program to be included in the queries to ensure -that only jobs for the active \ippdbcolumn{label}s are executed. The -results from the database query are stored in \ippmisc{books} within the -\ippprog{pantasks} environment, with each resulting result row saved -to a separate entry within the \ippmisc{book}. These \ippmisc{books} are -similar to an associative array indexing the entries (\ippmisc{pages} to -continue the analogy) to be accessed via a particular key. Keys for -most stages are a combination of the stage id and an identifier for -the individual component for the job that will be executed. The rest -of the column results from the query are stored to the \ippmisc{page}, -with each field stored with the database column name. An additional -field is added, the \ippdbcolumn{pantasksState}, which is set to -\ippmisc{INIT}, indicating that this \ippmisc{page} is a new addition to the -\ippmisc{book}. Once all results from the database query have been -saved, the \ippmisc{load} task terminates, and waits to be rerun after -some timeout (on the order of a minute). Once all new \ippmisc{pages} -have been added, the task then scans the \ippmisc{book} for any pages -with \ippdbcolumn{pantasksState} set to \ippmisc{DONE}, and removes them -from the book, as these represent jobs that have finished. - -Parallel to the \ippmisc{load} task, the \ippmisc{run} tasks are executed -by the \ippprog{pclient} code on one of the client nodes. These -clients can access the \ippmisc{books} from \ippprog{pantasks}, and scan -the \ippmisc{book} to determine if it contains \ippmisc{pages}, and if one -of those pages contains an entry with the \ippdbcolumn{pantasksState} -set to \ippmisc{INIT}. If such a \ippmisc{page} is found, the task sets -this field to \ippmisc{RUN} (ensuring no other \ippmisc{run} task will -attempt to process that page as well), and uses the information stored -on the page to construct a valid command line for the appropriate IPP -script. The script (described below) command line is then executed, -and upon completion, pass the exit code back to \ippprog{pantasks}, -allowing for an accounting of success and failures. The page is then -set to \ippmisc{DONE}, indicating the job has completed, and the -\ippmisc{page} can be removed from the \ippmisc{book} by the next call of -the \ippmisc{load} task. As the \ippmisc{run} tasks will involve processing -that may not complete in a known period of time, there is a timeout -limit set for the task, after which the \ippprog{pantasks} process can -terminate the task. This prevents unproductive jobs from building up. - -In addition to these tasks, most stages have \ippmisc{revert} tasks -paired with the \ippmisc{run} task. These tasks issue an appropriate -\ippmisc{ippTool} command to remove entries from the stage's secondary -database table that have the \ippdbcolumn{fault} field set to a value -other than zero. This step allows these failures to be cleared from -the system, and schedule those jobs again for a retry. Again, this -query is limited by the set of active \ippdbcolumn{label}s, ensuring -that only the data currently being processed is reverted. Similarly, -some stages have \ippmisc{advance} tasks that update the primary table to -indicate that all of its components are complete. For many of the -early stages of the pipeline (the \ippstage{chip} through -\ippstage{warp} stages), this \ippmisc{advance} task also adds an entry -into the database table for the next stage of processing for the -exposure being considered. This step allows the data to process +\subsubsection{Pantasks scripts: ippTasks} + +Pantasks provides an environment in which commands can be generated +and extensive parallel processing managed. The details of how to +implement the different stages of IPP processing are captured in a +collection of scripts written for pantasks in the \code{opihi} +language. In general, each stage is defined by an associated script +collected together under the \ippmisc{ippTasks} collection. While +each script has its own details, there are a number of common +elements. + +Most stages consist of two related tasks: a \ippmisc{load} task, which +is responsible to querying the processing database to identify entries +to be processed, and a \ippmisc{run} task, which is responsible for +managing the processing of the individual entries. + +The \ippmisc{load} task for a particular stage generates +\ippmisc{load} jobs which query the processing database via a +dedicated database interface program (see the discussion of +\ippmisc{ippTool} in section \ref{sec:ipptools} below) for a list of +processing stage entries that are waiting to be run. The +\ippmisc{load} jobs are executed on the host running the +\ippprog{pantasks} server. Only one of each type of \ippmisc{load} +job is permitted to run simultaneously, preventing race conditions. + +The results from the database query job are stored in an opihi data +structure called a \ippmisc{book} within the \ippprog{pantasks} +environment. Each row in the result set is saved to a separate entry +within the \ippmisc{book}. These \ippmisc{books} are a hierarchical +associative array indexing the entries (\ippmisc{pages} to continue +the analogy) to be accessed via a particular key. Keys for most +stages are a combination of the stage id and an identifier for the +individual component for the job that will be executed. For a given +row in the result set, each column in the row is stored as a separate +line on the \ippmisc{page}, identified by the database column name. An +additional line, the \ippdbcolumn{pantasksState}, is added so pantasks +can manage the processing of the job which will be generated by this +page. When the page is first generate, the +\ippdbcolumn{pantasksState} is set to \ippmisc{INIT}, indicating that +this \ippmisc{page} is a new addition to the \ippmisc{book}. Once all +new \ippmisc{pages} have been added, the task then scans the +\ippmisc{book} for any pages with \ippdbcolumn{pantasksState} set to +\ippmisc{DONE}, and removes them from the book, as these represent +jobs that have finished. \note{the manipulation above takes place in + the task.exit subscript} + +The associated \ippmisc{run} task generates jobs constructed from the +collection of pages in the book. The task examines the book and +selects the first available page with \ippdbcolumn{pantasksState} +of \ippmisc{INIT}. The task uses the information in the page to +construct the appropriate command-line (e.g., lines in the page may +include input file names and output file names for the specific item +in the database). The resulting command becomes a job in the pantasks +collection of jobs. Most IPP analysis stages specify that the jobs +are then sent to pcontrol for parallel process. Before task generates +the job, the \ippdbcolumn{pantasksState} is set to \ippmisc{RUN} so a +future execution of the task will not attempt to re-run this specific job. + +Upon completion of the job, it is necessary to update the processing +database with the results, specifically indicating in the database +that the job has completed and if was successful. Within the IPP, +this responsibility is left to the program which ran the analysis. +IPP analysis steps normally consist of two main elements: a C-language +program to do the data analysis work and a supporting perl script +which performs the database update upon completion. Upon completion, +the pantasks \ippmisc{RUN} tasks is responsible for updating the +status within the book, but not within the processing database. This +split keeps the interactions at the pantasks level relatively light, +leaving the overhead of the database interaction within the job +running on one of the computing machines in the cluster. + +In addition to these tasks, most stages have a \ippmisc{revert} task +paired with the \ippmisc{run} task. These tasks run infrequently and +generate jobs which perform an operation on the processing database to +clear jobs which have failed with one of the ephemeral failure modes +(see the discussion in Section~\ref{sec:processing.database}). This +step allows these failures to be cleared from the system, and schedule +those jobs again for a retry. + +Similarly, some stages have \ippmisc{advance} tasks that update the +primary table to indicate that all of its components are complete. +For many of the early stages of the pipeline (the \ippstage{chip} +through \ippstage{warp} stages), this \ippmisc{advance} task also adds +an entry into the database table for the next stage of processing for +the exposure being considered. This step allows the data to process automatically from stage to stage without intervention. -\subsubsection{Stage automation} +The IPP processing database is used to manage all versions of an +analysis for all analysis stages. In addition to the regular +processing of the nightly data products, there may be large-scale +re-processing analysis tasks or tests of various kinds. It may be +necessary for a test analysis of a particular item to use a different +version of the processing software from the regular nightly analysis +(for example, when testing a new algorithm for release). A mechanism +is needed to manage these different processing attempts of the same +items. With the IPP, this is accomplished with an extra field, +\ippmisc{label}, for each processing stage. Within the +\ippmisc{load}, \ippmisc{revert}, and \ippmisc{advance} tasks +discussed above, the query to the processing database for new items is +restricted to a set of user-defined labels. A given instance of +pantasks will be supplied a set of labels which are then applied to +all tasks managed by that pantasks. For example, the pantasks which +manages the nightly processing of the basic science analysis stages +(chip - warp, stack, diff) is supplied with several labels which +correspond to the different kinds of observations being performed. In +this way, the analysis of the nightly observations is kept separate +from other processing attempts. + + + +\subsection{Stage automation} \label{sec:automation} -\note{I'm not convinced this is the right place for it, but it felt like a natural extension of the ``advance''}. - -\note{wording..} Beyond the warp stage, there is no longer a single -``next'' stage into which data can be queued. Because of this, more -sophisticated methods are used to advance the data. For processing -data that is actively being observed at the summit, this is handled by -a set of ``nightly science'' tasks and an associated -\ippmisc{ippScript}. The goal of these tasks is to ensure that -exposures are correctly paired into sets of -\ippstage{warp}-\ippstage{warp} difference images, and that nightly -stacks are generated for MD fields. - -Queuing the diffs is done by first examining the set of all exposures -that have been taken at the summit on the current night of observing, -and querying information from each stage up through \ippstage{warp} -stage. These exposures are grouped by \ippdbcolumn{filter} and -\ippdbcolumn{object}, which is a unique identifier for each telescope -pointing on the sky. Exposures in each group are then sorted by -increasing observation date (\ippdbcolumn{dateobs}), and the -information from each stage is used to ensure that the observations -have completed all stages through \ippstage{warp} successfully. If -this is not the case, the script ignores that group until a future -scan, to allow the data to finish processing. If the lack of -completion is due to a failure at a stage prior to \ippstage{warp}, -then that exposure is dropped from the group to allow the remainder to -be considered for \ippstage{diff}s. The remaining exposures are then -paired sequentially, with the final exposure ignored in the case of an -odd number of exposures (caused by either a failure in the processing -of an exposure or due to the telescope closing for weather prior to -completing all scheduled observations). Once the observing process -completes (signalled to the nightly science task and script by the -presence in the database of entries for dark exposures that are taken -each morning after the telescope closes), and the script has generated -all \ippstage{diff} pairs that can be made with the above rule, a -second pass is performed, this time with the exposures in each group -sorted by decreasing observation date. This change in ordering allows -exposures that were excluded due to an odd number of exposures to be -paired with the exposure closest in time (with the exposure that was -previously first ignored). + +\note{start with a discussion of the standard sequencing (end-stage)} + +\note{then discuss the addstar sequences with manual triggering} + +Outside of the basic sequence of chip to warp, there is no single +natural next step. For example: a stack can be generated with any +number of input warps; a difference image can be generated between a +warp and any one of many other warps or stacks. Without a single +sequence, more complex and sophisticated decisions much be made. + +For nightly processing of data obtained at the summit, this is handled +by a set of ``nightly science'' tasks and an associated +\ippmisc{ippScript}. These scripts have a well-defined and restricted +set of goals: to ensure that difference images are generated for each +exposures (either by pairing together warps or pairs warps with +pre-defined stacks), that nightly stacks are generated for MD fields, +and that the stacks are also differenced against an appropriate +reference. + +Pairing warps together is simplified by the observing strategy in +which the same pointing is observed multiple times in a night. By +limiting to warp-warp pairs from the same pointing, the problem is +significantly reduced from the arbitrary case. + +{\em Queuing the diffs} is done by first examining the set of all +exposures that have been taken at the summit on the current night of +observing, and querying information from each stage up through +\ippstage{warp} stage. These exposures are grouped by +\ippdbcolumn{filter} and \ippdbcolumn{object}, which is a unique +identifier for each telescope pointing on the sky. Exposures in each +group are then sorted by increasing observation date +(\ippdbcolumn{dateobs}). The database results for each stage +(chip-warp) are checked to ensure that the selected exposures have +been successfully processed for all stages through \ippstage{warp}. +Exposure groups are ignored until all exposures have either been +processed through warp or have failed with a bad quality, meaning the +exposure (or portion) cannot be processed. Failed exposures are +rejected. The remaining exposures are then paired sequentially, with +the final exposure ignored in the case of an odd number of accepted +exposures. Exposures paired in this way are sent to the +ippstage{diff} analysis stage. + +Once observations have been completed for the night (signalled by the +end-of-night dark exposures that are taken each morning after the +telescope closes), and the script has generated all \ippstage{diff} +pairs that can be made with the above rules, a second pass is +performed, this time with the exposures in each group sorted by +decreasing observation date. This change in ordering allows exposures +that were excluded due to an odd number of exposures to be paired with +the exposure closest in time (with the exposure that was previously +first ignored). Exposure pairs in which at least one exposures does +not have a pre-existing difference image are queued for difference +image analysis. The nightly stacks are queued based on checking that a minimum number @@ -1788,14 +1849,19 @@ \ippstage{stack} database entries can be queued with the \ippstage{warp} entries. Again, failures and weather can reduce the -number of usable exposures, so the task generates \ippstage{stack} -entries using whatever exposures are available if one has not yet been -constructed by the time the morning dark exposures are registered into -the database. \note{wording} - -Automating the nightly processing is important, as it ensures that -data is processed as soon as it is downloaded from the summit, -reducing the lag between an observation and the reduced data. The -other processing task that requires automation is the reprocessing of -the entire $3\pi$ survey, as the size of the dataset make it +number of usable exposures. If no stack could be made for a given MD +field with the minimum number of inputs by the time of the +end-of-night darks, stacks are generated using using whatever +exposures are available. + +The automatic nightly processing ensures that data is processed as +soon as it is downloaded from the summit, reducing the lag between an +observation and the reduced data. \note{some numbers here about + completion times and such? Words about getting data to MOPS and SN + transient folks} + +\note{re-read paragraph below and cleanup} + +The other processing task that requires automation is the reprocessing +of the entire $3\pi$ survey, as the size of the dataset make it challenging to do manually. To manage this, the ``large area processing'' (LAP) task and script are used. The first stage of this @@ -1823,69 +1889,128 @@ \ippdbcolumn{projection\_cell}. + \subsection{Nebulous} \label{sec:nebulous} + +\subsubsection{Motivation and Concept} A major concern recognized early in the Pan-STARRS project is the challenge of storing and managing the large volume of data that is generated by the GPC1 camera. The \ippprog{Nebulous} system was -designed to organize this data. The main components of this system -are a database storing the locations of the files, with a Simple -Object Access Protocol (SOAP) interface between the database and the -other IPP programs \note{define / mention http}. The actual files are -stored on a collection of computers with substantial disk partitions -in the IPP cluster, shared within the cluster via NFS. This -distribution of files is useful to balance the disk I/O, as this -parallelizes the load. - -The original design of \ippprog{Nebulous} was intended to aid in the -targetted processing of data, by having specific image data (such as -all the images from one OTA device) located on a single storage node. -This would allow any jobs involving that data to be assigned to the -storage node, eliminating network I/O. Important data could be -duplicated to additional data nodes, with the alternate locations -stored in the database. In practice, however, hardware failures and -increases in hard drive storage volumes and network bandwidth have -reduced the degree to which the IPP processing is targeted. - -\note{this is a critical paragraph and needs to be re-written to be - more accessible} -When a program creates a new file in \ippprog{Nebulous}, it supplies -an URI of the form \code{neb://HOST.VOLUME/PATH/FILENAME}. The host -and volume specifiers are optional, and allow a file to be created on -a specific node. The path and filename appear similar to a standard -full file location, and are used internally as the ``external id''. -\note{mention the nebulous schema before this?} A storage object -entry is then created in the database for this id, and an instance of -the file created on the specified node (or at random from available -nodes if left empty). This instance is created in a deterministic -filename location. The external id is hashed using the SHA-1 -function, and the first four hexadecimal digits of this hash are -separated into two two-digit strings and used as the top and second -level directory location for the disk file. The disk file is created -using the database instance id, and a transformed version of the -external id, which has colons replacing any forward slash characters, -to convert the external id path into a filename. For the example URI +designed to aid in thie process. \ippprog{Nebulous} is not a file +system per-se, but only method of tracking the locations of files +within the file system, and of tracking duplicate copies of the same +file. The core of \ippprog{Nebulous} is a dedicated database engine +which tracks ``storage objects'', the concept of a file exists in the +system. Each storage object may be associated with a number of copies +of the actual files on the disks in the storage system (called +``instances''), which are also recorded by the database. In the IPP +cluster, the file instances are stored on a collection of computers +with substantial disk partitions shared via NFS. + +\ippprog{Nebulous} also explicitly tracks the different computers on +which the file instances are stored. This allows the system to expose +files to the user only on machines which are currently active in +\ippprog{Nebulous}. If, for example, a storage computer crashes or +needs to be taken offline, the machine can be made unavailable in +\ippprog{Nebulous}, in which case only instances on other machines +will be supplied to users. + +This localization is also useful for allowing the IPP processing to +target processing to computers based on the location of the data. For +example, all raw images from a specific chip in the camera could be +stored on a specific computer (for at least one of the instances). +All of the analysis stages which interact with that chip could then be +preferentially targetted to be run on that computer. The localization +in \ippprog{Nebulous} and the host targetted processing in pantasks +can therefore work together to encourage processing to require only +local disk access, reducing the I/O local on the network +infrastructure. In the early stages of the Pan-STARRS project, this +was important because network bandwidth was an expensive resource. In +practice, the as-built IPP has had sufficient network bandwidth that +this targetting was not required. In practice, due to the timing of +hardware aquisition, occasional hardware failures, and other +organizational details, targetted processing has only been used to a +moderate degree within the Pan-STARRS cluster. \note{can we get a + number here?} + +\subsubsection{Implementation Details} + +The user interfaces to Nebulous consist of command-line programs as +well as APIs in both C and Perl. The basic user commands to interact +with Nebulous are to 1) create a new storage object and associated +instance; 2) add a new instance to an existing storage object; 3) +remove (cull) an instance; 4) delete a storage object; and 5) find a +file associated with a given storage objects. Note that these user +commands do not affect the files on disk \note{true for cull?} +(exception: the create function will create an empty file if one does +not exist). They only change the state of the Nebulous database; it +is the responsibility of the user program to read and write data to a +file and to create the copies, etc. + +For the Nebulous users, the identifier of a storage object is a unique +string with the form of a UNIX file path: e.g. a/b/c/file. When a +program creates a new file in \ippprog{Nebulous}, it supplies a URI of +the form \code{neb://HOST.VOLUME/PATH/FILE}. The host and volume +specifiers are optional, and allow a file to be created on a specific +node. The path and filename portions become the identifier and are +recorded in the \ippmisc{storage_object} table in the +\ippmisc{extern_id} field. A storage object entry is then created in +the database for this id, and an instance of the file created on the +specified node (or at random from available nodes if left empty). + +Files are stored on specific computers in a \ippprog{Nebulous} +directory or directories on that computer. In the IPP system, the +top-level Nebulous directories are usually placed at the root of the +storage device as mounted on the machine, in a subdirectory named +\code{nebulous}. Beneath the top-level directory are 256 +subdirectories with names of the form 00 - ff (i.e., 2 digit +hexadecimate number). Each subdirectory again as 256 subdirectories +with the same naming scheme. + +The filename of an instance in Nebulous is deterministic and derived +from the \ippmisc{extern_id}: the \ippmisc{extern_id} is hashed using +the SHA-1 function, and the first four hexadecimal digits of this hash +are separated into two two-digit strings and used as the top and +second level directory location for the disk file. The instance table +in the Nebulous database includes a unique auto-incrementing index to +provide a unique SQL ID for each instance. Under the subdirectory +identified above, the disk file name is by appending the database +instance id with a string derived from the \code{extern_id}: forward +slash characters are replaced in the name with colons so the string +can represent a file in the UNIX filesystem. For the example URI above, this results in a file located on disk in a location like -\code{/data/HOST.VOLUME/nebulous/d5/d8/9244993440.PATH:FILENAME}. -This file naming structure is useful, as it duplicates database -contents on disk. \note{rephrase} - -The storage volumes that contain the data on disk are defined in the -\ippprog{Nebulous} database in a number of ways. First, the locations -and mount points for the actual NFS storage are listed in the +\code{/data/HOST.VOLUME/nebulous/d5/d8/9876.PATH:FILE}. +This file naming structure has the benefit of providing redundancy +between the filename on disk and the instance in the database. + +Nebulous tracks additional information beyond just the storage objects +and the associated instances. As mentioned above, the storage volumes +are tracked to provide a link between a top level nebulous directory +and the computer which contains that directory. The locations and +mount points for the actual NFS storage are listed in the \ippdbtable{volume} table. This table contains columns indicating if the volume should be used for reading (\ippdbcolumn{available}) and -writing (\ippdbcolumn{allocate}), allowing these properties to be -manually set, which is useful in scheduling downtime for hardware -issues. Another column, \ippdbcolumn{xattr}, is used to control the -behavior of this volume, with specific values used to denote desired -behavior. For instance, the value of $1$ is used to indicate that a -volume should only be used as a backup volume (which accepts only -replicated copies), and the value of $5$ is used to indicate that the -volume is permanently unavailable, and should be ignored. \note{more - detail, more specific} - -In addition to this permanent table describing the volumes, a -\ippdbtable{mountedvol} table also exists that only lists those +writing (\ippdbcolumn{allocate}). As noted above, Nebulous will not +return a file to the user if the storage volume is marked as not +\ippdbcolumn{available}. If a storage volume is marked as not to be +\ippdbcolumn{allocate}ed, then new storage objects will not generate +instances on that volume, but existing instances may be supplied to +the user. + +\note{we care about the concepts here, but not the numbers. reword} +Another column, \ippdbcolumn{xattr}, is used to control the behavior +of this volume, with specific values used to denote desired behavior. +For instance, the volume may be marked to be used only for backup, in +which case it will not be used to store an instance by default, but +will be used preferentially if an instance noted as a backup when it +is generated. Alternatively, a volume may be marked as permanently +unavailable, and should be ignored in most contexts. This latter +option allows the system to retain the memory of hardware which has +been retired (and potentially to retain information about instances +which were previously on such machines). + +In addition to the static table describing the volumes, a second +dynamically-generated table, \ippdbtable{mountedvol}, lists those volumes that are currently visible and accessible from the \ippprog{Nebulous} database server. This table also lists the total @@ -1897,15 +2022,23 @@ \ippdbtable{volume} table. -The final table controlling the operations of the \ippprog{Nebulous} -volumes is the \ippdbtable{cabinet} table, which organizes the -individual volumes into ``cabinets,'' a concept loosely based on the -physical arrangement of the storage servers in the data center. These -cabinets are used to prevent the replication of a storage object -within a group of volumes where all instances of the object could be -taken off line by a single failure. Since servers within a given -cabinet in the data center share a common set of PDUs \note{define}, -it is important to ensure physical distance between replicated copies -to guarantee that a temporary failure of one of the cabinet PDUs does -not significantly impact processing. +The \ippdbtable{cabinet} table organizes the individual volumes into +``cabinets,'' a concept loosely based on the physical arrangement of +the storage servers in the data center. These cabinets are used to +prevent the replication of a storage object within a group of volumes +where all instances of the object could be taken off line by a single +failure. Since servers within a given cabinet in the data center +share a common set of power delivery units (PDUs), it is important to +ensure physical distance between replicated copies to guarantee that a +temporary failure of one of the cabinet PDUs does not significantly +impact processing. + +The nebulous user APIs do not interact directly with the nebulous +mysql database. Instead, they interact with one of several computers +with an Apache web server. Interactions with the Apache server are +performed using the Simple Object Access Protocol (SOAP) interface, +while the Apache servers interact directly with the Mysql database +server. This architecture avoids the overhead of setting up and +tearing down the Mysql connection for each Nebulous command, instead +using only the low-latency SOAP communications. \note{need a paragraph or two on stats: how many objects, how many