Changeset 40022

Timestamp:

Apr 30, 2017, 6:23:26 AM (9 years ago)

Author:

eugene

Message:

extensive wordsmithing

Location:

trunk/doc/release.2015

Files:

• Makefile.Common (modified) (1 diff)
• ps1.datasystem/datasystem.tex (modified) (9 diffs)

trunk/doc/release.2015/Makefile.Common

@@ -14 +14 @@
 
 %.pdf: %.tex
-        $(MY_LATEX) $*.tex
-
-#       $(MY_LATEX) $*.tex; if [ $? -eq 1 ]; then rm $*.pdf; exit 1; fi
-
-junk:
-        if [ $(DO_BIBTEX) -eq 1 ]; then $(BIBTEX) $*; fi
-        $(MY_LATEX) $*.tex; if [ $? -eq 1 ]; then rm $*.pdf; exit 1; fi
-        if [ $(DO_BIBTEX) -eq 1  ]; then $(MY_LATEX) $*.tex; fi
+#       need to remove the output pdf file if the latex fails or we cannot re-run
+        $(MY_LATEX) $*.tex || (rm $*.pdf; exit 1)
+        if [ $(DO_BIBTEX) -eq 1 ]; then $(BIBTEX) $* || (echo fail; exit 1); fi
+#       need to re-run latex to get the reference right
+        $(MY_LATEX) $*.tex || (rm $*.pdf; exit 1)
+#       if we are running bibtex, we need to re-run latex AGAIN to get the reference right
+        if [ $(DO_BIBTEX) -eq 1  ]; then $(MY_LATEX) $*.tex || (rm $*.pdf; exit 1); fi
 #       thumbpdf --modes=dvips $*.pdf
 #       $(MY_LATEX) $*.tex 

trunk/doc/release.2015/ps1.datasystem/datasystem.tex

@@ -1082 +1082 @@
 context.  
 
-One of the main purposes of the DVO system is to define the
-relationship between individual detections of an astronomical object
-and the definition of that object.  New detections are generally added
-to the database in a group associated with, for example, an image from
-the GPC1 camera.  As new detections are loaded, they are compared to
-the objects already stored in the database.  If an object is already
-found in the database within the match radius of \note{one arcsecond}, the new detection is
-associated to that object. If more than one object exists within the
+One of the main purposes of DVO is to define the relationship between
+individual detections of an astronomical object and the definition of
+that object.  New detections are generally added to the database in a
+group associated with, for example, an image from the GPC1 camera.  As
+new detections are loaded, they are compared to the objects already
+stored in the database.  If an object is already found in the database
+within the match radius of \note{one arcsecond}, the new detection is
+assigned to that object. If more than one object exists within the
 database, the detection is associated with the closest object.
 
@@ -1439 +1439 @@
 \section{Operations and Automation}
 
+
 \subsection{Pantasks and Parallel Processing}
 \label{sec:pantasks}
+
+\subsubsection{Pantasks}
 
 Sections~\ref{sec:subsystesm} \& \ref{sec:postprocessing} describe the
@@ -1565 +1568 @@
 prevent them from running for some reason.
 
-\note{discuss pcontrol}
+\subsubsection{pantasks passes jobs to pcontrol}
 
 Jobs which are generated by \ippprog{pantasks} may be run locally on
@@ -1598 +1601 @@
 them to the list of jobs to execute.  It also accepts from pantasks
 the names of computers on which it is allowed to run those jobs.
+
+\subsubsection{pcontrol passes jobs to pclient}
 
 When pcontrol is provided with the name of a computer, it will attempt
@@ -1638 +1643 @@
 behavior depending on the exit code from the job.  
 
-\note{discuss pclient}
-
-\begin{verbatim}
-* pantasks_client vs pantasks_server
-* opihi scripting language
-* task options (show example?)
-
-\end{verbatim}
+The \ippprog{pantasks} program can be run as a stand-alone program
+which presents an opihi shell interface to the user when it is
+started.  This mode is useful for testing as all errors are reported
+back to the opihi shell.  However, when the user exits the shell, the
+pantasks instance exits, shutting down pcontrol and all remote client
+connections.  In standard operations, pantasks is run in a client
+server mode.  The server runs continuously in the background and
+multiple users may connect via the \ippprog{pantasks_client} program.
+Users can the send commands to the server to load scripts, add
+parallel hosts, check status, and start or stop the pantasks
+operations. 
 
 \begin{figure}
@@ -1655 +1663 @@
   periods  -exec 5.0
   npending 1
-  stdout   NULL
-  stderr   /data/local/example.task
+  stdout   /data/local/example.task.output
+  stderr   /data/local/example.task.errors
 end  
 \end{verbatim}
  \caption{\label{fig:simple.static.task} Example of a simple static
-   task in the opihi-based scripting language used by pantasks. }
+   task in the opihi-based scripting language used by pantasks.  In
+   this example, pantasks would run a single instance of the command
+   \code{ls /tmp} every 5 seconds, sending the stdout and stderr to
+   the listed files. }
   \end{center}
 \end{figure}
 
-\note{this section needs to be re-written : pclient vs pcontrol vs pantasks}
-
-The actual processing of data is managed by the \ippprog{pantasks}
-program.  This program launches a series of connections to other
-computers in the IPP compute cluster, and launches instances of the
-\ippprog{pclient} program that wait for jobs to be submitted.  These
-jobs are generated from \ippmisc{ippTasks} command scripts that each
-contain one set of tasks, written in the opihi language shared with
-DVO.  These task scripts are generally sorted by the IPP stage.  The
-tasks listed in these scripts are mainly written in the form of a pair
-of \ippmisc{load} and \ippmisc{run} tasks.
-
-The \ippmisc{load} task for a particular stage queries the processing
-database via an appropriate \ippmisc{ippTool} (see section \ref{sec:ipptools} below) for a list of jobs that are waiting to be run.
-This task is executed on the host running the \ippprog{pantasks}
-server, and only one of each type of \ippmisc{load} task is permitted to
-run simultaneously, preventing race conditions.  To manage the
-processing, the primary database tables for each stage contain a
-\ippdbcolumn{label} field.  This field groups data into categories,
-and the appropriate list of \ippdbcolumn{label}s are supplied to the
-\ippprog{pantasks} program to be included in the queries to ensure
-that only jobs for the active \ippdbcolumn{label}s are executed.  The
-results from the database query are stored in \ippmisc{books} within the
-\ippprog{pantasks} environment, with each resulting result row saved
-to a separate entry within the \ippmisc{book}.  These \ippmisc{books} are
-similar to an associative array indexing the entries (\ippmisc{pages} to
-continue the analogy) to be accessed via a particular key.  Keys for
-most stages are a combination of the stage id and an identifier for
-the individual component for the job that will be executed.  The rest
-of the column results from the query are stored to the \ippmisc{page},
-with each field stored with the database column name.  An additional
-field is added, the \ippdbcolumn{pantasksState}, which is set to
-\ippmisc{INIT}, indicating that this \ippmisc{page} is a new addition to the
-\ippmisc{book}.  Once all results from the database query have been
-saved, the \ippmisc{load} task terminates, and waits to be rerun after
-some timeout (on the order of a minute).  Once all new \ippmisc{pages}
-have been added, the task then scans the \ippmisc{book} for any pages
-with \ippdbcolumn{pantasksState} set to \ippmisc{DONE}, and removes them
-from the book, as these represent jobs that have finished.
-
-Parallel to the \ippmisc{load} task, the \ippmisc{run} tasks are executed
-by the \ippprog{pclient} code on one of the client nodes.  These
-clients can access the \ippmisc{books} from \ippprog{pantasks}, and scan
-the \ippmisc{book} to determine if it contains \ippmisc{pages}, and if one
-of those pages contains an entry with the \ippdbcolumn{pantasksState}
-set to \ippmisc{INIT}.  If such a \ippmisc{page} is found, the task sets
-this field to \ippmisc{RUN} (ensuring no other \ippmisc{run} task will
-attempt to process that page as well), and uses the information stored
-on the page to construct a valid command line for the appropriate IPP
-script.  The script (described below) command line is then executed,
-and upon completion, pass the exit code back to \ippprog{pantasks},
-allowing for an accounting of success and failures.  The page is then
-set to \ippmisc{DONE}, indicating the job has completed, and the
-\ippmisc{page} can be removed from the \ippmisc{book} by the next call of
-the \ippmisc{load} task.  As the \ippmisc{run} tasks will involve processing
-that may not complete in a known period of time, there is a timeout
-limit set for the task, after which the \ippprog{pantasks} process can
-terminate the task.  This prevents unproductive jobs from building up.
-
-In addition to these tasks, most stages have \ippmisc{revert} tasks
-paired with the \ippmisc{run} task.  These tasks issue an appropriate
-\ippmisc{ippTool} command to remove entries from the stage's secondary
-database table that have the \ippdbcolumn{fault} field set to a value
-other than zero.  This step allows these failures to be cleared from
-the system, and schedule those jobs again for a retry.  Again, this
-query is limited by the set of active \ippdbcolumn{label}s, ensuring
-that only the data currently being processed is reverted.  Similarly,
-some stages have \ippmisc{advance} tasks that update the primary table to
-indicate that all of its components are complete.  For many of the
-early stages of the pipeline (the \ippstage{chip} through
-\ippstage{warp} stages), this \ippmisc{advance} task also adds an entry
-into the database table for the next stage of processing for the
-exposure being considered.  This step allows the data to process
+\subsubsection{Pantasks scripts: ippTasks}
+
+Pantasks provides an environment in which commands can be generated
+and extensive parallel processing managed.  The details of how to
+implement the different stages of IPP processing are captured in a
+collection of scripts written for pantasks in the \code{opihi}
+language.  In general, each stage is defined by an associated script
+collected together under the \ippmisc{ippTasks} collection.  While
+each script has its own details, there are a number of common
+elements.
+
+Most stages consist of two related tasks: a \ippmisc{load} task, which
+is responsible to querying the processing database to identify entries
+to be processed, and a \ippmisc{run} task, which is responsible for
+managing the processing of the individual entries.
+
+The \ippmisc{load} task for a particular stage generates
+\ippmisc{load} jobs which query the processing database via a
+dedicated database interface program (see the discussion of
+\ippmisc{ippTool} in section \ref{sec:ipptools} below) for a list of
+processing stage entries that are waiting to be run.  The
+\ippmisc{load} jobs are executed on the host running the
+\ippprog{pantasks} server.  Only one of each type of \ippmisc{load}
+job is permitted to run simultaneously, preventing race conditions.
+
+The results from the database query job are stored in an opihi data
+structure called a \ippmisc{book} within the \ippprog{pantasks}
+environment.  Each row in the result set is saved to a separate entry
+within the \ippmisc{book}.  These \ippmisc{books} are a hierarchical
+associative array indexing the entries (\ippmisc{pages} to continue
+the analogy) to be accessed via a particular key.  Keys for most
+stages are a combination of the stage id and an identifier for the
+individual component for the job that will be executed.  For a given
+row in the result set, each column in the row is stored as a separate
+line on the \ippmisc{page}, identified by the database column name.  An
+additional line, the \ippdbcolumn{pantasksState}, is added so pantasks
+can manage the processing of the job which will be generated by this
+page.  When the page is first generate, the
+\ippdbcolumn{pantasksState} is set to \ippmisc{INIT}, indicating that
+this \ippmisc{page} is a new addition to the \ippmisc{book}.  Once all
+new \ippmisc{pages} have been added, the task then scans the
+\ippmisc{book} for any pages with \ippdbcolumn{pantasksState} set to
+\ippmisc{DONE}, and removes them from the book, as these represent
+jobs that have finished. \note{the manipulation above takes place in
+  the task.exit subscript}
+
+The associated \ippmisc{run} task generates jobs constructed from the
+collection of pages in the book.  The task examines the book and
+selects the first available page with \ippdbcolumn{pantasksState}
+of \ippmisc{INIT}.  The task uses the information in the page to
+construct the appropriate command-line (e.g., lines in the page may
+include input file names and output file names for the specific item
+in the database).  The resulting command becomes a job in the pantasks
+collection of jobs.  Most IPP analysis stages specify that the jobs
+are then sent to pcontrol for parallel process.  Before task generates
+the job, the \ippdbcolumn{pantasksState} is set to \ippmisc{RUN} so a
+future execution of the task will not attempt to re-run this specific job.
+
+Upon completion of the job, it is necessary to update the processing
+database with the results, specifically indicating in the database
+that the job has completed and if was successful.  Within the IPP,
+this responsibility is left to the program which ran the analysis.
+IPP analysis steps normally consist of two main elements: a C-language
+program to do the data analysis work and a supporting perl script
+which performs the database update upon completion.  Upon completion,
+the pantasks \ippmisc{RUN} tasks is responsible for updating the
+status within the book, but not within the processing database.  This
+split keeps the interactions at the pantasks level relatively light,
+leaving the overhead of the database interaction within the job
+running on one of the computing machines in the cluster.
+
+In addition to these tasks, most stages have a \ippmisc{revert} task
+paired with the \ippmisc{run} task.  These tasks run infrequently and
+generate jobs which perform an operation on the processing database to
+clear jobs which have failed with one of the ephemeral failure modes
+(see the discussion in Section~\ref{sec:processing.database}).  This
+step allows these failures to be cleared from the system, and schedule
+those jobs again for a retry.  
+
+Similarly, some stages have \ippmisc{advance} tasks that update the
+primary table to indicate that all of its components are complete.
+For many of the early stages of the pipeline (the \ippstage{chip}
+through \ippstage{warp} stages), this \ippmisc{advance} task also adds
+an entry into the database table for the next stage of processing for
+the exposure being considered.  This step allows the data to process
 automatically from stage to stage without intervention.
 
-\subsubsection{Stage automation}
+The IPP processing database is used to manage all versions of an
+analysis for all analysis stages.  In addition to the regular
+processing of the nightly data products, there may be large-scale
+re-processing analysis tasks or tests of various kinds.  It may be
+necessary for a test analysis of a particular item to use a different
+version of the processing software from the regular nightly analysis
+(for example, when testing a new algorithm for release).  A mechanism
+is needed to manage these different processing attempts of the same
+items.  With the IPP, this is accomplished with an extra field,
+\ippmisc{label}, for each processing stage.  Within the
+\ippmisc{load}, \ippmisc{revert}, and \ippmisc{advance} tasks
+discussed above, the query to the processing database for new items is
+restricted to a set of user-defined labels.  A given instance of
+pantasks will be supplied a set of labels which are then applied to
+all tasks managed by that pantasks.  For example, the pantasks which
+manages the nightly processing of the basic science analysis stages
+(chip - warp, stack, diff) is supplied with several labels which
+correspond to the different kinds of observations being performed.  In
+this way, the analysis of the nightly observations is kept separate
+from other processing attempts.
+
+
+
+\subsection{Stage automation}
 \label{sec:automation}
-\note{I'm not convinced this is the right place for it, but it felt like a natural extension of the ``advance''}.
-
-\note{wording..}  Beyond the warp stage, there is no longer a single
-``next'' stage into which data can be queued.  Because of this, more
-sophisticated methods are used to advance the data.  For processing
-data that is actively being observed at the summit, this is handled by
-a set of ``nightly science'' tasks and an associated
-\ippmisc{ippScript}.  The goal of these tasks is to ensure that
-exposures are correctly paired into sets of
-\ippstage{warp}-\ippstage{warp} difference images, and that nightly
-stacks are generated for MD fields.
-
-Queuing the diffs is done by first examining the set of all exposures
-that have been taken at the summit on the current night of observing,
-and querying information from each stage up through \ippstage{warp}
-stage.  These exposures are grouped by \ippdbcolumn{filter} and
-\ippdbcolumn{object}, which is a unique identifier for each telescope
-pointing on the sky.  Exposures in each group are then sorted by
-increasing observation date (\ippdbcolumn{dateobs}), and the
-information from each stage is used to ensure that the observations
-have completed all stages through \ippstage{warp} successfully.  If
-this is not the case, the script ignores that group until a future
-scan, to allow the data to finish processing.  If the lack of
-completion is due to a failure at a stage prior to \ippstage{warp},
-then that exposure is dropped from the group to allow the remainder to
-be considered for \ippstage{diff}s.  The remaining exposures are then
-paired sequentially, with the final exposure ignored in the case of an
-odd number of exposures (caused by either a failure in the processing
-of an exposure or due to the telescope closing for weather prior to
-completing all scheduled observations).  Once the observing process
-completes (signalled to the nightly science task and script by the
-presence in the database of entries for dark exposures that are taken
-each morning after the telescope closes), and the script has generated
-all \ippstage{diff} pairs that can be made with the above rule, a
-second pass is performed, this time with the exposures in each group
-sorted by decreasing observation date.  This change in ordering allows
-exposures that were excluded due to an odd number of exposures to be
-paired with the exposure closest in time (with the exposure that was
-previously first ignored).
+
+\note{start with a discussion of the standard sequencing (end-stage)}
+
+\note{then discuss the addstar sequences with manual triggering}
+
+Outside of the basic sequence of chip to warp, there is no single
+natural next step.  For example: a stack can be generated with any
+number of input warps; a difference image can be generated between a
+warp and any one of many other warps or stacks.  Without a single
+sequence, more complex and sophisticated decisions much be made.
+
+For nightly processing of data obtained at the summit, this is handled
+by a set of ``nightly science'' tasks and an associated
+\ippmisc{ippScript}.  These scripts have a well-defined and restricted
+set of goals: to ensure that difference images are generated for each
+exposures (either by pairing together warps or pairs warps with
+pre-defined stacks), that nightly stacks are generated for MD fields,
+and that the stacks are also differenced against an appropriate
+reference.  
+
+Pairing warps together is simplified by the observing strategy in
+which the same pointing is observed multiple times in a night.  By
+limiting to warp-warp pairs from the same pointing, the problem is
+significantly reduced from the arbitrary case.  
+
+{\em Queuing the diffs} is done by first examining the set of all
+exposures that have been taken at the summit on the current night of
+observing, and querying information from each stage up through
+\ippstage{warp} stage.  These exposures are grouped by
+\ippdbcolumn{filter} and \ippdbcolumn{object}, which is a unique
+identifier for each telescope pointing on the sky.  Exposures in each
+group are then sorted by increasing observation date
+(\ippdbcolumn{dateobs}).  The database results for each stage
+(chip-warp) are checked to ensure that the selected exposures have
+been successfully processed for all stages through \ippstage{warp}.
+Exposure groups are ignored until all exposures have either been
+processed through warp or have failed with a bad quality, meaning the
+exposure (or portion) cannot be processed.  Failed exposures are
+rejected.  The remaining exposures are then paired sequentially, with
+the final exposure ignored in the case of an odd number of accepted
+exposures.  Exposures paired in this way are sent to the
+ippstage{diff} analysis stage.
+
+Once observations have been completed for the night (signalled by the
+end-of-night dark exposures that are taken each morning after the
+telescope closes), and the script has generated all \ippstage{diff}
+pairs that can be made with the above rules, a second pass is
+performed, this time with the exposures in each group sorted by
+decreasing observation date.  This change in ordering allows exposures
+that were excluded due to an odd number of exposures to be paired with
+the exposure closest in time (with the exposure that was previously
+first ignored).  Exposure pairs in which at least one exposures does
+not have a pre-existing difference image are queued for difference
+image analysis.
 
 The nightly stacks are queued based on checking that a minimum number
@@ -1788 +1849 @@
 \ippstage{stack} database entries can be queued with the
 \ippstage{warp} entries.  Again, failures and weather can reduce the
-number of usable exposures, so the task generates \ippstage{stack}
-entries using whatever exposures are available if one has not yet been
-constructed by the time the morning dark exposures are registered into
-the database. \note{wording}
-
-Automating the nightly processing is important, as it ensures that
-data is processed as soon as it is downloaded from the summit,
-reducing the lag between an observation and the reduced data.  The
-other processing task that requires automation is the reprocessing of
-the entire $3\pi$ survey, as the size of the dataset make it
+number of usable exposures.  If no stack could be made for a given MD
+field with the minimum number of inputs by the time of the
+end-of-night darks, stacks are generated using using whatever
+exposures are available.
+
+The automatic nightly processing ensures that data is processed as
+soon as it is downloaded from the summit, reducing the lag between an
+observation and the reduced data. \note{some numbers here about
+  completion times and such?  Words about getting data to MOPS and SN
+  transient folks}
+
+\note{re-read paragraph below and cleanup}
+
+The other processing task that requires automation is the reprocessing
+of the entire $3\pi$ survey, as the size of the dataset make it
 challenging to do manually.  To manage this, the ``large area
 processing'' (LAP) task and script are used.  The first stage of this
@@ -1823 +1889 @@
 \ippdbcolumn{projection\_cell}.
 
+
 \subsection{Nebulous}
 \label{sec:nebulous}
+
+\subsubsection{Motivation and Concept}
 
 A major concern recognized early in the Pan-STARRS project is the
 challenge of storing and managing the large volume of data that is
 generated by the GPC1 camera.  The \ippprog{Nebulous} system was
-designed to organize this data.  The main components of this system
-are a database storing the locations of the files, with a Simple
-Object Access Protocol (SOAP) interface between the database and the
-other IPP programs \note{define / mention http}.  The actual files are
-stored on a collection of computers with substantial disk partitions
-in the IPP cluster, shared within the cluster via NFS.  This
-distribution of files is useful to balance the disk I/O, as this
-parallelizes the load.
-
-The original design of \ippprog{Nebulous} was intended to aid in the
-targetted processing of data, by having specific image data (such as
-all the images from one OTA device) located on a single storage node.
-This would allow any jobs involving that data to be assigned to the
-storage node, eliminating network I/O.  Important data could be
-duplicated to additional data nodes, with the alternate locations
-stored in the database.  In practice, however, hardware failures and
-increases in hard drive storage volumes and network bandwidth have
-reduced the degree to which the IPP processing is targeted.
-
-\note{this is a critical paragraph and needs to be re-written to be
-  more accessible}
-When a program creates a new file in \ippprog{Nebulous}, it supplies
-an URI of the form \code{neb://HOST.VOLUME/PATH/FILENAME}.  The host
-and volume specifiers are optional, and allow a file to be created on
-a specific node.  The path and filename appear similar to a standard
-full file location, and are used internally as the ``external id''.
-\note{mention the nebulous schema before this?}  A storage object
-entry is then created in the database for this id, and an instance of
-the file created on the specified node (or at random from available
-nodes if left empty).  This instance is created in a deterministic
-filename location.  The external id is hashed using the SHA-1
-function, and the first four hexadecimal digits of this hash are
-separated into two two-digit strings and used as the top and second
-level directory location for the disk file.  The disk file is created
-using the database instance id, and a transformed version of the
-external id, which has colons replacing any forward slash characters,
-to convert the external id path into a filename.  For the example URI
+designed to aid in thie process.  \ippprog{Nebulous} is not a file
+system per-se, but only method of tracking the locations of files
+within the file system, and of tracking duplicate copies of the same
+file.  The core of \ippprog{Nebulous} is a dedicated database engine
+which tracks ``storage objects'', the concept of a file exists in the
+system.  Each storage object may be associated with a number of copies
+of the actual files on the disks in the storage system (called
+``instances''), which are also recorded by the database.  In the IPP
+cluster, the file instances are stored on a collection of computers
+with substantial disk partitions shared via NFS.
+
+\ippprog{Nebulous} also explicitly tracks the different computers on
+which the file instances are stored.  This allows the system to expose
+files to the user only on machines which are currently active in
+\ippprog{Nebulous}.  If, for example, a storage computer crashes or
+needs to be taken offline, the machine can be made unavailable in
+\ippprog{Nebulous}, in which case only instances on other machines
+will be supplied to users.
+
+This localization is also useful for allowing the IPP processing to
+target processing to computers based on the location of the data.  For
+example, all raw images from a specific chip in the camera could be
+stored on a specific computer (for at least one of the instances).
+All of the analysis stages which interact with that chip could then be
+preferentially targetted to be run on that computer.  The localization
+in \ippprog{Nebulous} and the host targetted processing in pantasks
+can therefore work together to encourage processing to require only
+local disk access, reducing the I/O local on the network
+infrastructure.  In the early stages of the Pan-STARRS project, this
+was important because network bandwidth was an expensive resource.  In
+practice, the as-built IPP has had sufficient network bandwidth that
+this targetting was not required.  In practice, due to the timing of
+hardware aquisition, occasional hardware failures, and other
+organizational details, targetted processing has only been used to a
+moderate degree within the Pan-STARRS cluster. \note{can we get a
+  number here?}
+
+\subsubsection{Implementation Details}
+
+The user interfaces to Nebulous consist of command-line programs as
+well as APIs in both C and Perl.  The basic user commands to interact
+with Nebulous are to 1) create a new storage object and associated
+instance; 2) add a new instance to an existing storage object; 3)
+remove (cull) an instance; 4) delete a storage object; and 5) find a
+file associated with a given storage objects.  Note that these user
+commands do not affect the files on disk \note{true for cull?}
+(exception: the create function will create an empty file if one does
+not exist).  They only change the state of the Nebulous database; it
+is the responsibility of the user program to read and write data to a
+file and to create the copies, etc.
+
+For the Nebulous users, the identifier of a storage object is a unique
+string with the form of a UNIX file path: e.g. a/b/c/file.  When a
+program creates a new file in \ippprog{Nebulous}, it supplies a URI of
+the form \code{neb://HOST.VOLUME/PATH/FILE}.  The host and volume
+specifiers are optional, and allow a file to be created on a specific
+node.  The path and filename portions become the identifier and are
+recorded in the \ippmisc{storage_object} table in the
+\ippmisc{extern_id} field.  A storage object entry is then created in
+the database for this id, and an instance of the file created on the
+specified node (or at random from available nodes if left empty).
+
+Files are stored on specific computers in a \ippprog{Nebulous}
+directory or directories on that computer.  In the IPP system, the
+top-level Nebulous directories are usually placed at the root of the
+storage device as mounted on the machine, in a subdirectory named
+\code{nebulous}.  Beneath the top-level directory are 256
+subdirectories with names of the form 00 - ff (i.e., 2 digit
+hexadecimate number).  Each subdirectory again as 256 subdirectories
+with the same naming scheme.  
+
+The filename of an instance in Nebulous is deterministic and derived
+from the \ippmisc{extern_id}: the \ippmisc{extern_id} is hashed using
+the SHA-1 function, and the first four hexadecimal digits of this hash
+are separated into two two-digit strings and used as the top and
+second level directory location for the disk file.  The instance table
+in the Nebulous database includes a unique auto-incrementing index to
+provide a unique SQL ID for each instance.  Under the subdirectory
+identified above, the disk file name is by appending the database
+instance id with a string derived from the \code{extern_id}: forward
+slash characters are replaced in the name with colons so the string
+can represent a file in the UNIX filesystem.  For the example URI
 above, this results in a file located on disk in a location like
-\code{/data/HOST.VOLUME/nebulous/d5/d8/9244993440.PATH:FILENAME}.
-This file naming structure is useful, as it duplicates database
-contents on disk.  \note{rephrase}
-
-The storage volumes that contain the data on disk are defined in the
-\ippprog{Nebulous} database in a number of ways.  First, the locations
-and mount points for the actual NFS storage are listed in the
+\code{/data/HOST.VOLUME/nebulous/d5/d8/9876.PATH:FILE}.
+This file naming structure has the benefit of providing redundancy
+between the filename on disk and the instance in the database.
+
+Nebulous tracks additional information beyond just the storage objects
+and the associated instances.  As mentioned above, the storage volumes
+are tracked to provide a link between a top level nebulous directory
+and the computer which contains that directory.  The locations and
+mount points for the actual NFS storage are listed in the
 \ippdbtable{volume} table.  This table contains columns indicating if
 the volume should be used for reading (\ippdbcolumn{available}) and
-writing (\ippdbcolumn{allocate}), allowing these properties to be
-manually set, which is useful in scheduling downtime for hardware
-issues.  Another column, \ippdbcolumn{xattr}, is used to control the
-behavior of this volume, with specific values used to denote desired
-behavior.  For instance, the value of $1$ is used to indicate that a
-volume should only be used as a backup volume (which accepts only
-replicated copies), and the value of $5$ is used to indicate that the
-volume is permanently unavailable, and should be ignored. \note{more
-  detail, more specific}
-
-In addition to this permanent table describing the volumes, a
-\ippdbtable{mountedvol} table also exists that only lists those
+writing (\ippdbcolumn{allocate}).  As noted above, Nebulous will not
+return a file to the user if the storage volume is marked as not
+\ippdbcolumn{available}.  If a storage volume is marked as not to be
+\ippdbcolumn{allocate}ed, then new storage objects will not generate
+instances on that volume, but existing instances may be supplied to
+the user.  
+
+\note{we care about the concepts here, but not the numbers. reword}
+Another column, \ippdbcolumn{xattr}, is used to control the behavior
+of this volume, with specific values used to denote desired behavior.
+For instance, the volume may be marked to be used only for backup, in
+which case it will not be used to store an instance by default, but
+will be used preferentially if an instance noted as a backup when it
+is generated.  Alternatively, a volume may be marked as permanently
+unavailable, and should be ignored in most contexts.  This latter
+option allows the system to retain the memory of hardware which has
+been retired (and potentially to retain information about instances
+which were previously on such machines). 
+
+In addition to the static table describing the volumes, a second
+dynamically-generated table, \ippdbtable{mountedvol}, lists those
 volumes that are currently visible and accessible from the
 \ippprog{Nebulous} database server.  This table also lists the total
@@ -1897 +2022 @@
 \ippdbtable{volume} table.
 
-The final table controlling the operations of the \ippprog{Nebulous}
-volumes is the \ippdbtable{cabinet} table, which organizes the
-individual volumes into ``cabinets,'' a concept loosely based on the
-physical arrangement of the storage servers in the data center.  These
-cabinets are used to prevent the replication of a storage object
-within a group of volumes where all instances of the object could be
-taken off line by a single failure.  Since servers within a given
-cabinet in the data center share a common set of PDUs \note{define},
-it is important to ensure physical distance between replicated copies
-to guarantee that a temporary failure of one of the cabinet PDUs does
-not significantly impact processing.
+The \ippdbtable{cabinet} table organizes the individual volumes into
+``cabinets,'' a concept loosely based on the physical arrangement of
+the storage servers in the data center.  These cabinets are used to
+prevent the replication of a storage object within a group of volumes
+where all instances of the object could be taken off line by a single
+failure.  Since servers within a given cabinet in the data center
+share a common set of power delivery units (PDUs), it is important to
+ensure physical distance between replicated copies to guarantee that a
+temporary failure of one of the cabinet PDUs does not significantly
+impact processing.
+
+The nebulous user APIs do not interact directly with the nebulous
+mysql database.  Instead, they interact with one of several computers
+with an Apache web server.  Interactions with the Apache server are
+performed using the Simple Object Access Protocol (SOAP) interface,
+while the Apache servers interact directly with the Mysql database
+server.  This architecture avoids the overhead of setting up and
+tearing down the Mysql connection for each Nebulous command, instead
+using only the low-latency SOAP communications.
 
 \note{need a paragraph or two on stats: how many objects, how many