Index: trunk/doc/release.2015/ps1.datasystem/datasystem.tex
===================================================================
--- trunk/doc/release.2015/ps1.datasystem/datasystem.tex	(revision 39886)
+++ trunk/doc/release.2015/ps1.datasystem/datasystem.tex	(revision 39887)
@@ -222,4 +222,33 @@
 
 \subsection{Nebulous}
+
+Storing the large volume of data that is generated by the GPC1 camera
+was recognized early in the Pan-STARRS project as a major concern.
+The \ippprog{Nebulous} system was designed to organize this data.  The
+main components of this system is a database storing the locations of
+the files, with a Simple Object Access Protocol interface between the
+database and the other IPP programs.  The actual files are stored on
+NFS mounted partitions on a series of storage nodes in the IPP
+cluster.
+
+The original design of \ippprog{Nebulous} intended to aid in the
+targetted processing of data, by having specific image data (such as
+all the images from one OTA device) located on a single storage node.
+This would allow any jobs involving that data to be assigned to the
+storage node, eliminating network IO.  Important data could be
+duplicated to a further data nodes, with the alternate locations
+stored in the database.  In practice, however, hardware failures and
+increases in hard drive storage volumes and network bandwidth have
+reduced the amount that the IPP processing is targeted.
+
+When a program creates a new file in \ippprog{Nebulous}, it supplies
+an URI of the form \texttt{neb://HOST.VOLUME/PATH/FILENAME}.  The host
+and volume specifiers are optional, and allow a file to be created on
+a specific node.  The path and filename appear as a standard full file
+location, and are used internally as the ``external id''.  A
+storage\_object is then created in the database for this id, and an
+instance of the file created on the specified node (or at random from
+available nodes if left empty).  
+
 
 \subsection{Pantasks \& Parallel Processing}
@@ -907,5 +936,128 @@
 \subsection{Los Alamos National Labs} 
 
+In order to increase the processing rate for the $3\Pi$ PV3 data, we
+partnered with Los Alamos National Lab to gain access to the Mustang
+supercomputer.  The supercomputer is comprised of 3088 processing
+nodes, each with 12 cores and 64GB of RAM.  The processing nodes do
+not have significant local disks, but are connected to multiple
+petabyte scale scratch disks.  Job management is controlled by the
+Moab HPC system\footnote{\url{http://www.adaptivecomputing.com/}},
+which schedules resource requests among users, allocating processing
+nodes to satisfy jobs, and terminating those jobs if they exceed their
+scheduled time limit.
+
+This system is part of the lab's ``Turquoise'' network, allowing it to
+be used for research projects that do not handle sensitive data.  It
+is, however, subject to stricter access controls than are in place at
+the main IPP processing cluster.  Login sessions are terminated after
+12 hours, requiring new sessions to be initiated regularly.  Network
+access is also filtered, with only SSH connections allowed between the
+IPP cluster and Los Alamos.  This restriction removes the ability for
+the processing to contact the IPP processing database directly.
+
+To work around this, additional steps were needed to ensure efficient
+use of the computing resources.  A periodic poll of outstanding tasks
+was done on the IPP cluster, using the information stored in the
+database, and those tasks assigned to a processing bundle.  Each
+component task in the bundle was then checked to identify the set of
+input files needed to complete the task, the commands necessary to
+complete the task, and the set of output files that should be
+generated if the task completed successfully.  Once this information
+had been generated for all tasks, the component lists were merged, and
+the Moab job control file was constructed.
+
+The control file contains the resource requests for the job, as well
+as the commands to be executed to complete it.  The resource request
+was calculated based on the number of tasks included in the job bundle
+$N_\mathrm{tasks}$, and scaled by the expected execution time
+($t_\mathrm{task}$) and computational intensity of the component tasks
+($S_\mathrm{task}$).  For a given job bundle, an initial estimate of
+the number of compute nodes needed is simply $\mathrm{nodes} =
+S_\mathrm{task} * N_\mathrm{tasks} / 12$.  To ensure that jobs were
+not prematurely terminated, we attempted to design the requested job
+processing time to be 25\% longer than the expected time to complete
+the component tasks.  Based on the initial node count, we calculated
+the request time as $t_\mathrm{request} = \lfloor 1.25
+\frac{\mathrm{nodes} * t_\mathrm{task}}{\mathrm{nodes}_\mathrm{max}}
+\rfloor + 1$, where $\mathrm{nodes}_\mathrm{max}$ is the maximum
+number of nodes that can be requested in a single job (1000 for
+Mustang).  Table \ref{tab:SC processing parameters} contains the cost
+values used for the various IPP processing stages.
+
+\begin{deluxetable}{lcc}
+  \tablecolumns{3}
+  \tablewidth{0pc}
+  \tablecaption{Cost values for remote processing}
+  \tablehead{\colhead{IPP Stage}&\colhead{$t_\mathrm{task}$ (s)}&\colhead{$S_\mathrm{task}$}}
+  \startdata
+  \ippstage{chip} & 150 & 2 \\
+  \ippstage{camera} & 1700 & 2 \\
+  \ippstage{warp} & 110 & 2 \\
+  \ippstage{stack} & 1500 & 6 \\
+  \ippstage{staticsky} & 7200 & 6 \\
+%  \ippstage{diff} & 300 & 2 \\
+  \ippstage{fullforce} & 300 & 2
+  \enddata
+  \label{tab:SC processing parameters}
+\end{deluxetable}
+
+Once the preparation for the job is complete, the input and output
+file lists, the task list, and the job control file are transferred
+via SCP to the Mustang cluster.  Local tasks are then initiated on the
+user interface nodes to SCP the input files onto the scratch storage
+disks if they do not already exist.  Once all the input files have
+been copied, the job is submitted to Moab for scheduling.  The Moab
+interface is periodically polled to determine the job status, and
+after it has completed, the results are retrieved in a similar way.
+Local tasks again SCP the output products, but to copy the results
+back to the IPP cluster.
+
+In addition to the standard output products, ``dbinfo'' files are
+constructed as part of the job execution.  These files contain
+database update commands to ensure that the IPP processing database
+has the correct entries for the tasks that were remotely executed.
+These commands are executed after confirming that all retrieved output
+products are present.
+
+Approximately half of the chip through warp processing for the PV3
+reduction was performed on Mustang, with 201,040 / 375,573 of the
+\ippstage{camera} stage products reduced there.  Only processing
+through the \ippstage{stack} stage was attempted, although with a
+smaller fraction of the total compared to the \ippstage{camera} stage,
+with 290,257 / 998,886 being produced at Los Alamos.  One reason for
+this decrease is that due to the memory constraints on the Mustang
+processing nodes, we were unable to run stacks with more than 25
+inputs there.  Stacks with this larger number of inputs overflow the
+memory of the processing node, and as they do not have disk space
+available for use as virtual memory, cause the machine to hang until
+the job time limit is reached.  These stacks were instead processed on
+the regular IPP cluster, where hosts with sufficent memory were
+available.
+
 \subsection{UH Cray Cluster} 
+
+In December 2014, the University of Hawaii installed a 178-compute
+node Cray supercomputer on the main Manoa campus.  As part of the
+initial commissioning of this computer, Pan-STARRS was invited to use
+this resource in February 2015, roughly corresponding with the
+completion of the initial processing of the \ippstage{chip} through
+\ippstage{stack} processing.  Although the number of nodes was much
+smaller than that available on Mustang, the nodes were more robust,
+with 20 cores and 128 GB of memory.  The scratch data storage was
+somewhat smaller than at Los Alamos, with only a single 600 TB volume.
+We had the unique ability to rapidly deploy to the UH Cray, using
+almost all nodes for IPP processing as other users at the university
+were designing code.  This rapid deployment was made possible by the
+similarity of the Slurm\footnote{\url{https://slurm.schedmd.com/}}
+scheduler and tools to those used by Moab (although the UH Cray has a
+smaller $\mathrm{nodes}_\mathrm{max}$ of 10).
+
+The UH Cray was used to do processing for the \ippstage{staticsky}
+stage, running approximately half of that photometry (101,528 /
+200,720).  We were also able to run part of the \ippstage{fullforce}
+photometry there as well, although more had to be run on the IPP
+cluster as other users started to utilize the system, with 168,685 /
+994,890 runs processed there.
+
 
 \acknowledgments