Changes between Version 39 and Version 40 of External_Supercomputing_Notes

Timestamp:

May 16, 2014, 5:28:41 PM (12 years ago)

Author:

watersc1

Comment:

--

External_Supercomputing_Notes

@@ +1 @@
+== 2014-05-16 ==
+
+After sending Ken a summary of how the remote processing works, I thought it would useful to copy that information here.
+
+
+
+[[Image(IMG_20140516_165558_259.jpg)]]
+
+* Password issue (box 1 on the diagram).
+
+The head node runs the pantasks controlling all processing.  The user
+doing the processing needs to have a key-card validated connection to
+LANL.  Due to the LANL security system, this connection is
+disconnected every 12 hours, and must be re-established to do any
+executing or polling of remote jobs.
+
+ * Slow preparation issue (box 2 on the diagram).
+
+Each remote job needs to have all of its file depenencies and commands
+structured, and this can happen on any node in the pantasks.  This
+does not require an active connection to LANL, as it is purely
+IPP-side.  As each remote job consists of ~1000 exposures, and each
+exposure has on order 60-80 sub-components, these jobs take a long
+time (each subcomponent seems to take on the order of a second or
+less).  However, as we can run many of these simultaneously, the
+average cost of these prep jobs can be brought down.  This runs into
+the issue that if we have many jobs being prepared, we will need to
+run them all.
+
+ * Data persistence issue (not directly visible on diagram).
+
+Having many jobs in the queue requires a lot of data to be transfered,
+and subsequent stages (cam/warp/stack) need to be executed before the
+LANL disk quota removes needed results from previous stages.  We are
+only transferring back (box 7) permanent data products that we want to
+keep, not all intermediate products.
+
+ * Remainder of explanation (boxes 2+ on diagram)
+
+ * Box 2: Prepare
+Each stage has a different prepare script, but they all output the
+same products: a list of filename requirements (in multiple formats to
+match nebulous/disk formats), a list of commands to execute (the
+ppImage/psastro/pswarp/ppStack, but also various mkdir commands to
+ensure output products can be constructed, and symlink commands to
+alias files from their nebulous key to nebulous disk representation),
+and a resource requirement.
+
+ * Box 3-5: Exec
+Once the prepare is finished, pantasks can launch an exec script [3]
+via the ssh link that feeds that data through to "mu-fe", the Mustang
+front-end computer.  This launches the transfer tool on mu-fe [4],
+which in turn connects via ssh and tar to the IPP storage accessible
+from ippc2X.  Once all required data has been downloaded to the
+scratch disk at LANL, the job resource requiest is submitted to the
+MOAB scheduler [5].
+
+ * Box 6-7: Poll
+
+After some amount of time (currently one hour after the GPC1 database
+entry for the remoteRun was last updated), pantasks launches a poll
+command over the ssh link to mu-fe [6].  If the job has completed, the
+filelists calculated in the prepare step are used to pass back any
+permanent data to the IPP storage nodes [7].  Included in this are the
+dbinfo files, which contain the update commands necessary to update
+the GPC1 database, and these are executed by the poll command before
+it sets the remoteRun state to 'full'.
+
+Unpictured are the IPP advance commands that can also run in the
+pantasks, which launch the next stage processing for the data
+returned.
+
+
 == 2014-04-08 ==