Changes between Version 7 and Version 8 of External_Supercomputing_Notes

Timestamp:

Dec 19, 2013, 9:59:08 AM (13 years ago)

Author:

eugene

Comment:

--

External_Supercomputing_Notes

@@ -82 +82 @@
 
 As of 2013-12-18, I have not run an IPP command through this stask scripts.
+
+=== comments by EAM ===
+
+Issues which we need to consider for large-scale processing on this cluster:
+
+* lack of a database
+ * how do we provide the right detrends?
+ * how do we synchronize results with IfA / gpc1?
+ * we cannot use the database for job sequencing (gpc1)
+* interacting with Moab 
+ * can we use stask.py to run under pantasks or equivalent?
+ * can we use pantasks to talk directly to Moab? (note that Serge wrote a pantasks backend to talk to condor as a drop-in replacement for pcontrol; the same could potentially be done for Moab)
+
+The primary problem is that, without a database, we cannot coordinate our operations as we currently do.  Even if pantasks could talk to Moab, without a database, we cannot sequence and schedule jobs.  This precludes the use of a straight build of IPP locally to run the full system.  
+
+In any arrangement, we will certainly want to ship all of our detrend data and the reference photometry / astrometry database up front.  We will have to define a way to automatically generate the commands and to retrieve results for some chunk of the processing, including the info that needs to be pushed into the gpc1 database.
+
+Some possible ways of handling the interaction:
+
+1. treat the remote cluster as a set of nodes and use our pantasks (talking directly to gpc1) to send the jobs.  This would be a very fine-grained integration with the current IPP processing.  It would require any job (say chip) to generate a command and a bundle with the data and the pointers to the appropriate local references (detrends, etc).  then the bundle would be shipped to the remote cluster.  when the job is done, that fact needs to be carried back to the local pantasks, and the results retrieved.
+1. identify a complete sequence (say, chip, cam, warp) and send that as a bundle.  This sounds easier that #1 above but is in fact equivalent in terms of the need for automatic generation of the bundle / command, shipping the data and discovering the completion
+1. choose a large sequence (eg, all chip processing and all downstream processing to stack a complete projection cell).  this could be done with less automation at least at first, though there are still ~2k projection cells per filter, so some automation would be needed eventually.  
+
+It seems to me that any solution is going to require us to automatically ship data on some scale to LANL, automatically send a job to Moab, and automatically capture the result.  We should start on achieving those goal in a generic way rather than worrying about running IPP jobs specifically up front.