| | 1 | == 2014-03-28 == |
| | 2 | |
| | 3 | The main final block on smoothly doing processing is the data transfer operations. I've spent some time this week thinking over how to structure this, and I think I've finally decided on the way that will work best. I attempted to see if the processing nodes could initiate connections outside the cluster, but that does not seem to be possible ("ssh: Could not resolve hostname ippc20.ipp.ifa.hawaii.edu: Temporary failure in name resolution"). |
| | 4 | |
| | 5 | This means relying on the front end nodes to do all the work. Mark sent me some forwards from the UMD team, as well as the perl script they're using for their data transfers. This script appears to stream a series of forks to run wget to fetch the data from the webservers on the ippcXX nodes. This is also being run in a virtual machine, although the emails suggest that that isn't a requirement. However, it does seem that the script is somewhat specific to the UMD downloading setup. For the IPP/SC interface, we need to transfer things over scp, and there are filename translations that need to be taken care of (disk file locations are different between the local and remote disks). |
| | 6 | |
| | 7 | Because of this, I think the best solution is to have the prepare_* steps generate the expected output file/local instances. Pushing this to the exec phase means recalculating information that is no longer available. Therefore, as part of the prepare step, while generating lists of files to transfer initially, and defining the remote commands to execute, a list of expected output files that need to be transferred also needs to be created. In addition, due to the constraints of the nebulous filesystem, neb-touch commands will be run to instantiate a disk file that can then be added to a list of transfer destinations. This also simplifies the requirement for the transfer engine, which only needs to know how to run "scp --options" on an element from file A and a destination from file B (so the same program can do both the fetching and the pushing). |
| | 8 | |
| | 9 | Since the required transfer code is not complicated, I think pushing the work to GNU parallel is probably the easiest option. This allows all front end nodes to do scp jobs, and the job/host parameter can be tuned as well. |
| | 10 | |
| | 11 | |
