Shuffle

Using a purely Nebulous system to shuffle data around is inefficient because it operates on one file at a time, requires numerous database interactions, and forces disk syncs that can slow systems down. To work around this, we switched to an rsync/reinsert process which has proven to be faster and less problematic.

rsync

The easiest part is launching the rsync processes. For the shuffle to clear ippb02, I ran this with four clients:

rsync -au /data/ippb02.0/nebulous /data/ippb07.0/retire_rsyncX/ippb02.0/
rsync -au /data/ippb02.1/nebulous /data/ippb07.1/retire_rsyncX/ippb02.1/
rsync -au /data/ippb02.2/nebulous/[0-7]* /data/ippb08.0/retire_rsyncX/ippb02.2a/nebulous/
rsync -au /data/ippb02.2/nebulous/[8-f]* /data/ippb08.1/retire_rsyncX/ippb02.2b/nebulous/

These are still running in screen sessions on ippb02 for the 0 and 1 partitions:

2435.pts-0.ippb02
2583.pts-1.ippb02

update from EAM 2018.06.13 : I have set up a pantasks-based set of rsync scripts in ~ipp/b-node.shuffle.20180530. These are running for b01, b02, b03. b02 is done, b01 & b03 should finish in the next few days.

reinsert

The reinsert jobs use the trunk/tools/stsci_disks/p4_restore.pl script. Unfortunately, I never made the script easy-to-use, so at the minimum, some edits will be needed (the $root variable should almost certainly be a command line option). For the rsyncs above, it should be /export/${host}.${volume}/retire_rsyncX/${source}.${source_vol}/nebulous/${dir} . The script must also run locally to the destination host. Since there are some IO operations, this helps with speed.

Additionally, since we are using data on the backup nodes in this shuffle, Gavin will need to open the firewall to allow the database to be accessed from those IP addresses. The last time this was needed, I added a user entry to the database for the IP range, but revoked all permissions afterward. Those permissions will also need to granted again.

Running the reinsert is a series of

p4_restore.pl --host ipp118 --volume 0 --source ipp004 --source_vol 0 --dir 03

This inserts data that was on ipp004.0 into ipp118.0 for the top-level Nebulous directory 03. The script iterates over all subdirectories, and for each file found does the following:

1. Identifies ins_id from the filename
2. Queries the database to be sure the file is listed as being on the specified source host/volume
3. Moves the file from the rsync location into the correct destination nebulous directory
4. Updates the database vol_id and uri for that ins_id to reflect the new location.

This process is usually fast, as the move operation does not need to copy data to a new partition.

I grouped these commands into a Makefile (generated by a perl script) for execution out of convenience.