New Nebulous

Goal

The "New Nebulous" update should enable client code to transparently request a local version of a nebulous key. It should also be backwards compatible, so that an "Old Nebulous" client will still receive valid responses. The code change should be entirely on the Nebulous-Server side, meaning only the apache servers and the database need to be updated.

Location awareness will be managed by a "distance" calculation defined in the Nebulous-Server/lib/Server/SQL.pm "get_object_instances_by_proximity" function as

distance = abs(vol_id_instance - vol_id_request) + 20 * abs(cab_id_instance - cab_id_request) + 100 * abs(site_id_instance - site_id_request)

where the _instance values are for the instances of the key under consideration, and the _request values are defined by a look up based on the host issuing the nebulous query.

Database update

In order to allow location awareness, we need to add more information to the database. This will need to be manually updated, but after an initial configuration period, should not need to be changed until hosts are added or removed from the system.

cabinet table

The cabinet table needs to have a site_id column added to allow the new location awareness to operate:

ALTER TABLE cabinet ADD site_id INT NOT NULL DEFAULT 0 AFTER name;
ALTER TABLE cabinet ADD KEY(site_id);

The current trunk version of neb-cabadd assumes the existence of this column, and will not function without it. Examples of neb-cabadd and associated neb-vol operations to add new cabinets and volumes are listed below (from the ITC test installation of the new code):

neb-cabadd --update --db nebulous --host $NEB_DBSERVER --user $NEB_USER --pass $NEB_PASS --cname ITC_simA1 --location ITC --site_id 1 --cab_id 1
neb-cabadd --update --db nebulous --host $NEB_DBSERVER --user $NEB_USER --pass $NEB_PASS --cname ITC_simA2 --location ITC --site_id 1 --cab_id 2
neb-cabadd --update --db nebulous --host $NEB_DBSERVER --user $NEB_USER --pass $NEB_PASS --cname ITC_simB1 --location ITC_2 --site_id 2 --cab_id 3
neb-cabadd --update --db nebulous --host $NEB_DBSERVER --user $NEB_USER --pass $NEB_PASS --cname ITC_simB2 --location ITC_2 --site_id 2 --cab_id 4

neb-voladd --db nebulous --host $NEB_DBSERVER --user $NEB_USER --pass $NEB_PASS --vhost ipp105 --vname ipp105.0 --uri file:///data/ipp105.0/nebulous_test --mountpoint /data/ipp105.0
neb-voladm --db nebulous --host $NEB_DBSERVER --user $NEB_USER --pass $NEB_PASS --vname ipp105.0 --cab_id 1
neb-voladd --db nebulous --host $NEB_DBSERVER --user $NEB_USER --pass $NEB_PASS --vhost ipp105 --vname ipp105.1 --uri file:///data/ipp105.1/nebulous_test --mountpoint /data/ipp105.1
neb-voladm --db nebulous --host $NEB_DBSERVER --user $NEB_USER --pass $NEB_PASS --vname ipp105.1 --cab_id 1

neb-voladd --db nebulous --host $NEB_DBSERVER --user $NEB_USER --pass $NEB_PASS --vhost ipp106 --vname ipp106.0 --uri file:///data/ipp106.0/nebulous_test --mountpoint /data/ipp106.0
neb-voladm --db nebulous --host $NEB_DBSERVER --user $NEB_USER --pass $NEB_PASS --vname ipp106.0 --cab_id 2
neb-voladd --db nebulous --host $NEB_DBSERVER --user $NEB_USER --pass $NEB_PASS --vhost ipp106 --vname ipp106.1 --uri file:///data/ipp106.1/nebulous_test --mountpoint /data/ipp106.1
neb-voladm --db nebulous --host $NEB_DBSERVER --user $NEB_USER --pass $NEB_PASS --vname ipp106.1 --cab_id 2

aliasvol table

For storage volumes that are already in the nebulous database, the look up for the _request values can use the information in the volume and cabinet tables. However, we have a large number of compute nodes that do the actual processing that are not known to the nebulous system. To get around this, a new aliasvol table is defined that can map the request host to a known volume:

CREATE TABLE aliasvol (
       alias_id INT NOT NULL AUTO_INCREMENT,
       alias VARCHAR(255) NOT NULL UNIQUE,
       name  VARCHAR(255) NOT NULL,
       vol_id INT NOT NULL,
       PRIMARY KEY(alias_id),
       KEY(alias),
       KEY(name),
       FOREIGN KEY(vol_id) REFERENCES volume(vol_id),
       FOREIGN KEY(name) REFERENCES volume(name)
) ENGINE=innodb DEFAULT CHARSET=latin1;

This then needs to be populated with entries:

neb-aliasadd --db nebulous --host $NEB_DBSERVER --user $NEB_USER --pass $NEB_PASS --alias_name ipp110.0 --target_name ipp105.0
neb-aliasadd --db nebulous --host $NEB_DBSERVER --user $NEB_USER --pass $NEB_PASS --alias_name ipp111.0 --target_name ipp106.0
neb-aliasadd --db nebulous --host $NEB_DBSERVER --user $NEB_USER --pass $NEB_PASS --alias_name ipp112.0 --target_name ipp107.0
neb-aliasadd --db nebulous --host $NEB_DBSERVER --user $NEB_USER --pass $NEB_PASS --alias_name ipp113.0 --target_name ipp108.0
neb-aliasadd --db nebulous --host $NEB_DBSERVER --user $NEB_USER --pass $NEB_PASS --alias_name ipp114.0 --target_name ipp109.0

For processing nodes that share a cabinet with a storage node, the target is simple. For other processing nodes, we should simply choose one in a nearby cabinet.

cabinet entries

After moving from the MRTCB to the ITC, we never reorganized the cabinet table to ensure all hosts are in the same database cabinet as they are physically. neb-voladm can be used as above to correct this issue, and should be before declaring the transition complete. neb-cabadd also has a --update mode to allow cabinet entries to be modified.

Server code installation

My documentation notes that the code should be installed as root on the apache servers.

svn export https://svn.pan-starrs.ifa.hawaii.edu/repo/ipp/trunk/Nebulous-Server
cd Nebulous-Server && perl Build.PL && ./Build && ./Build install && cd ..
/etc/init.d/apache2 stop
sleep 1
/etc/init.d/apache2 start

Client code installation

Although updated client code is not needed to interact with the new server code, it also will not benefit from the location awareness. Both lib/Nebulous/Client.pm and nebclient/src/nebclient.c must be r39926 or greater, as this change added the calls to hostname to supply the _request information.