Raw Images Files in Nebulous

Shuffling Analysis

(Unless noted all files are referenced from /export/ippc11.1/schastel/FilesStatistics/slocate on host ippc11)

On week 5 (i.e. around 1/30/2012-2/3/2012), the slocate utility was ran on the different cluster nodes hosting data. Those databases can be found in slocate_entries.

From those databases, the number of raw images files for gpc1 were extracted for each host, for each ota, and for each nebulous first-level directory (that is, one of the 256 directories from /export/<hostname>.0/nebulous/00, /export/<hostname>.0/nebulous/01, ..., /export/<hostname>.0/nebulous/ff). The script slocate_statistics_multiple.py has been used to create the files in slocate_data (the script can be executed: slocate_statistics_multiple.py <hostname_1> ... <hostname_n> and creates the files slocate_data/<hostname_1>...)

Note: 60 nodes are hosting the data (namely: ipp006, ipp007, ipp008, ipp009, ipp010, ipp011, ipp012, ipp013, ipp014, ipp015, ipp016, ipp017, ipp018, ipp019, ipp020, ipp021, ipp023, ipp024, ipp025, ipp026, ipp027, ipp028, ipp029, ipp030, ipp031, ipp032, ipp033, ipp034, ipp035, ipp036, ipp037, ipp038, ipp039, ipp040, ipp041, ipp042, ipp043, ipp044, ipp045, ipp046, ipp047, ipp048, ipp049, ipp050, ipp051, ipp052, ipp053, ipp054, ipp055, ipp056, ipp057, ipp058, ipp059, ipp060, ipp061, ipp062, ipp063, ipp064, ipp065, ipp066)

On the ippc11 mysql server, the FilesStatistics database was created:

CREATE DATABASE FilesStatistics; 
USE FilesStatistics;
CREATE TABLE RawImages (id INT PRIMARY KEY AUTO_INCREMENT, host CHAR(7), INDEX(host), directory CHAR(3), INDEX(directory), ota CHAR(6), INDEX(ota), how_many INT, INDEX(how_many));

It has been populated with the various files from slocate_data.

cat slocate_data/* | sed 's/^/INSERT INTO RawImages(host, directory, ota, how_many) VALUES /g' | sed 's/$/;/g' > populate.sql
cat populate_db.sql | mysql -u ippuser -pipp FilesStatistics

• Average number of ota files by host: SELECT SUM(how_many)/60 FROM RawImages;: 580145
• Difference (absolute and relative) between that average number and the current number on one host: SELECT host, SUM(how_many)-580145 AS adiff, (SUM(how_many)-580145)/580145*100 AS rdiff FROM RawImages GROUP BY host;
• View without the directory information: CREATE VIEW RawImagesSimple AS (SELECT host, ota, SUM(how_many) AS ota_count FROM RawImages GROUP BY host, ota);