Changes between Version 3 and Version 4 of Raw_Images_Files_in_Nebulous

Timestamp:

Feb 29, 2012, 12:17:41 PM (14 years ago)

Author:

Serge CHASTEL

Comment:

--

Raw_Images_Files_in_Nebulous

@@ -10 +10 @@
 From those databases, the number of raw images files for gpc1 were extracted for each host, for each ota, and for each nebulous first-level directory (that is, one of the 256 directories from /export/<hostname>.0/nebulous/00, /export/<hostname>.0/nebulous/01, ..., /export/<hostname>.0/nebulous/ff). The script slocate_statistics_multiple.py has been used to create the files in slocate_data (the script can be executed: slocate_statistics_multiple.py <hostname_1> ... <hostname_n> and creates the files slocate_data/<hostname_1>...) 
 
+Note: 60 nodes are hosting the data (namely: ipp006, ipp007, ipp008, ipp009, ipp010, ipp011, ipp012, ipp013, ipp014, ipp015, ipp016, ipp017, ipp018, ipp019, ipp020, ipp021, ipp023, ipp024, ipp025, ipp026, ipp027, ipp028, ipp029, ipp030, ipp031, ipp032, ipp033, ipp034, ipp035, ipp036, ipp037, ipp038, ipp039, ipp040, ipp041, ipp042, ipp043, ipp044, ipp045, ipp046, ipp047, ipp048, ipp049, ipp050, ipp051, ipp052, ipp053, ipp054, ipp055, ipp056, ipp057, ipp058, ipp059, ipp060, ipp061, ipp062, ipp063, ipp064, ipp065, ipp066)
 
+On the ippc11 mysql server, the FilesStatistics database was created:
+{{{
+CREATE DATABASE FilesStatistics; 
+USE FilesStatistics;
+CREATE TABLE RawImages (id INT PRIMARY KEY AUTO_INCREMENT, host CHAR(7), INDEX(host), directory CHAR(3), INDEX(directory), ota CHAR(6), INDEX(ota), how_many INT, INDEX(how_many));
+}}}
+
+It has been populated with the various files from slocate_data.
+{{{
+cat slocate_data/* | sed 's/^/INSERT INTO RawImages(host, directory, ota, how_many) VALUES /g' | sed 's/$/;/g' > populate.sql
+cat populate_db.sql | mysql -u ippuser -pipp FilesStatistics
+}}}
+
+ * Average number of ota files by host: {{{SELECT SUM(how_many)/60 FROM RawImages;}}}: 580145
+ * Difference (absolute and relative) between that average number and the current number on one host:
+   {{{SELECT host, SUM(how_many)-580145 AS adiff, (SUM(how_many)-580145)/580145*100 AS rdiff FROM RawImages GROUP BY host;}}}
+ * View without the directory information: {{{CREATE VIEW RawImagesSimple AS (SELECT host, ota, SUM(how_many) AS ota_count FROM RawImages GROUP BY host, ota);}}}
+