Changes between Version 6 and Version 7 of ITC_shuffle

Timestamp:

Dec 7, 2016, 5:05:30 PM (10 years ago)

Author:

watersc1

Comment:

--

ITC_shuffle

@@ -1 +1 @@
 = ITC shuffle =
+
+== First segment results ==
+
+After finishing the first 100k shuffle, the end statistics were:
+||            ||           alljobs          ||           success          ||           failure          ||
+||  AV Name   || Njobs   Tmin   Tave   Tmax || Njobs   Tmin   Tave   Tmax || Njobs   Tmin   Tave   Tmax ||
+||  -+ runjob || 100010   1.01 1622.12 850661.23 || 91463   8.13 1453.07 203219.15 ||  8547   1.01 3456.48 850661.23 ||
+
+This is slower than expected from the initial tests, but doesn't take into account two factors:
+ * The first ~50k were run using only the stare nodes, not with the additional ~200 pantasks clients.
+ * The majority of those stare nodes became hung during the week of thanksgiving, clogging the task list with jobs with very long execution times.
+
+I had to stop the second shuffle today because the network to the ITC was being worked on, so the only results from the second batch are based on a small sample:
+
+||            ||           alljobs          ||           success          ||           failure          ||
+||  AV Name   || Njobs   Tmin   Tave   Tmax || Njobs   Tmin   Tave   Tmax || Njobs   Tmin   Tave   Tmax ||
+||  ++ runjob ||   605   5.20 2060.32 4659.94 ||   473 1098.32 3145.94 4659.94 ||   132   5.20 1632.36 3697.19 ||
+
+These still have very long execution times.  I'm not sure where the bottleneck is.
+
+Looking at the ganglia plots for the network:
+
+|| stare00 || [[Image(stare00.gif)]] || Have been used since the beginning (5 nodes, 8x loading) ||
+|| ipp110  || [[Image(ipp110.gif)]] ||  Target storage nodes (13 nodes) ||
+|| ippx055 || [[Image(ippx055.gif)]] || Addition processing nodes (34 nodes, 6x loading) ||
+
+The target nodes appear to all cap at 8MB/s, so the transfer is moving ~104MB/s to ITC.  When this was using only the stare nodes, ~22MB/s was flowing through each (roughly the same total).  Adding the additional processing nodes dropped the stare node throughput to ~4MB/s, similar to what the lighter-loaded x-nodes at ~3MB/s.
+
+I suspect this means we're currently being limited by the 1G ITC-MRTCB connection.  
+
+== Shuffle operations ==
+
 === rawcheck.pl ===