Changes between Initial Version and Version 1 of ThroughputDemo_20091104

Timestamp:

Nov 6, 2009, 10:48:10 AM (17 years ago)

Author:

eugene

Comment:

--

ThroughputDemo_20091104

@@ +1 @@
+= Throughput Demonstration 2009.11.04 =
+
+== start @ 11:30am ==
+
+Bill, Heather & I have been running small-scale tests of the full
+processing -> distribution to ensure all of the steps are working.
+We've ironed out a few bugs and have a solid demonstration of the full
+sequence including automatic diffs, magic, destreak, and distribution.
+
+I've just (at 11:35) launched a full-scale throughput demonstration
+using 611 exposures.  These are 3pi exposures from July for the RA,DEC
+range 270.0 - 300.0, 0.0 - 20.0.  There are g,r,i exposures,
+supposedly all taken as TTI pairs.  Obviously, the fact that 611 is
+odd, and 612 is the closest multiple of 6, says that at least one is
+missing.
+
+I've also modified pantasks.pro (in ~ipp, not yet checked in) to fix
+the problem where we get paths of the form /data/any.0/foo/bar
+(something of a hack, I attempt to replace @HOST@.0 and @HOST@.1
+before doing just @HOST@).  This should help with nfs load during
+camera.
+
+=== update 1 ===
+
+my fix to the camera paths had no effect: the paths are now of the form:
+
+neb://any/gpc1/throughput.20091104/o4999g0217o.78668/o4999g0217o.78668.cm.29757.smf
+
+but they are still all going to ipp039, who is getting quite a load.
+I'm going to remove it as a processing node so it can deal with the
+camera I/O.
+
+=== update 2 ===
+
+we are having some trouble in this test because ipp039 is getting
+really slammed.
+
+the situation is that the 'any' neb host problem is affecting more
+than just camera: a number of every diff and warp run are also going
+to 'any', which means ipp039.  I'm checking into the cause, but it
+shows we need to fix this nebulous bug.
+
+=== update 3 ===
+
+i found a bug in the skycell by host pantasks function that allowed
+the range to be larger than count, which in turn set a number of the
+target hosts and nebulous volumes to 'any'.  i've fixed this, so
+hopefully the load on ipp039 will fall again...
+
+== report @ 3pm ==
+
+looks like the fix I put in for this is now working: we have no
+skycells with path_base including 'any' since the fix, and the load on
+ipp039 has been mostly sensible since then as well.  there are
+probably still a few warps sitting there that are being used for
+diffs, so it will be elevated until those are done.
+
+the processing speed is looking mostly reasonable, except for the
+diff_skycells, which were heavily hit by the ipp039 collisions.  I'll
+be looking for those times to drop somewhat as well.
+
+So far, we are 3 hours into the test, and we have only processed 1/6
+of the chips (and somewhat less of the other stages).  Since the goal
+is 12 hours, we are running a bit behind.  I'm hopeful that the ipp039
+collisions account for most of the gap...
+
+== report @ 6pm ==
+
+It has been 6 hours now, and the chip and diff processing times are
+not much better than before.  We've only done about 30% of the data
+set, so it is still going slowly.  At this rate, it will be 20 hours
+before we finish this batch, which is kind of slow.  I'm going to
+change my plan and try to boost the number of hosts.
+
+At the moment, our limiting steps have used the following number of seconds:
+
+chip: 1,021,709.35 sec
+warp: 610,303.59 sec
+diff: 540,536.4 sec
+destreak: 259,726 sec
+dist process: 177,430.2 sec
+camera: 69,067.08 sec
+
+=== update 1 ===
+
+oops.
+
+I just attempted to load another set of wave 2 nodes and got a
+segfault from pantasks...
+
+I'm going to wait 5 min for jobs to complete, and then I'll restart pantasks...
+
+=== update 2 ===
+
+OK, I've restarted the stdscience pantasks, and I've bumped up the
+number of hosts to:
+
+1x1 3x2 4x3 4xC
+
+continuing to exclude ipp005 and ipp039 from processing.
+
+== report @ 9pm ==
+
+we've managed to get through 110 chips in the last 3 hours, compared
+with about 100 in the first 3 hours and 72 in the second three hours,
+so the rate is a bit faster now due to the larger number of machines.
+but, compared with the previous setup (1x1, 2x2, 3x3, 3xC) vs (1x1 3x2
+4x3 4xC), we have 15 + 15 + 17 = 32 more CPUs out of a previous total
+of 137 CPUs, or 25% more CPUs.  I don't think the processing is going
+25% faster that the first 3 hours (before the ipp039 bottleneck became
+a problem), but it is a little hard to judge.  If not, then it is
+evidence that we have still some I/O bottlenecks.  I'll look more
+closely at the times reported by the warp, diff, magic, and rc stages
+
+== report @ 9 am ==
+
+I've gone through most errors and found the following:
+
+* all of the chip failures were caused by the issue with ipp014 ->
+these would be addessed by fixing the nebulous randomization.
+* all but one of the warp failures were caused by the chmod problem ->
+these can be addressed with one of the options I listed
+* the remaining warp failure was a more traditional NFS I/O problem,
+and would be naturally cleaned up with the revert pass
+* the diff failures I checked seemed to be the traditional NFS I/O
+problems, but we do not seem to have a diff revert task -- that is
+surprising!
+
+there are also some destreak failures which I did not investigate.
+also, destreak advance was getting timeouts, which says either the
+query needs to be adjust or the timeout period in pantasks does.
+finally, dist_component.pl had some failures, which are probably NFS
+I/O issues as well.
+
+== throughput summary ==
+
+last night I proposed that we might be I/O bound since the processing
+rate did not seem to increase as much as I would have expected.  Here
+is evidence that I was wrong.  Attached is a plot of the number of
+completed 'camera' stage analyses per half hour during the test
+yesterday.  You can see that we start at about the 13/bin rate for the
+first period of the test, during which I had jobs running on 137
+nodes[1].  You can see the period around 6pm last night when I crashed
+pantasks and waited a bit to restart the system.  Then you can see the
+rate goes up to about 20/bin for the last part of the test, during
+which I had jobs running in 169 nodes.  So, adding 23% more CPU
+increased the rate by 54%.  hmm.  I think we are seeing two effects:
+the added CPUs and the reduced amount of nebulous collisions from the
+change in the skycell assignments.
+
+All in all, we are not in bad shape.  The green line indicates a good
+comfort level where we would be doing 600 exposures in 12 hours; the
+yellow line is 600 exposures in 18 hours, which is slower than we
+would like; the red line is bad: 600 exposures in 24 hours.  So, it
+looks like we are comfortably in the yellow to green range if we use
+the same collection of machines as last night.  We also have a bit
+more room for increased processing: there are several machines not
+currently in the list, and we probably could add another set of jobs
+to the wave 1 machines which have not been crashing.   There were also
+some I/O contention issues during this experiment that we have partly
+addressed, and more that can be improved on (ie, the nebulous
+randomization).
+
+[1] -- roughly : ipp005 and ipp039 came in and out of the list at
+different times.