Changes between Version 4 and Version 5 of ThroughputDemo_20091104

Timestamp:

Nov 6, 2009, 10:58:52 AM (17 years ago)

Author:

eugene

Comment:

--

ThroughputDemo_20091104

@@ -118 +118 @@
 == report @ 9 am ==
 
+processing effectively stopped around 6am this morning, though the
+full analysis was not complete:
+
+ * there are a number of chipRuns that have not finished: 37 exposures
+are still pending.  This is basically because of point 2:
+
+ * ipp014 is unhappy: it is not so unhappy that it has been kicked out
+of nebulous, but files on it are giving I/O errors.  The failing chips
+are being caused by this, but coupled to a lingering nebulous error:
+the rules are forcing nebulous to return the ipp014 instance rather
+than randomize among the instances.  i'm going to reboot ipp014 and
+kick it out of nebulous for now.
+
+I'm attaching a snapshot of the ganglia load for the past 24 hours.
+you can see where the test started a bit before noon yesterday with
+the initial set of machines, and then the point where I crashed
+pantasks and restarted with and additional set of wave 2, 3, and C
+machines.  you can also see the problem machines caused by the
+nebulous distribution (ipp039 and a couple of the ipp04X machines),
+and the ones that have been giving occasional nfs issues (eg ipp016).
+I'm not certain if the latter issues are being cause by glitches in
+those raids, something equivalent to the causes of the ipp005 crashes,
+or because of poor data distribution.  this is something we need to
+investigate.
+
+=== update ===
+
 I've gone through most errors and found the following: