Changes between Version 38 and Version 39 of PS1_IPP_CzarLog_20110808

Timestamp:

Aug 14, 2011, 1:22:39 PM (15 years ago)

Author:

eugene

Comment:

--

PS1_IPP_CzarLog_20110808

@@ -78 +78 @@
  * 18:30 Mark: set MD10.GR0 chips to goto_cleaned to start clearing space for MD01.
 
-=== Saturday : YYYY.MM.DD ===
+=== Saturday : 2011-08-13 ===
  * 02:00 Mark: queued up MD01-y chip to warp for new reference stack (little nightly data and M31 mostly through).
  * 15:00 Mark: MD01 chip failing, OTA file 0 size. Copied over with good file with
@@ -86 +86 @@
  * 17:45 Mark: warps continuing to stall again around 2pm? did a warp.reset and seems to have finished the one M31.nightly remaining and running the MD01.GR0 now.
 
-=== Sunday : YYYY.MM.DD ===
+=== Sunday : 2011-08-14 ===
+ * 13:15 EAM : around 10:30 I noticed that processing was stuck.  It seemed that we were getting many nebulous related errors.  I discovered some possible problems with the /tmp/nebulous_server.log files and the apache access_log files that could have been causing this.  I stopped all processing and shutdown the neb apache servers on ippc01-09.  I modified the config for apache to limit MaxClients to 127, moved /var/log/apache2 to /export/$HOSTNAME/apache2, moved access_log to a save name and recreated it with 0 size, and moved the /tmp/nebulous_server.log to /var/log/apache2 (linked to /tmp), and zeroed it out as well.  I also found a few zombie jobs on some machines and a hung nfs mount on ippc17.  I restarted all processing at about 13:00.
 
+
+
+