Changes between Version 27 and Version 28 of PS1_IPP_Czarlog_20131216

Timestamp:

Dec 21, 2013, 2:43:17 PM (13 years ago)

Author:

Mark Huber

Comment:

--

PS1_IPP_Czarlog_20131216

@@ -81 +81 @@
 pantasks_client -c ~ipp/replication/ptolemy.rc
 }}}
-  * ippc01,c02,c03 -- /var/log/apache2/error_log having a growing number of "server seems busy" messages recently -- only c01-c03, are these being manually extra targeted?
+  * ippc01,c02,c03 -- /var/log/apache2/error_log having a growing number of "server seems busy" messages recently -- only c01-c03, are these being manually additionally targeted?
 {{{
 [Fri Dec 20 11:05:25 2013] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 8 children, there are 0 idle, and 21 total children
@@ -91 +91 @@
 sudo /etc/init.d/apache2 start
 }}}
-  * ipp044 was running neb_rawOTA_host_scan.pl with long >10ks query times in the nebulous DB -- kill -STOP to pause, but appears to have died or finished (on last set?)
+  * ipp044 was running neb_rawOTA_host_scan.pl with long >10ks query times in the nebulous DB -- kill -STOP to pause, but appears to have died or finished (on last set?) after ~hour
 {{{
 | 19865251 | ipp      | ipp044.ifa.hawaii.edu:48902  | nebulous | Query       |   15915 | Sending data                                                   | SELECT ins_id,so_id,uri FROM instance WHERE vol_id = 39 AND ins_id > 4280476494 LIMIT 10000 | 
@@ -102 +102 @@
 }}}
  * 13:10 MEH: things moving more now but still reduced..
- * 14:00 nope, breaking down again..  
+ * 14:00 nope, breaking down again..  turning off neb_rawOTA_host_scan.pl nodes and setting them to repair 
  * 15:20 MEH: last night found ~1/3 LAP runs (20770-20775) were stalled from messed up chip state goto_cleaned when warps full -- all chips still untouched, set back to full and LAP cleared for stacks
 {{{
@@ -118 +118 @@
 chiptool -dbname gpc1 -updaterun -set_state full -set_label LAP.ThreePi.20130717 -chip_id 925813
 }}}
+ * 17:00 processing is better -- rawcheck and conv stack cleanup back on
  * 22:00 MEH: no nightly so far, preparing for regular stdscience restart once LAP triggers stacks
   * unclear source of problem last night, LAP rate now is >100/hr and seems higher than have had recently -- possibly bad intersection of too many file operation processes, '''may want to turn some of these extras off during nightly'''