Changes between Initial Version and Version 1 of PS1_IPP_Czarlog_20190318

Timestamp:

Mar 18, 2019, 9:44:36 AM (7 years ago)

Author:

tdeboer

Comment:

--

PS1_IPP_Czarlog_20190318

@@ +1 @@
+== PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD == 
+[[PageOutline]]
+
+(Up to [wiki:PS1_IPP_CzarLogs PS1 IPP Czar Logs])
+
+=== Monday : 2019.03.18 ===
+
+ * TdB: In the morning (05:45), I noticed ippb18 was reported as down by nagios. When trying to get more information, I noticed neb-host was reporting too many connections and no machines where showing up in IPP monitor (all greyed out). Following a quick back and forth with Gene I then set all pantasks to stop using:
+{{{
+check_system.sh stop
+}}}
+  Furthermore, the compute nodes ippc71-75 were showing huge load on ganglia, so we turned off apache on those machines, using:
+{{{
+/etc/init.d/apache2 stop
+}}}
+  as root on each of the machines.
+
+ * Following this, we (John, Roger, Gene, myself) all came in at 09:00 to diagnose and fix the issue. First, we looked at the dump of processlist on ippc75:
+{{{
+mysql -h localhost -u root -e "show processlist" > neb.proc.20190318
+}}}
+  This showed 64 connections from ippc76, which is not on the list of nebservers (see ~ippitc/ippconfig/nebservers.csh) and also showed soem connections from ippc28.
+  Looking at the configuration file /etc/apache2/modules.d/apache2-mod_perl-startup.pl showed 256 MaxClients connections set up for the compute nodes. Machine ippc76 has StartServers = 64, so it will start with normally 64 connections, but could go up to 256 total. Combining all machines, this results in a max of 1349 connections, which is a lot. 
+
+ Looking at the max connections in MYSQL using:
+{{{
+select @@max_connections;
+}}}
+ shows that the maximum is set to 1024, which is less than 1349, so that could be a reason for the failure.
+ We are setting that higher now, to avoid this issue:
+{{{
+SET GLOBAL max_connections = 2048 (in mysql)
+}}}
+
+ The full reason for the break is not totally clear, but the huge load spikes on the apache servers roughly coincide with ippb18 going down, so that might well be related.
+
+ For now, we are setting the apache servers back to on, using:
+{{{
+/etc/init.d/apache2 start
+}}}
+ as root on each of the machines.
+
+ Finally, the pantasks, are set back to run using:
+{{{
+check_system.sh run
+}}}
+
+=== Tuesday : 2019.03.19 ===
+
+=== Wednesday : 2019.03.20 ===
+
+=== Thursday : 2019.03.21 ===
+
+=== Friday : 2019.03.22 ===
+
+=== Saturday : 2019.03.23 ===
+
+=== Sunday : 2019.03.24 ===