PS1 IPP Czar Logs for the week YYYY.MM.DD - YYYY.MM.DD

(Up to PS1 IPP Czar Logs)

Monday : 2019.03.18

• TdB: In the morning (05:45), I noticed ippb18 was reported as down by nagios. When trying to get more information, I noticed neb-host was reporting too many connections and no machines where showing up in IPP monitor (all greyed out). Following a quick back and forth with Gene I then set all pantasks to stop using:

  check_system.sh stop
  

  Furthermore, the compute nodes ippc71-75 were showing huge load on ganglia, so we turned off apache on those machines, using:

  /etc/init.d/apache2 stop
  

  as root on each of the machines.

• Following this, we (John, Roger, Gene, myself) all came in at 09:00 to diagnose and fix the issue. First, we looked at the dump of processlist on ippc75:

  mysql -h localhost -u root -e "show processlist" > neb.proc.20190318
  

  This showed 64 connections from ippc76, which is not on the list of nebservers (see ~ippitc/ippconfig/nebservers.csh) and also showed soem connections from ippc28. Looking at the configuration file /etc/apache2/modules.d/apache2-mod_perl-startup.pl showed 256 MaxClients connections set up for the compute nodes. Machine ippc76 has StartServers = 64, so it will start with normally 64 connections, but could go up to 256 total. Combining all machines, this results in a max of 1349 connections, which is a lot.

Looking at the max connections in MYSQL using:

select @@max_connections;

shows that the maximum is set to 1024, which is less than 1349, so that could be a reason for the failure. We are setting that higher now, to avoid this issue:

SET GLOBAL max_connections = 2048 (in mysql)

The full reason for the break is not totally clear, but the huge load spikes on the apache servers roughly coincide with ippb18 going down, so that might well be related.

For now, we are setting the apache servers back to on, using:

/etc/init.d/apache2 start

as root on each of the machines.

Finally, the pantasks, are set back to run using:

check_system.sh run

• MEH: bulk updates now continuing again in ~ippmops/stdscience

Tuesday : 2019.03.19

Wednesday : 2019.03.20

Thursday : 2019.03.21

Friday : 2019.03.22

Saturday : 2019.03.23

Sunday : 2019.03.24