Apache Tuning

The goal of this page is to give some details about experiments for Apache tuning in Nebulous use context. When many requests are sent to Nebulous, the Apache server displays messages in its log looking like:

[Thu Mar 10 09:11:09 2011] [info] server seems busy, (you may need to increase StartServers,
                           or Min/MaxSpareServers), spawning 8 children, there are 0 idle, 
                           and 128 total children
[Thu Mar 10 09:11:10 2011] [notice] child pid 6790 exit signal Segmentation fault (11)
[Thu Mar 10 09:11:10 2011] [notice] child pid 6791 exit signal Segmentation fault (11)
[Thu Mar 10 09:11:10 2011] [notice] child pid 6792 exit signal Segmentation fault (11)
[Thu Mar 10 09:11:10 2011] [notice] child pid 6793 exit signal Segmentation fault (11)

Such messages are seen when the server is overwhelmed by requests. The goal of the experiments is to determine the Apache configuration where such messages could be avoided.

Introduction

Apache Configuration

For convenience, the Apache log level has been set to debug (on ippdb00, it is set to warn).

The original mpm_prefork_module configuration for Apache is the following:

<IfModule mpm_prefork_module>
        StartServers            5
        MinSpareServers         5
        MaxSpareServers         10
        MaxClients              150
        MaxRequestsPerChild     10000
</IfModule>

Such a configuration will be written: [5, 5, 10, 150] (to avoid any confusion, if the MinSpareServers value was 6, the configuration would be denoted [5, 6, 10, 150]).

Constraints on the configuration can be seen in the ​prefork module configuration documentation.

Experiment Description

I wrote a small utility script that starts nbProcesses processes. Each process attempts to create nbFiles files in Nebulous using neb-touch. To ensure that files do not previously exist in Nebulous, each instance of the utility script is called with an index argument (let's call it experimentNb). Each experiment is run this way:

  python startNebFilesCreation.py <experimentNb> <nbProcesses> <nbFiles>

and creates <nbProcesses>*<nbFiles> files in Nebulous named:

  neb:///<a path to have a total path length of 200 bytes>/<experimentNb>/<processId>/<fileIdWithProcess>

The path length is adjusted through the <a path to have a total path length of 200 bytes> which is basically a sequence of "1234567890/" strings to have a total length of about 200 bytes which is the average size of a Nebulous instance entry.

Since an experiment depends on the Apache configuration, an experiment is fully described by (<experimentNb>, <nbProcesses>, <nbFiles>, <Apache Configuration>) where <Apache Configuration> is the 5-uple shown in Apache Configuration paragraph.

Operating Conditions

The experiments are performed on ippc01 where a local Nebulous database has been installed. To avoid any conflict with existing tools, a local neb-touch was used (URL: ​https://svn.pan-starrs.ifa.hawaii.edu/repo/ipp/tags/ipp-20110218/Nebulous/bin/neb-touch; Revision: 30846).

Conclusions

(since no one will read the details of experiments)

I recommend to change the number of StartServers to a large value, e.g. 50 and to set MaxClients to 256.

Setting MaxRequestsPerChild to 0 will deactivate it. If we observe memory leak, we can set it to a larger value than the current one.

Experiments

The template for "server seems busy" messages is:

[info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), 
       spawning ''<nbChildren>'' children, 
       there are ''<nbIdle>'' idle, and ''<nbTotal>'' total children

The values of (<nbChildren>, <nbIdle>, <nbTotal>) are provided when the message is observed.

Experiment 01

Original configuration: (01, 100, 10, [5, 5, 10, 150, 10000]). Apache was just restarted (/etc/init.d/apache2 restart)

10:52 No "server seems busy" message (and therefore no "Segmentation fault")

Experiment 02

No parameter change (but the experiment ID): (02, 100, 10, [5, 5, 10, 150, 10000]). Apache was not restarted.

10:53 4 "server seems busy messages" (values were (8, 0, 17),(16,0, 25), (32, 0, 41), (32, 0, 73)). No "Segmentation fault" observed.

Experiment 03

No parameter change (but the experiment ID): (03, 100, 10, [5, 5, 10, 150, 10000]). Apache was not restarted.

10:55 No "server seems busy" message (and therefore no "Segmentation fault")

Experiment 04

No parameter change (but the experiment ID): (04, 100, 10, [5, 5, 10, 150, 10000]). Apache was not restarted.

10:57 No "server seems busy" message (and therefore no "Segmentation fault")

Experiment 05

No parameter change (but the experiment ID): (05, 100, 10, [5, 5, 10, 150, 10000]). Apache was not restarted.

10:58 4 "server seems busy messages" (values: (8, 0, 20), (16,0,28), (32,0,44), (32,0,76)). No "Segmentation fault" observed

Conclusion It seems that when the delay between two experiments is short, the "server is busy" messages can be observed.

Experiments 06 and 07

No parameter change (but the experiment IDs): (06 and 07, 100, 10, [5, 5, 10, 150, 10000]). Apache was not restarted. 07 is run immediately after 06.

No messages observed for 06.

For 07: 4 "server seems busy messages" (values: (8, 0, 19), (16,0,27), (32,0,43), (32,0,75)). No "Segmentation fault" observed

Experiment 08

Number of processes was changed (was 100; is 150): (08, 150, 10, [5, 5, 10, 150, 10000]). Apache was not restarted.

3 "busy" ((8, 0, 28), (16,0,36), (32,0,52)). No seg faults.

Experiments 09 and 10

No change from 08. 10 was run immediately after 09. Configuration: (09-10, 150, 10, [5, 5, 10, 150, 10000]). Apache was not restarted.

09: No messages

10: 5 busy messages ((8,0,21),(16,0,29),(32,0,45),(32,0,77),(32,0,109)) then 15 segfaults, and then, 1 busy message (8,4,132) followed by 12 segfaults.

Experiment 11

Nb of processes changed from 150 to 200: (11, 200, 10, [5, 5, 10, 150, 10000]).

5 busy messages ((8,0,25),(16,0,33),(32,0,49),(32,0,81),(32,0,113)) then 17 segfaults messages.

Experiment 12

No change in configuration: (12, 200, 10, [5, 5, 10, 150, 10000]).

6 busy message ((8,0,16) then after 27 seconds (8,0,26),(16,0,34),(32,0,50),(32,0,82),(32,0,114)) then 17 segfaults messages.

Experiment 13

(13, 500, 100, [5, 5, 10, 150, 10000]).

As expected, a lot of busy (142) and segfaults (1523) messages. BUT all insertions were successfully performed. We do not lose any data (even if neb-touch complains that entries could not be created).

Experiment 14

Apache configuration changed: StartServers is 10 (was 5) (14, 200, 10, [10, 5, 19, 150, 10000]) Apache server restarted.

Exp. 14: 4 busy messages. No segfault.

Experiment 15

Same parameters as 14, i.e. (15, 200, 10, [10, 5, 19, 150, 10000]) but started immediately after 14 completed.

4 busy, 19 segfaults, 1 busy, 15 segfaults, 1 busy, 14 segfaults.

No real improvement regarding exp XX

Experiment 16

Lowered the processes number from 200 to 150 (16, 150, 10, [10, 5, 19, 150, 10000])

No busy, no segfaults (was run 10 minutes after exp. 15).

Experiment 17

Same parameters as 16 (17, 150, 10, [10, 5, 19, 150, 10000]). Run shortly after 16.

4 busy, no segfaults.

Experiment 18

Same parameters as 16 and 17 (18, 150, 10, [10, 5, 19, 150, 10000]). Run immediately after 17. 2 busy, no segfaults.

Conclusion Increasing the value of StartingServers apparently prevents segfaulting.

Experiment 19

Let us set the StartServers to something "large": (19, 150, 10, [50, 5, 10, 150, 10000]) Apache restarted.

No message.

Experiments 20 and 21

(Same conditions as 19): (20-21, 150, 10, [50, 5, 10, 150, 10000]) 21 was started immediately after 20.

5 busy messages. No segfault.

Experiment 22

(IPP started warps on the host) Experiment results discarded.

Experiment 23

Increase the number of processes (from 150 to 200): (23, 200, 10, [50, 5, 10, 150, 10000])

5 busy, then 16 segfaults.

Experiment 24

Same parameters as 23: (24, 200, 10, [50, 5, 10, 150, 10000])

4 busy, then 20 segfaults.

Experiment 25

(25, 200, 10, [50, 50, 51, 150, 10000])

Some busy, many segfaults. Got the following error message though: [Thu Mar 10 15:25:23 2011] [error] server reached MaxClients setting, consider raising the MaxClients setting

Experiment 26

(26, 200, 10, [50, 50, 51, 150, 10000]) Same parameters as 25 (I want to check if data are lost)

Before experiment: 86500 entries in instance table After experiment: 88500 entries (expected was 86500+200*10=88500) No data loss.

However I got only 2 busy messages, no segfaults and no error message as in 25.

Experiment 27

Oops.

Experiment 28

(28, -, -, [150, 150, 200, 250, 10000]) The script was not started.

I tried the previous configuration to see how apache was behaving.

When started, error log immediately shows segfaults! Doing nothing, busy and segfaults messages can be observed. The busy error message is: [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 8 children, there are 132 idle, and 132 total children

Experiment 29

(29, 200, 10, [150, 50, 200, 250, 10000]) I sat the MinSpareServers to a more sensible (well...) value.

The error log immediately shows segfaults when the apache server is started but after about 20 messages, steady state seems to be reached and no more message is displayed.

When the script is run, a few busy messages are displayed but a lot of segfaults (I didn't count but far much than in experiments 23-24)

Experiment 30

(30, 200, 10, [50, 50, 200, 250, 10000]) StartServers is set to a more sensible value.

No segfault messages in error log at apache start up.

Both busy and segfault messages.

Experiment 31

(31, 200, 10, [100, 50, 150, 256, 10000])

The number of MaxClients is maxed (i.e. 256).

No messages!

Experiment 32

(32, 200, 10, [100, 50, 150, 256, 10000]) Same parameters. Exp. 32 was started a few seconds after exp. 31.

No messages!

Experiment 33

(33, 200, 10, [100, 50, 150, 256, 10000]) Same parameters again. Exp. 33 was started immediately after exp. 32.

Busy and segfaults messages. No data loss.

Experiments 34, 35, and 36

Same parameters as 33.

I want to check the evolution of the number of tcp connections which are opened by the apache server (netstat -a | grep tcp | grep -v mysql | wc -l, and netstat -a | grep tcp | grep mysql | wc -l).

When apache is stopped:

38 tcp connections not counting mysql ones (mainly because I'm connected 7 times on this host) 1 for mysql (LISTEN state)

(when apache is stopped it is around 38).

When apache is running but the script is not started:

31 tcp connections not counting mysql ones 201 tcp mysql connections (all in ESTABLISHED state)

When the script is running (apache is running as well):

Maximum observed: 4011 tcp-non-mysql connections almost all in TIME_WAIT state mysql connections count are stable (max 215 even if the max_connections variable is set to 1024).

Most of them look like:

Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp6       0      0 ippc01.ifa.hawaii.:http ippc01.ifa.hawaii:34115 TIME_WAIT  

so clearly re-negociations of the apache server.

It would be worth to explicitely close the connection when the client disconnects instead of letting it die?

Experiment 37

I implemented disconnect methods both on the client and the server. I modified the neb-touch script in bin to call disconnect. I restarted the Apache server

(37, 1, 1, [100, 50, 150, 256, 10000])

Test the new neb-touch

Experiment 38

(38, 200, 10, [100, 50, 150, 256, 10000])

Busy messages and segfaults.

Experiments 39 and 40

(39-40, 150, 10, [100, 50, 150, 256, 10000]) 40 run immediately after 39 Both busy and segfaults messages in 40. No messages for 39.

Experiment 41

(41, 150, 10000, [100, 50, 150, 256, 10000]) The purpose of this experiment is to see if there is any data loss while the server is overwhelmed.

The instance table has 109502 entries at the beginning of the experiment.

Start: Fri Mar 11 14:22:41 HST 2011 End: