= Apache Tuning =

[[PageOutline]]

The goal of this page is to give some details about experiments for Apache tuning in Nebulous use context. When many requests are sent to Nebulous, the Apache server displays messages in its log looking like:

{{{
[Thu Mar 10 09:11:09 2011] [info] server seems busy, (you may need to increase StartServers,
                           or Min/MaxSpareServers), spawning 8 children, there are 0 idle, 
                           and 128 total children
[Thu Mar 10 09:11:10 2011] [notice] child pid 6790 exit signal Segmentation fault (11)
[Thu Mar 10 09:11:10 2011] [notice] child pid 6791 exit signal Segmentation fault (11)
[Thu Mar 10 09:11:10 2011] [notice] child pid 6792 exit signal Segmentation fault (11)
[Thu Mar 10 09:11:10 2011] [notice] child pid 6793 exit signal Segmentation fault (11)
}}}

Such messages are seen when the server is overwhelmed by requests. The goal of the experiments is to determine the Apache configuration where such messages could be avoided.
== Introduction ==

=== Apache Configuration ===
For convenience, the Apache log level has been set to debug (on ippdb00, it is set to warn).

The original mpm_prefork_module configuration for Apache is the following:
{{{
<IfModule mpm_prefork_module>
        StartServers            5
        MinSpareServers         5
        MaxSpareServers         10
        MaxClients              150
        MaxRequestsPerChild     10000
</IfModule>
}}}

Such a configuration will be written: '''[5, 5, 10, 150]''' (to avoid any confusion, if the MinSpareServers value was 6, the configuration would be denoted [5, 6, 10, 150]).

Constraints on the configuration can be seen in the [http://httpd.apache.org/docs/2.0/mod/prefork.html prefork] module configuration documentation.
=== Experiment Description ===
I wrote a small utility script that starts ''nbProcesses'' processes. Each process attempts to create ''nbFiles'' files in Nebulous using ''neb-touch''. To ensure that files do not previously exist in Nebulous, each instance of the utility script is called with an index argument (let's call it ''experimentNb''). Each experiment is run this way:
{{{
  python startNebFilesCreation.py <experimentNb> <nbProcesses> <nbFiles>
}}}
and creates ''<nbProcesses>*<nbFiles>'' files in Nebulous named:
{{{
  neb:///<a path to have a total path length of 200 bytes>/<experimentNb>/<processId>/<fileIdWithProcess>
}}}
The path length is adjusted through the ''<a path to have a total path length of 200 bytes>'' which is basically a sequence of "1234567890/" strings to have a total length of about 200 bytes which is the average size of a Nebulous instance entry.

Since an experiment depends on the Apache configuration, an experiment is fully described by (<experimentNb>, <nbProcesses>, <nbFiles>, <Apache Configuration>) where <Apache Configuration> is the 5-uple shown in [wiki:ApacheTuning#ApacheConfiguration Apache Configuration paragraph].
=== Operating Conditions ===
The experiments are performed on ippc01 where a local Nebulous database has been installed. To avoid any conflict with existing tools, a local ''neb-touch'' was used (URL: https://svn.pan-starrs.ifa.hawaii.edu/repo/ipp/tags/ipp-20110218/Nebulous/bin/neb-touch; Revision: 30846).

== Experiments ==

The template for "server seems busy" messages is: 
{{{
[info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), 
       spawning ''<nbChildren>'' children, 
       there are ''<nbIdle>'' idle, and ''<nbTotal>'' total children
}}}
The values of (<nbChildren>, <nbIdle>, <nbTotal>) are provided when the message is observed.
=== Experiment 01 ===
Original configuration: (01, 100, 10, [5, 5, 10, 150, 10000]).
Apache was just restarted (/etc/init.d/apache2 restart)

10:52 No "server seems busy" message (and therefore no "Segmentation fault")

=== Experiment 02 ===
No parameter change (but the experiment ID): (02, 100, 10, [5, 5, 10, 150, 10000]).
Apache was not restarted.

10:53 4 "server seems busy messages" (values were (8, 0, 17),(16,0, 25), (32, 0, 41), (32, 0, 73)). No "Segmentation fault" observed.

=== Experiment 03 ===
No parameter change (but the experiment ID): (03, 100, 10, [5, 5, 10, 150, 10000]).
Apache was not restarted.

10:55 No "server seems busy" message (and therefore no "Segmentation fault")

=== Experiment 04 ===
No parameter change (but the experiment ID): (04, 100, 10, [5, 5, 10, 150, 10000]).
Apache was not restarted.

10:57 No "server seems busy" message (and therefore no "Segmentation fault")

=== Experiment 05 ===
No parameter change (but the experiment ID): (05, 100, 10, [5, 5, 10, 150, 10000]).
Apache was not restarted.

10:58 4 "server seems busy messages" (values: (8, 0, 20), (16,0,28), (32,0,44), (32,0,76)). No "Segmentation fault" observed

'''''Conclusion 1''''' It seems that when the delay between two experiments is short, the "server is busy" messages can be observed.

=== Experiments 06 and 07 ===
No parameter change (but the experiment IDs): (06 and 07, 100, 10, [5, 5, 10, 150, 10000]).
Apache was not restarted.
07 is run immediately after 06.

No messages observed for 06.

For 07: 4 "server seems busy messages" (values: (8, 0, 19), (16,0,27), (32,0,43), (32,0,75)). No "Segmentation fault" observed

=== Experiment 08 ===
Number of processes was changed (was 100; is 150): (08, '''150''', 10, [5, 5, 10, 150, 10000]).
Apache was not restarted.

3 "busy" ((8, 0, 28), (16,0,36), (32,0,52)). No seg faults.

=== Experiments 09 and 10 ===
No change from 08. 10 was run immediately after 09.
Configuration: (09-10, 150, 10, [5, 5, 10, 150, 10000]).
Apache was not restarted.

09: No messages

10: 5 busy messages ((8,0,21),(16,0,29),(32,0,45),(32,0,77),(32,0,109)) then 15 segfaults, and then, 1 busy message (8,4,132) followed by 12 segfaults.

=== Experiment 11 ===
Nb of processes changed from 150 to 200: (11, '''200''', 10, [5, 5, 10, 150, 10000]).

5 busy messages ((8,0,25),(16,0,33),(32,0,49),(32,0,81),(32,0,113)) then 17 segfaults messages.

=== Experiment 12 ===
No change in configuration: (12, 200, 10, [5, 5, 10, 150, 10000]).

6 busy message ((8,0,16) then after 27 seconds (8,0,26),(16,0,34),(32,0,50),(32,0,82),(32,0,114)) then 17 segfaults messages.

=== Experiment 13 ===
(13, '''500''', '''100''', [5, 5, 10, 150, 10000]).

As expected, a lot of busy (142) and segfaults (1523) messages. '''BUT''' all insertions were successfully performed.  We do not lose any data (even if neb-touch complains that entries could not be created).

=== Experiment 14  ===
Apache configuration changed: StartServers is 10 (was 5) (14, 200, 10, ['''10''', 5, 19, 150, 10000])
Apache server restarted.

Exp. 14: 4 busy messages. No segfault.

=== Experiment 15 ===
Same parameters as 14, i.e. (15, 200, 10, [10, 5, 19, 150, 10000]) but started immediately after 14 completed.

4 busy, 19 segfaults, 1 busy, 15 segfaults, 1 busy, 14 segfaults.

No real improvement regarding exp XX

=== Experiment 16 ===
Lowered the processes number from 200 to 150 (16, '''150''', 10, [10, 5, 19, 150, 10000])

No busy, no segfaults (was run 10 minutes after exp. 15).

=== Experiment 17 ===
Same parameters as 16 (17, 150, 10, [10, 5, 19, 150, 10000]). Run shortly after 16.

4 busy, no segfaults.

=== Experiment 18 ===
Same parameters as 16 and 17 (18, 150, 10, [10, 5, 19, 150, 10000]). Run immediately after 17.
2 busy, no segfaults.

'''''Conclusion''''' Raising the number of starting servers apparently prevents segfaulting.

=== Experiment 19 ===