Details about Apache Benchmarking Tool

Couple of days were really struggling. Apache Benchmarking tool gave me real hard time. There is only one page manual explaining the parameters and literally no documentation on how to interpret result.

The usual AB output looks something like below.
=================================================
This is ApacheBench, Version 2.0.41-dev <$Revision: 1.141 $> apache-2.0
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/

Benchmarking epglocal.mystro.mystrotv.com (be patient)


Server Software: Jetty(8.0.4.v20111024)
Server Hostname: HOST_NAME
Server Port: 8080

Document Path: YOUR_URL
Document Length: 1397 bytes

Concurrency Level: 4000
Time taken for tests: 1.649018 seconds
Complete requests: 12000
Failed requests: 0
Write errors: 0
Non-2xx responses: 15868
Total transferred: 19339416 bytes
HTML transferred: 16801719 bytes
Requests per second: 7277.06 [#/sec] (mean)
Time per request: 549.673 [ms] (mean)
Time per request: 0.137 [ms] (mean, across all concurrent requests)
Transfer rate: 11452.88 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 23 38.1 1 120
Processing: 2 134 140.0 125 861
Waiting: 1 131 140.3 118 860
Total: 3 158 163.0 175 962

Percentage of the requests served within a certain time (ms)
50% 175
66% 225
75% 244
80% 266
90% 361
95% 434
98% 481
99% 617
100% 962 (longest request)
===============================================
AB tool results seem simple on first look but when we started understanding the numbers little bit more, we started noticing issues with them. For example the response time calculated by AB tool for each request is "total time taken/total number of requests". Whereas the response time should be calculated based on the time taken by each request. Since in multiprocessor environment there are more than one request getting processed at a given point of time.

As per AB if a female delivers triplet in 9 months then a baby can be delivered in 3 months, which you and I know it's not possible. :)
AB formula of calculating the mean will give us deceptive results, so instead we relied on gnuplot to get meaningful results. The connect time (standard deviation) was coming close to what firefox was showing for each request. After plotting the gnuplot data things started to make little sense.

So if you want to know really how much time each request is taking do plot the gnuplot data which can be retrieved by -g option in ab. These number would be little close to real world.

Failure also had interesting behavior. If you will notice I fired total number of 12K request but there were 15K Non 2xx responses i.e. those response failed because of some reason but failure section is showing zero. I was so confused.

AB reports failure based on length and exceptions. If the length of response does not match with first one then it starts recording those request as failure. In my case rest service should return same response hence it's worked.

Another reason for failure can be any request which does not come back within specified timeout is also recorded as failure. Hence many time I noticed failure but had no exceptions or error logged any where.

Still could not resolve why sometime result shows failure and it passed next time when I try to run.

I was also getting all weird errors most the time. .Connection reset by peers, Cann't assign address, Connection timeout. I will not see any exceptions in the logs neither jetty nor applications. To prove that it's OS related, decided to look into source code.

Ab tool uses Apache Portable Runtime library for all it's call. here is a very nice link which helped me a lot. http://apr.apache.org/docs/apr/0.9/group__apr__network__io.html#g945bdbe807ec5635d65a6fd9ddb78c29.

After going through the code it was sure that all those errors were related to socket i.e. OS errors which AB was wrapping and throwing back in apr_status variable.

Lots of information in one blog. I will try splitting in in future.
Manisha

"socket: Too many open files (24) Error" and "apr_socket_recv: Connection timed out (110)" Errors

I continued with performance testing of my Rest services and had another interesting day :).

I was trying to run the Apache benchmarking tool (AB Tool) for total number of 10,000 requests with 1000 concurrent request. My services behaved like real gentleman. But soon after I bumped the 1000 concurrent requests to 1500 concurrent requests, I started getting "socket: Too many open files (24) Error" error.

After little bit of googling realized that I am hitting the linux systems open file limit. I ran "ulimit -a" command to see the list of provided resources available to the shell and to the processes started by it. Then I notices that limit for open files (ulimit -n) is set to 1024. I edited the limit by running "ulimit -n 5000" command and could get over this error.

You might have to add a new line under /etc/profile to make this change permanent . The changes for ulimit hold good only for running shell.

Time for Samba dance again. Soon I wanted to test my services for 100,000 total number of request and guess what I ran into another error. :(. "apr_socket_recv: Connection timed out (110)" . :( . Realized that another OS property has to be modified to accomodate that many request. somaxconn.

To view the existing setting , please look into /proc/sys/net/core/somaxconn file and to change max connection parameter from default 128 to 10240, you need to add “net.core.somaxconn = 10240” line in /etc/sysctl.conf file. Then run “/sbin/sysctl -p /etc/sysctl.conf” command. Now the /proc/sys/net/core/somaxconn file should have our new number. Now the /proc/sys/net/core/somaxconn file should have our new number.
echo "10240" > /proc/sys/net/core/somaxconn command will change the settings temporarily.

Now I am being able to fire 5000 concurrent request for total number of 5000000 request. Happy again.

Hope it helps somebody
Manisha