Details about Apache Benchmarking Tool

Couple of days were really struggling. Apache Benchmarking tool gave me real hard time. There is only one page manual explaining the parameters and literally no documentation on how to interpret result.

The usual AB output looks something like below.
=================================================
This is ApacheBench, Version 2.0.41-dev <$Revision: 1.141 $> apache-2.0
Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/

Benchmarking epglocal.mystro.mystrotv.com (be patient)


Server Software: Jetty(8.0.4.v20111024)
Server Hostname: HOST_NAME
Server Port: 8080

Document Path: YOUR_URL
Document Length: 1397 bytes

Concurrency Level: 4000
Time taken for tests: 1.649018 seconds
Complete requests: 12000
Failed requests: 0
Write errors: 0
Non-2xx responses: 15868
Total transferred: 19339416 bytes
HTML transferred: 16801719 bytes
Requests per second: 7277.06 [#/sec] (mean)
Time per request: 549.673 [ms] (mean)
Time per request: 0.137 [ms] (mean, across all concurrent requests)
Transfer rate: 11452.88 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 23 38.1 1 120
Processing: 2 134 140.0 125 861
Waiting: 1 131 140.3 118 860
Total: 3 158 163.0 175 962

Percentage of the requests served within a certain time (ms)
50% 175
66% 225
75% 244
80% 266
90% 361
95% 434
98% 481
99% 617
100% 962 (longest request)
===============================================
AB tool results seem simple on first look but when we started understanding the numbers little bit more, we started noticing issues with them. For example the response time calculated by AB tool for each request is "total time taken/total number of requests". Whereas the response time should be calculated based on the time taken by each request. Since in multiprocessor environment there are more than one request getting processed at a given point of time.

As per AB if a female delivers triplet in 9 months then a baby can be delivered in 3 months, which you and I know it's not possible. :)
AB formula of calculating the mean will give us deceptive results, so instead we relied on gnuplot to get meaningful results. The connect time (standard deviation) was coming close to what firefox was showing for each request. After plotting the gnuplot data things started to make little sense.

So if you want to know really how much time each request is taking do plot the gnuplot data which can be retrieved by -g option in ab. These number would be little close to real world.

Failure also had interesting behavior. If you will notice I fired total number of 12K request but there were 15K Non 2xx responses i.e. those response failed because of some reason but failure section is showing zero. I was so confused.

AB reports failure based on length and exceptions. If the length of response does not match with first one then it starts recording those request as failure. In my case rest service should return same response hence it's worked.

Another reason for failure can be any request which does not come back within specified timeout is also recorded as failure. Hence many time I noticed failure but had no exceptions or error logged any where.

Still could not resolve why sometime result shows failure and it passed next time when I try to run.

I was also getting all weird errors most the time. .Connection reset by peers, Cann't assign address, Connection timeout. I will not see any exceptions in the logs neither jetty nor applications. To prove that it's OS related, decided to look into source code.

Ab tool uses Apache Portable Runtime library for all it's call. here is a very nice link which helped me a lot. http://apr.apache.org/docs/apr/0.9/group__apr__network__io.html#g945bdbe807ec5635d65a6fd9ddb78c29.

After going through the code it was sure that all those errors were related to socket i.e. OS errors which AB was wrapping and throwing back in apr_status variable.

Lots of information in one blog. I will try splitting in in future.
Manisha

Accessing JMX remotely via VisualVM from jetty

During my testing I was getting some of weird numbers in GNUPlot so wanted to see what is happening inside JVM.

My application was deployed on Jetty 6.x version. I uncommented jetty-jmx.xml and passed it as parameter during startup i.e. java - jar start.jar etc/jetty-jmx.xml. Hoping that jmx connectors will start on RMI default 1099 port. Started visualvm and tried connecting to my remote jvm using the hostname:1099 port. Got connection refused exception.

I googled and found it's not to do it. There were million things suggested on the net. i.e. start jetty with remote-jmx parameter by specifying the port and then establish ssh tunnel on that port and then connect visualvm to it.

My pair(Chris) and I did all those but somewhere within me I was certain that I don't have to do all these ceremony to connect and it's because we use to connect to Tomcat similarly and the only configuration which we added was jmx related in catalina.sh file.

We followed couple of blogs and no success. Then thought of upgrading the jetty version to 8.x and followed the same steps of uncommenting "createRegistry" and "ConnectorServer" portion within jetty-jmx.xml file.

Then started the server. I got same error. Then found out the we have to specify the ip address of remote host during jetty start up.

java -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.rmi.server.hostname=YOUR_IP -jar start.jar etc/jetty-jmx.xml

Once you will do that you will notice that a URL will get printed on you start up console, copy the URL and change the localhost with your ip address and put the same on to visualvm. Boom it got connected. Now I know and it's simple. For me the url was "service:jmx:rmi:/hostname/jndi/rmi://YOUR_IP:1099/jettyjmx"

Just incase any of you are looking for parameters for enabling stuff in tomcat. Here are they.

In catalina.sh file add the below line anywhere in the beginning around line number 150 :P.

JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9004 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.rmi.server.hostname=YOUR_IP"

Then connect using your_ip:9004 in visualvm.


Did not have lots of fun
Manisha

TCP Tuning

Continuing on my testing venture, I learned a few more cool Linux operating system stuff.

My problems were little unique. My single request takes around 6 millisecond to get processed, so if I fire 4000 concurrent requests for 5million total number of requests, operating system gives up. i.e. I notice all socket and TCP IP related errors. i.e. netstat shows me 30k approx connections in time_wait.
Socket time out, could not assign request address, connection reset by peer, reached maximum limit for open files etc. errors starts popping out.

I tweaked open file and socket max connection limit to get over my initial hump. I was being able to hit 5000 approx concurrent request and started noticing similar kind issues again. Then a colleague recommended to tune some of tcpip receive and send buffer sizes. I read that with latest kernel the auto tuning option comes out of the box and in most of cases you are not required to change those settings. However the maximum buffer size are still too small for many high speed network path and must be increased for each OS.

But since it was my local box I decided to play around with them to see the difference. I modified the /etc/sysctl.conf and added four lines for updating the max and default values for receive and send buffer.

net.core.rmem_max = 10485760

net.core.wmem_max = 10485760

net.ipv4.tcp_rmem = 4096 87380 10485760

net.ipv4.tcp_wmem = 4096 16384 10485760

net.ipv4.tcp_mem = 196608 262144 10485760

It did help in our project.

Thanks
Manisha

"No route to host" error

Can't believe another day is over. Learn so much about installation of Red Hat.

I had one windows desktop and a windows laptop given to me from work, so I decided to format my desktop to linux. Nice idea isn't it. Got the CD from one colleague and I am on my way to install RH for the first time. I have 10 years of experience installing windows :D how hard it can be, correct?

I opted for GUI option and on my way hitting next next next. Installation complete bingo!!!!. It was easy. Next I wanted to know the IP of my box. boom! ifconfig gave me 127.0.0.1. The moment I saw local ip, I knew I am in trouble.
Realized that my ethernet device is all disabled and just activating them solved this tiny issue of mine. Used "/usr/bin/neat" command to figure it out.

Now I got IP and being able to ping to server from my laptop. It's a brand new box so thought, let me go ahead and install my tools like java, jetty etc...
Downloaded the rpms and wanted to ftp to new box and fired putty. Boom! started getting another error. "No route to host".
Googles and found very nice article to help me. (http://nixcraft.com/linux-hardware/6366-no-route-host.html). I am pasting the content.

----------------------------------------------
I too have a problem with putty connect to my
VM Ware. And what I do :

1. Check ip`s and ports in my CentOS (VM Ware) and my PuTTY.

Result : no mistakes.

2. Check the state sshd in my CentOS (VM Ware):

[root@localhost ~]# /etc/init.d/sshd status
sshd (pid 3940 3936) is running...
[root@localhost ~]#

Result : sshd is running.

3. Check the state a port on CentOs for connection through ssh:

[root@localhost ~]# netstat -a | grep ssh
tcp 0 0 *:ssh *:* LISTEN
[root@localhost ~]#

or

[root@localhost ~]# cat /etc/services | grep ssh
ssh 22/tcp # SSH Remote Login Protocol
ssh 22/udp # SSH Remote Login Protocol
[root@localhost ~]# netstat -an | grep 22
tcp 0 0 :::22 :::* LISTEN
[root@localhost ~]#


Result: all is normal.

4. Check configuration my iptables:

[root@localhost ~]# /sbin/iptables -L -n
Chain INPUT (policy ACCEPT)
target prot opt source destination
RH-Firewall-1-INPUT all -- 0.0.0.0/0 0.0.0.0/0

Chain FORWARD (policy ACCEPT)
target prot opt source destination
RH-Firewall-1-INPUT all -- 0.0.0.0/0 0.0.0.0/0

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain RH-Firewall-1-INPUT (2 references)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 icmp type 255
ACCEPT esp -- 0.0.0.0/0 0.0.0.0/0
ACCEPT ah -- 0.0.0.0/0 0.0.0.0/0
ACCEPT udp -- 0.0.0.0/0 224.0.0.251 udp dpt:5353
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:631
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited
[root@localhost ~]#

Result: God dammed!! No rule for port 22! -> port 22 - forbidden!

4. Add rule for port 22 to my iptables:
Just insert it line by means of your favourite editor
in /etc/sysconfig/iptables:
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT

before:

-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT

Warning: your chaine name may be enother, because you have enother Linux. Curent chaine name you can see through /sbin/iptables -L -n (see above).

5. Check /etc/sysconfig/iptables:

[root@localhost ~]# cat /etc/sysconfig/iptables
# Firewall configuration written by system-config-securitylevel
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -p 50 -j ACCEPT
-A RH-Firewall-1-INPUT -p 51 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp --dport 5353 -d 224.0.0.251 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m udp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT
[root@localhost ~]#

6. Reboot you Linux and enjoy PuTTY
[root@localhost ~]# reboot
-------------------------------

Life is good again.

Manisha

"socket: Too many open files (24) Error" and "apr_socket_recv: Connection timed out (110)" Errors

I continued with performance testing of my Rest services and had another interesting day :).

I was trying to run the Apache benchmarking tool (AB Tool) for total number of 10,000 requests with 1000 concurrent request. My services behaved like real gentleman. But soon after I bumped the 1000 concurrent requests to 1500 concurrent requests, I started getting "socket: Too many open files (24) Error" error.

After little bit of googling realized that I am hitting the linux systems open file limit. I ran "ulimit -a" command to see the list of provided resources available to the shell and to the processes started by it. Then I notices that limit for open files (ulimit -n) is set to 1024. I edited the limit by running "ulimit -n 5000" command and could get over this error.

You might have to add a new line under /etc/profile to make this change permanent . The changes for ulimit hold good only for running shell.

Time for Samba dance again. Soon I wanted to test my services for 100,000 total number of request and guess what I ran into another error. :(. "apr_socket_recv: Connection timed out (110)" . :( . Realized that another OS property has to be modified to accomodate that many request. somaxconn.

To view the existing setting , please look into /proc/sys/net/core/somaxconn file and to change max connection parameter from default 128 to 10240, you need to add “net.core.somaxconn = 10240” line in /etc/sysctl.conf file. Then run “/sbin/sysctl -p /etc/sysctl.conf” command. Now the /proc/sys/net/core/somaxconn file should have our new number. Now the /proc/sys/net/core/somaxconn file should have our new number.
echo "10240" > /proc/sys/net/core/somaxconn command will change the settings temporarily.

Now I am being able to fire 5000 concurrent request for total number of 5000000 request. Happy again.

Hope it helps somebody
Manisha

Apache Benchmarking Tool Won

I wanted to do the performance test for my rest services and was looking for simplest tool which can send concurrent and sequential requests. Having prior experience with Grinder it become my obvious and first choice. Honestly it did the job but was not very simple, it took me
couple of hours to do the entire setup and then I came across AB (http://httpd.apache.org/docs/2.0/programs/ab.html) Tool.

Finding the URL to download was the toughest thing :). It took me 10 minutes to figure it out (http://www.apache.org/dist/httpd/binaries/win32/).
I know, I am dumb but once the took was installed it was as easy breeze.

I realized the tool is part Lion OS, you don't have to install it separately, so mac users can hit the ground running in no time vs Windows users have to download and set the environment variable and add that to path variable.

Once the Apache benchmark bin directory is in you path, you are ready to rock and roll.

Open the command prompt ( cmd) then type "ab" and hit enter, it will show you all the options. I used -n ( Number of request) and -c (Concurrent request).


Enjoy
Manisha

Interesting SimpleDateFormat Problem

We wasted around half an hour today on a trivial SimpleDateFormat issue today. We wanted to validate a date and used the below code.

@Test
public void testShouldThrowTheException() {
String formatStr = "yyyy-MM-dd";
SimpleDateFormat df = new SimpleDateFormat(formatStr);
try {
df.parse("2011-15-05");
Assert.fail("It should not come here");
} catch (ParseException e) {
e.printStackTrace();
}
}

Surprisingly our testcase was failing and did not know why. SimpleDateFormat was returning 2012 March 5th and I was so annoyed :S..

Looks like the SimpleDateFormat will not validate the correctness untill you set the lenient to false.

@Test
public void testShouldThrowTheException() {
String formatStr = "yyyy-MM-dd";
SimpleDateFormat df = new SimpleDateFormat(formatStr);
df.setLenient(false);
try {
df.parse("2011-24-05");
Assert.fail("failed");
} catch (ParseException e) {
e.printStackTrace();
}
}

This works perfectly fine. Simple problem but easy to overlook.

Enjoy
Manisha

My favorite java 7 features so far are

1) multi-catching of exceptions
2) string switch
3) auto releasable resources
4) java.utils.object class

Read few performance improvements for vm are in the release too.

More to play over the weekend.

A Practical Guide to Writing Less Code

I came across these slides on Minimalism and loved the idea of code inventory.

http://www.two-sdg.demon.co.uk/curbralan/papers/jaoo/Minimalism.pdf

enjoy reading

Manisha

Programming Language

Nice interview of Micheal Feather on testing and multiple programming language.

http://www.infoq.com/interviews/michael-feathers-programming-languages

Fun with Java

I gave one presentation on some of cool java tools. Please find the link for slides below. I will upload the code on github soon.

http://www.slideshare.net/mmahawar/final-java-presentation

10 Tools for Distributed Developer Teams

Good List of tools, interesting read.

http://mashable.com/2010/07/14/distributed-developer-teams/?utm_source=TweetMeme&utm_medium=widget&utm_campaign=retweetbutton

TDD A Religion not Process

Now a days at work I can safely place TDD to the side, even though my boss would love me to practice it. At the moment I’m working in a large, legacy code base. This code base is about 3 years old but looks like it’s thirty :). Let me not get distracted.
Some how TDD has infused itself into my brain today. I can’t stop thinking about it. All the resources out there would make you believe that TDD is some process that all you have to do is adopt and it will lead you to the promised land. In reality, I believe it’s more like a RELIGION. You have to first find your faith and then perform the rituals before the promised land materializes.
I’ve been practicing TDD for over 2 years now and I still find it challenging. Anybody can write unit test after the fact, but only the best developers know how to make great decisions throughout a project. And it’s this ability to make quality decisions consistently that I struggle with day to day. It’s a design activity after all.

Here are couple of very high level challenges which I think I have with TDD

1. How to determine at what scope I should write a test
2. Make sure I capture behavior and not implementation details
3. Learning not to over mock
4. Force myself to write only enough code to make the test pass

There are other challenges which I have seen in projects, when a team does TDD like slowness of tests so team stops running all the tests or UI tests which tend to be fragile, because they often fall because of refactoring. So they often become fragile, failing
erratically and difficult to maintain.

Humm I am thinking of elaborating my bullets points, nahhhhhh not today probably will do that in my next blog.

Manisha

Remember to give a hint

We ran into once interesting discussion during our last code review session. As we all do it's a common trick to use relax access restrictions to test a particular property/method/function, so that you can use it in a unit test, which resides in the same package (though in different catalog).

Whether you think it's good or bad, I believe always remember to give a hint about that to the developer. A simple annotation that tells you why a particular property access restriction has been relaxed.

Consider:

public class foo {
private Long x;
String y;
}

Hummm Why is 'y' package scoped?

public class foo {
private Long x;
@VisibleForTesting String y;
}

Ah, that's why

Nice isn't it.
Manisha

Estimating for size vs Estimating for duration

“Accurate estimates is an oxymoron”

Today at our planning meeting we had an interesting discussion regarding sizing our features. We were questioning how to do right sizing and what is the basis and all good, logical and so called fun questions. Here is my take on it :).

Agile teams estimate SIZE of a feature (Story) rather than estimating the DURATION it will take to complete. Our team likes to estimate size with t-shirt sizes. Story points or T-shirt sizing are a unit of measure for expressing the overall size of a feature. When we estimate with t-shirt sizing or store points, we assign a point value to each item. The raw values we assign are unimportant. What matters are the relative values?

Lets take an example of restaurant before we dive too deep. Suppose a new restaurant opens nearby, and you decide to try it. For the first course, you can have either a bowl of soup. You can have the entree as either a full or half portion. You have probably been to many restaurants like this and can quite easily order about the right amount of food without asking how many ounces are in the cups of soup and exactly how big the entree portions are. At most, you may ask the server something like "How big is the salad?". The server will likely respond by holding his hands apart to illustrate the size. In cases such as these, you are ordering by relative rather than measured size. You are saying, "Give me the large portion" or "I would like the small serving". You are not ordering by exact size, such as I would like 14 ounces of sode, 6 ounces of rice and 3 ounces of veges.
It's possible to estimate an agile project's features in the same way. When I am at an unfamiliar restaurant and order a large soda, I don't really know how many ounces I will get. About all I do know is that large soda is larger than a small or medium soda and that it's smaller than an extra-large one. Similar analogy can be taken for spice levels.

Fortunately, this is all the knowledge I need :). All I need to know is whether a particular feature is larger or smaller than other feature. A feature that is assigned a 2 should be twice as much as a feature that is assigned a 1. THERE IS NO SET FORMULA FOR DEFINING THE SIZE OF A FEATURE. Rather, a sizing estimate is an amalgamation of the amount of effort involved in developing the feature, the complexity of developing it, risk inherent in it and so on.
We can get started by selecting a feature that we expect to be one of the smallest features we will or have work with and say the feature is estimated at 1 point. The best way to see how this works is to try it.

“Confidence comes not from always being right but from not fearing to be wrong.”
A key tenet of agile estimating and planning is that we estimate size but derive duration. (Estimates / Velocity = duration). Velocity is like weather. Based on past we try to predict but it may differ. (I will try to cover techniques of velocity estimation in next blog post.)

The beauty of this is that estimating in story points/ t-shirt sizing completely separates the estimation of effort from the estimation of duration. Of course, effort and schedule are related, but separating them allows each to be estimated independently. In fact, we are no longer even estimating the duration of a project; we are computing it or deriving it.

The distinction is subtle but important.

Nine steps to Better software design

Off late, I don't like to think about designing when I am coding. I know I know designing is important. I am not denying the importance but like to do it differently. I keep SOLID principle in my head when ever I code. These principles are applicable to any domain. And along with these principle I like to follow few rules defined by Jeff Bay in thoughtwork's book.

The Rules
------------
Here are the rules:

1. Use only one level of indentation per method. (My version is two level ;). It's too hard to stick with one.)
2. Don’t use the else keyword.
3. Wrap all primitives and strings.
4. Use only one dot per line.
5. Don’t abbreviate.
6. Keep all entities small.
7. Don’t use any classes with more than two instance variables.
8. Use first-class collections.
9. Don’t use any getters/setters/properties (hummmm take your own call.).

They are easy to follow and does magic with the code...