Return to AB tool

Once I had HAProxy up and running, it was time to do some performance test. I had two things in my mind for this round.

1) Making sure HAProxy is handling configuration changes gracefully, without breaking any existing connections.
2) Comparing the performance of server with and without HAProxy.

Started marching toward first goal. In HAProxy documentation there is a "-sf" option for soft restart. I started AB tool with 50000 request and while AB tool was busy firing those request to HAProxy, I took out one server from proxy setting and restarted the server using -sf option. I noticed AB tool gave me "connection reset by peer error". To understand what is happening and why my server is sending reset package to AB tool, I decided to capture the tcpdump and analyze it in wireshark.

I was surprised that to see that it's AB tool who is sending reset package to server. i.e. if I fired 100 requests using AB tool then I noticed 100 tcp connections with reset flag and 100 with fin, ack flag. Believe me at this point I am totally confused and not sure why the heck AB tool is behaving in such a way. I still don't know. Then I decided to use JMeter. Did similar exercise and the tcp dump was much cleaner. I noticed only 100 connection with fin,ack flag and no connections in reset mode. Much better.

So started looking to my original issue of dropping connection since JMeter log was also showing connection refused when I tried doing configuration swap. "option redispatch", "option httpchk" and "retries" options saved my day. After specifying these options I was not getting Connection refused any more while doing the changes in the configuration.

One down one more to go. Since I noticed tcp reset flag weirdness with AB tool thought it would be better to use JMeter for second round. Without HAProxy performance of my service was 4500+ req/sec using AB tool. I wanted to capture same thing with JMeter and it reported back 1800+ req/sec.

Ohhh Ohhh. Something wrong again. Started reading about JMeter and found out that JMeter’s HTTP client is slower than that of AB. You can use JMeter to find out if a change to your webapp, your tomcat installation or your JVM, accelerated or slows the response times of the webpage however you cannot use JMeter to determine the server’s maximum number of requests per second that it can successfully serve because of JMeter’s HTTP client appears to be slower than Tomcat’s server code.

So had to come back to my old love AB.
Manisha

Cause of "Connect reset by peer" error message using AB Tool

One more interesting finding about AB Tool (Apache Benchmarking tool) on linux.

During one of my recent runs I notices a very weird behavior. I was getting "Connection reset by peer" error message consistently when I was firing 15000 concurrent request with 90000 total number of request. I started getting successful response back when I fired 15000 concurrent request with 30000 total number of request. (i.e. ab -c 15000 -n 30000 vs. ab -c 15000 -n 90000) . I fired the 15k and 30k combination in a loop 100 times and every time I got successful result. I wanted to know why I am getting connection reset when I am firing 90K request in one process vs firing much more across multiple process.

I observed "Possible SYN flooding on port 80. Sending cookies" error message in the redhat's kernal log(/var/log/message). Since AB tool was firing lots of request the OS considered it as threat. :0.

I disabled tcp_syncookie by firing sysctl -w net.ipv4.tcp_syncookies=0 command and was being able to fire request much more than 90K. There are other parameters which you can be modify to increase the syncookie limit and time wait limit which might be helpful incase you don't have layers of firewall protecting your server. I will try mentioning them in my next blog.

Hope it helps

CXF Rest service renders JSON with @ for attributes

In my current project I am exposing lots of Rest Services in Java using CXF. All the service should be able to render response in json and xml format. This can very easily be done by just using a annotation(@Produces({ MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON })) on the exposed service method. To verify the service is rendering correct response based on passed header type (accept:application/json or accept:application/xml).

I used JsonView firefox plugin. (I like this plugin since you get formatted JSON back vs one line response which you get if you will use Poster firefox plugin).

I noticed that JSON output has extra '@' symbol for attributes. i.e.

......"Programs": {"Channel": {
"@ID": "E",
"TopName": "The Fresh Prince of Bel-Air",
"BottomName": "Bang the Drum Ashley",
"StartTime": {
"@utc": "1325741400000",
"$": "2012-01-05T05:30:00"
},...........

Then after googling I found that CXF default JSON render is Jettison which produced this sort of output. (Why Jettison behaves in such a way is still mystery. I have to it figure out sooner than later.)

If you will replace the default JSON renderer Jettison with Jackson problem will be resolved and it' very simple to provide your custom Provider. I added following lines into my spring.xml

<bean id="jacksonObjectMapper" class="org.codehaus.jackson.map.ObjectMapper">
<property name="serializationConfig.dateFormat">
<bean class="java.text.SimpleDateFormat">
<constructor-arg value="EEE, d MMM yyyy HH:mm:ss z" />
</bean>
</property>
</bean>

<bean id="jsonProvider" class="org.codehaus.jackson.jaxrs.JacksonJaxbJsonProvider">
<property name="mapper" ref="jacksonObjectMapper" />
</bean>

and added the provider into my server declaration i.e

<jaxrs:server id="xService" address="/x">
<jaxrs:serviceBeans>
<ref bean="xServices" />
</jaxrs:serviceBeans>
<jaxrs:providers>
  <ref bean="jsonProvider"/>
</jaxrs:providers>
</jaxrs:server&g

After adding Jackson as my provider. I redeployed my project and JSON was coming without @ sign. :))))

"programList": [{
"isHD": true,
"isClosedCaptioned": false,
"length": 120,
"description": "",
},
"startTime": {
"textValue": "2012-01-05T17:00:00",
"utcStartTime": 1325782800000 ....

-Enjoy
Manisha