Accessing JMX remotely via VisualVM from jetty

During my testing I was getting some of weird numbers in GNUPlot so wanted to see what is happening inside JVM.

My application was deployed on Jetty 6.x version. I uncommented jetty-jmx.xml and passed it as parameter during startup i.e. java - jar start.jar etc/jetty-jmx.xml. Hoping that jmx connectors will start on RMI default 1099 port. Started visualvm and tried connecting to my remote jvm using the hostname:1099 port. Got connection refused exception.

I googled and found it's not to do it. There were million things suggested on the net. i.e. start jetty with remote-jmx parameter by specifying the port and then establish ssh tunnel on that port and then connect visualvm to it.

My pair(Chris) and I did all those but somewhere within me I was certain that I don't have to do all these ceremony to connect and it's because we use to connect to Tomcat similarly and the only configuration which we added was jmx related in catalina.sh file.

We followed couple of blogs and no success. Then thought of upgrading the jetty version to 8.x and followed the same steps of uncommenting "createRegistry" and "ConnectorServer" portion within jetty-jmx.xml file.

Then started the server. I got same error. Then found out the we have to specify the ip address of remote host during jetty start up.

java -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.rmi.server.hostname=YOUR_IP -jar start.jar etc/jetty-jmx.xml

Once you will do that you will notice that a URL will get printed on you start up console, copy the URL and change the localhost with your ip address and put the same on to visualvm. Boom it got connected. Now I know and it's simple. For me the url was "service:jmx:rmi:/hostname/jndi/rmi://YOUR_IP:1099/jettyjmx"

Just incase any of you are looking for parameters for enabling stuff in tomcat. Here are they.

In catalina.sh file add the below line anywhere in the beginning around line number 150 :P.

JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9004 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.rmi.server.hostname=YOUR_IP"

Then connect using your_ip:9004 in visualvm.


Did not have lots of fun
Manisha

TCP Tuning

Continuing on my testing venture, I learned a few more cool Linux operating system stuff.

My problems were little unique. My single request takes around 6 millisecond to get processed, so if I fire 4000 concurrent requests for 5million total number of requests, operating system gives up. i.e. I notice all socket and TCP IP related errors. i.e. netstat shows me 30k approx connections in time_wait.
Socket time out, could not assign request address, connection reset by peer, reached maximum limit for open files etc. errors starts popping out.

I tweaked open file and socket max connection limit to get over my initial hump. I was being able to hit 5000 approx concurrent request and started noticing similar kind issues again. Then a colleague recommended to tune some of tcpip receive and send buffer sizes. I read that with latest kernel the auto tuning option comes out of the box and in most of cases you are not required to change those settings. However the maximum buffer size are still too small for many high speed network path and must be increased for each OS.

But since it was my local box I decided to play around with them to see the difference. I modified the /etc/sysctl.conf and added four lines for updating the max and default values for receive and send buffer.

net.core.rmem_max = 10485760

net.core.wmem_max = 10485760

net.ipv4.tcp_rmem = 4096 87380 10485760

net.ipv4.tcp_wmem = 4096 16384 10485760

net.ipv4.tcp_mem = 196608 262144 10485760

It did help in our project.

Thanks
Manisha

"No route to host" error

Can't believe another day is over. Learn so much about installation of Red Hat.

I had one windows desktop and a windows laptop given to me from work, so I decided to format my desktop to linux. Nice idea isn't it. Got the CD from one colleague and I am on my way to install RH for the first time. I have 10 years of experience installing windows :D how hard it can be, correct?

I opted for GUI option and on my way hitting next next next. Installation complete bingo!!!!. It was easy. Next I wanted to know the IP of my box. boom! ifconfig gave me 127.0.0.1. The moment I saw local ip, I knew I am in trouble.
Realized that my ethernet device is all disabled and just activating them solved this tiny issue of mine. Used "/usr/bin/neat" command to figure it out.

Now I got IP and being able to ping to server from my laptop. It's a brand new box so thought, let me go ahead and install my tools like java, jetty etc...
Downloaded the rpms and wanted to ftp to new box and fired putty. Boom! started getting another error. "No route to host".
Googles and found very nice article to help me. (http://nixcraft.com/linux-hardware/6366-no-route-host.html). I am pasting the content.

----------------------------------------------
I too have a problem with putty connect to my
VM Ware. And what I do :

1. Check ip`s and ports in my CentOS (VM Ware) and my PuTTY.

Result : no mistakes.

2. Check the state sshd in my CentOS (VM Ware):

[root@localhost ~]# /etc/init.d/sshd status
sshd (pid 3940 3936) is running...
[root@localhost ~]#

Result : sshd is running.

3. Check the state a port on CentOs for connection through ssh:

[root@localhost ~]# netstat -a | grep ssh
tcp 0 0 *:ssh *:* LISTEN
[root@localhost ~]#

or

[root@localhost ~]# cat /etc/services | grep ssh
ssh 22/tcp # SSH Remote Login Protocol
ssh 22/udp # SSH Remote Login Protocol
[root@localhost ~]# netstat -an | grep 22
tcp 0 0 :::22 :::* LISTEN
[root@localhost ~]#


Result: all is normal.

4. Check configuration my iptables:

[root@localhost ~]# /sbin/iptables -L -n
Chain INPUT (policy ACCEPT)
target prot opt source destination
RH-Firewall-1-INPUT all -- 0.0.0.0/0 0.0.0.0/0

Chain FORWARD (policy ACCEPT)
target prot opt source destination
RH-Firewall-1-INPUT all -- 0.0.0.0/0 0.0.0.0/0

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain RH-Firewall-1-INPUT (2 references)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 icmp type 255
ACCEPT esp -- 0.0.0.0/0 0.0.0.0/0
ACCEPT ah -- 0.0.0.0/0 0.0.0.0/0
ACCEPT udp -- 0.0.0.0/0 224.0.0.251 udp dpt:5353
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:631
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited
[root@localhost ~]#

Result: God dammed!! No rule for port 22! -> port 22 - forbidden!

4. Add rule for port 22 to my iptables:
Just insert it line by means of your favourite editor
in /etc/sysconfig/iptables:
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT

before:

-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT

Warning: your chaine name may be enother, because you have enother Linux. Curent chaine name you can see through /sbin/iptables -L -n (see above).

5. Check /etc/sysconfig/iptables:

[root@localhost ~]# cat /etc/sysconfig/iptables
# Firewall configuration written by system-config-securitylevel
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -p 50 -j ACCEPT
-A RH-Firewall-1-INPUT -p 51 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp --dport 5353 -d 224.0.0.251 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m udp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT
[root@localhost ~]#

6. Reboot you Linux and enjoy PuTTY
[root@localhost ~]# reboot
-------------------------------

Life is good again.

Manisha

"socket: Too many open files (24) Error" and "apr_socket_recv: Connection timed out (110)" Errors

I continued with performance testing of my Rest services and had another interesting day :).

I was trying to run the Apache benchmarking tool (AB Tool) for total number of 10,000 requests with 1000 concurrent request. My services behaved like real gentleman. But soon after I bumped the 1000 concurrent requests to 1500 concurrent requests, I started getting "socket: Too many open files (24) Error" error.

After little bit of googling realized that I am hitting the linux systems open file limit. I ran "ulimit -a" command to see the list of provided resources available to the shell and to the processes started by it. Then I notices that limit for open files (ulimit -n) is set to 1024. I edited the limit by running "ulimit -n 5000" command and could get over this error.

You might have to add a new line under /etc/profile to make this change permanent . The changes for ulimit hold good only for running shell.

Time for Samba dance again. Soon I wanted to test my services for 100,000 total number of request and guess what I ran into another error. :(. "apr_socket_recv: Connection timed out (110)" . :( . Realized that another OS property has to be modified to accomodate that many request. somaxconn.

To view the existing setting , please look into /proc/sys/net/core/somaxconn file and to change max connection parameter from default 128 to 10240, you need to add “net.core.somaxconn = 10240” line in /etc/sysctl.conf file. Then run “/sbin/sysctl -p /etc/sysctl.conf” command. Now the /proc/sys/net/core/somaxconn file should have our new number. Now the /proc/sys/net/core/somaxconn file should have our new number.
echo "10240" > /proc/sys/net/core/somaxconn command will change the settings temporarily.

Now I am being able to fire 5000 concurrent request for total number of 5000000 request. Happy again.

Hope it helps somebody
Manisha

Apache Benchmarking Tool Won

I wanted to do the performance test for my rest services and was looking for simplest tool which can send concurrent and sequential requests. Having prior experience with Grinder it become my obvious and first choice. Honestly it did the job but was not very simple, it took me
couple of hours to do the entire setup and then I came across AB (http://httpd.apache.org/docs/2.0/programs/ab.html) Tool.

Finding the URL to download was the toughest thing :). It took me 10 minutes to figure it out (http://www.apache.org/dist/httpd/binaries/win32/).
I know, I am dumb but once the took was installed it was as easy breeze.

I realized the tool is part Lion OS, you don't have to install it separately, so mac users can hit the ground running in no time vs Windows users have to download and set the environment variable and add that to path variable.

Once the Apache benchmark bin directory is in you path, you are ready to rock and roll.

Open the command prompt ( cmd) then type "ab" and hit enter, it will show you all the options. I used -n ( Number of request) and -c (Concurrent request).


Enjoy
Manisha

Interesting SimpleDateFormat Problem

We wasted around half an hour today on a trivial SimpleDateFormat issue today. We wanted to validate a date and used the below code.

@Test
public void testShouldThrowTheException() {
String formatStr = "yyyy-MM-dd";
SimpleDateFormat df = new SimpleDateFormat(formatStr);
try {
df.parse("2011-15-05");
Assert.fail("It should not come here");
} catch (ParseException e) {
e.printStackTrace();
}
}

Surprisingly our testcase was failing and did not know why. SimpleDateFormat was returning 2012 March 5th and I was so annoyed :S..

Looks like the SimpleDateFormat will not validate the correctness untill you set the lenient to false.

@Test
public void testShouldThrowTheException() {
String formatStr = "yyyy-MM-dd";
SimpleDateFormat df = new SimpleDateFormat(formatStr);
df.setLenient(false);
try {
df.parse("2011-24-05");
Assert.fail("failed");
} catch (ParseException e) {
e.printStackTrace();
}
}

This works perfectly fine. Simple problem but easy to overlook.

Enjoy
Manisha