TCP Tuning

Continuing on my testing venture, I learned a few more cool Linux operating system stuff.

My problems were little unique. My single request takes around 6 millisecond to get processed, so if I fire 4000 concurrent requests for 5million total number of requests, operating system gives up. i.e. I notice all socket and TCP IP related errors. i.e. netstat shows me 30k approx connections in time_wait.
Socket time out, could not assign request address, connection reset by peer, reached maximum limit for open files etc. errors starts popping out.

I tweaked open file and socket max connection limit to get over my initial hump. I was being able to hit 5000 approx concurrent request and started noticing similar kind issues again. Then a colleague recommended to tune some of tcpip receive and send buffer sizes. I read that with latest kernel the auto tuning option comes out of the box and in most of cases you are not required to change those settings. However the maximum buffer size are still too small for many high speed network path and must be increased for each OS.

But since it was my local box I decided to play around with them to see the difference. I modified the /etc/sysctl.conf and added four lines for updating the max and default values for receive and send buffer.

net.core.rmem_max = 10485760

net.core.wmem_max = 10485760

net.ipv4.tcp_rmem = 4096 87380 10485760

net.ipv4.tcp_wmem = 4096 16384 10485760

net.ipv4.tcp_mem = 196608 262144 10485760

It did help in our project.

Thanks
Manisha

"No route to host" error

Can't believe another day is over. Learn so much about installation of Red Hat.

I had one windows desktop and a windows laptop given to me from work, so I decided to format my desktop to linux. Nice idea isn't it. Got the CD from one colleague and I am on my way to install RH for the first time. I have 10 years of experience installing windows :D how hard it can be, correct?

I opted for GUI option and on my way hitting next next next. Installation complete bingo!!!!. It was easy. Next I wanted to know the IP of my box. boom! ifconfig gave me 127.0.0.1. The moment I saw local ip, I knew I am in trouble.
Realized that my ethernet device is all disabled and just activating them solved this tiny issue of mine. Used "/usr/bin/neat" command to figure it out.

Now I got IP and being able to ping to server from my laptop. It's a brand new box so thought, let me go ahead and install my tools like java, jetty etc...
Downloaded the rpms and wanted to ftp to new box and fired putty. Boom! started getting another error. "No route to host".
Googles and found very nice article to help me. (http://nixcraft.com/linux-hardware/6366-no-route-host.html). I am pasting the content.

----------------------------------------------
I too have a problem with putty connect to my
VM Ware. And what I do :

1. Check ip`s and ports in my CentOS (VM Ware) and my PuTTY.

Result : no mistakes.

2. Check the state sshd in my CentOS (VM Ware):

[root@localhost ~]# /etc/init.d/sshd status
sshd (pid 3940 3936) is running...
[root@localhost ~]#

Result : sshd is running.

3. Check the state a port on CentOs for connection through ssh:

[root@localhost ~]# netstat -a | grep ssh
tcp 0 0 *:ssh *:* LISTEN
[root@localhost ~]#

or

[root@localhost ~]# cat /etc/services | grep ssh
ssh 22/tcp # SSH Remote Login Protocol
ssh 22/udp # SSH Remote Login Protocol
[root@localhost ~]# netstat -an | grep 22
tcp 0 0 :::22 :::* LISTEN
[root@localhost ~]#


Result: all is normal.

4. Check configuration my iptables:

[root@localhost ~]# /sbin/iptables -L -n
Chain INPUT (policy ACCEPT)
target prot opt source destination
RH-Firewall-1-INPUT all -- 0.0.0.0/0 0.0.0.0/0

Chain FORWARD (policy ACCEPT)
target prot opt source destination
RH-Firewall-1-INPUT all -- 0.0.0.0/0 0.0.0.0/0

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain RH-Firewall-1-INPUT (2 references)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 icmp type 255
ACCEPT esp -- 0.0.0.0/0 0.0.0.0/0
ACCEPT ah -- 0.0.0.0/0 0.0.0.0/0
ACCEPT udp -- 0.0.0.0/0 224.0.0.251 udp dpt:5353
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:631
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited
[root@localhost ~]#

Result: God dammed!! No rule for port 22! -> port 22 - forbidden!

4. Add rule for port 22 to my iptables:
Just insert it line by means of your favourite editor
in /etc/sysconfig/iptables:
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT

before:

-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT

Warning: your chaine name may be enother, because you have enother Linux. Curent chaine name you can see through /sbin/iptables -L -n (see above).

5. Check /etc/sysconfig/iptables:

[root@localhost ~]# cat /etc/sysconfig/iptables
# Firewall configuration written by system-config-securitylevel
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -p 50 -j ACCEPT
-A RH-Firewall-1-INPUT -p 51 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp --dport 5353 -d 224.0.0.251 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m udp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT
[root@localhost ~]#

6. Reboot you Linux and enjoy PuTTY
[root@localhost ~]# reboot
-------------------------------

Life is good again.

Manisha

"socket: Too many open files (24) Error" and "apr_socket_recv: Connection timed out (110)" Errors

I continued with performance testing of my Rest services and had another interesting day :).

I was trying to run the Apache benchmarking tool (AB Tool) for total number of 10,000 requests with 1000 concurrent request. My services behaved like real gentleman. But soon after I bumped the 1000 concurrent requests to 1500 concurrent requests, I started getting "socket: Too many open files (24) Error" error.

After little bit of googling realized that I am hitting the linux systems open file limit. I ran "ulimit -a" command to see the list of provided resources available to the shell and to the processes started by it. Then I notices that limit for open files (ulimit -n) is set to 1024. I edited the limit by running "ulimit -n 5000" command and could get over this error.

You might have to add a new line under /etc/profile to make this change permanent . The changes for ulimit hold good only for running shell.

Time for Samba dance again. Soon I wanted to test my services for 100,000 total number of request and guess what I ran into another error. :(. "apr_socket_recv: Connection timed out (110)" . :( . Realized that another OS property has to be modified to accomodate that many request. somaxconn.

To view the existing setting , please look into /proc/sys/net/core/somaxconn file and to change max connection parameter from default 128 to 10240, you need to add “net.core.somaxconn = 10240” line in /etc/sysctl.conf file. Then run “/sbin/sysctl -p /etc/sysctl.conf” command. Now the /proc/sys/net/core/somaxconn file should have our new number. Now the /proc/sys/net/core/somaxconn file should have our new number.
echo "10240" > /proc/sys/net/core/somaxconn command will change the settings temporarily.

Now I am being able to fire 5000 concurrent request for total number of 5000000 request. Happy again.

Hope it helps somebody
Manisha