Mysql's THE GROUP_CONCAT() function

I must have written multiple custom code in Java to aggregate my results from a table. i.e. Assume a table like below.

Table A

-----------

name

-----------

Manisha

Nilanjan

Nima

To display (Manisha, Nilanjan, Nima ) I always use to aggregate all the values, using some sort of loop. With mysql group_concat function it can be done very easily.

SELECT GROUP_CONCAT(name) As Name FROM A;

Then the output will be:

name

----

Manisha, Nilanjan, Nima

We can also use some format of GROUP_CONCAT(). Like

SELECT GROUP_CONCAT( Language SEPARATOR ‘-’)-> It will use ‘-’ instead of ‘,’

SELECT GROUP_CONCAT( Language ORDER BY Language DESC ) -> To change the order and shorting output

Another thing to remember that GROUP_CONCAT() ignores NULL values. Group_Concat function can be used in join scenarios too. i.e.

Employee_TYPE Table

--------------------
Id | Desc

1        |  1 | FT  

2        |  2 | PT  

Employee table

---------------
ID | Name
1  | Manisha
1  | Nilanjan

2  | Nima

And to display something like below group_concat can be used too.

ID   | Desc | Name
   1 |  FT  |  Manisha, Nilanjan

   2 |  PT  |  Nima 

SELECT Employee_Type.*, (

    SELECT GROUP_CONCAT(Name SEPARATOR ',')

    FROM Employee

    WHERE Employee_Type.id = Employee.person_id

) as employee

FROM Employee_Type

   
Cool isn't it. 
Manisha 

Snippet to generate lines for TestSuite

I wanted to generate the TestSuite for all my classes and was not ready to click on all 200 test classes to generate from eclipse So wrote a small testcase which printed the lines for me to add into my TestSuite. Here is the code snippet.



    @Test
    public void testShouldPrintAllTheNameOfTestClasses() throws Exception {
        File dir = new File("/Project/DIF/dif_parent");
        LinkedList files = (LinkedList)FileUtils.listFiles(
                  dir,
                  new RegexFileFilter("^.*[tT]est(-\\d+)?\\.java$"),
                  DirectoryFileFilter.DIRECTORY
                );
        for (File file: files) {
          System.out.println("suite.addTestSuite(" + file.getName().replace("java","class" ) + ");");
        }
    }


Thanks
Manisha

Attaching Source code with Maven

Initially I use to download the source jars and then use to attach them to project. Then started adding source jar as dependencies to pom for dev profile. Recently I found a maven command which does i it seamlessly for us.

I ran below commands to download and attach the source code of all the projects in Eclipse for me.

mvn dependency:sources

mvn eclipse:eclipse -DdownloadSources=true 

Boom! refresh all project in Eclipse and the source code must be attached to your project.                    

Enjoy looking into source
Manisha

JMeter is not good tool for finding Request per second

I played around with performance tools a lot in my last project and this was one blog which I never got to finish. My goal was to come with a tool which is best for benchmarking an application on Tomcat. While working with JMeters all the numbers compare to AB tool were low, I suspected Tomcat and started reading the tomcat performance tuning book and found below lines.

(Page 136 - Chapter 4 - Tomcat Performance Tuning ) - JMeter's HTTP client is slower than that of AB or siege. You can use JMeter to  find out if a change to your webapp, your tomcat installation or your JVM, accelerated or slows the response times of the webpage however you cannot use JMeter to determine the server's maximum number of requests per second that it can successfully serve because of JMeter's HTTP client appears to be slower than Tomcat's server code.


http://books.google.com/books?id=vJttHyVF0SUC&pg=PA136&lpg=PA136&dq=%22maximum+number+of+requests+per+second%22++jmeter&source=bl&ots=i-5yv-tLh0&sig=0JnmfPtq2PwKWdEDkfSamUQMBEg&hl=en&sa=X&ei=6L0QT-3DGc6lsQKmio2LBA&ved=0CDIQ6AEwAg#v=onepage&q=%22maximum%20number%20of%20requests%20per%20second%22%20%20jmeter&f=false

I thought it was really valuable find :)
Manisha 

Get rid of ^M in VI editor

^M magically appears on scripts when we copy a file from windows to Linux OS after modification.

To remove the ^M characters at the end of all lines in VI use below command:

:%s/^V^M//g

The ^V is a CONTROL-V character and ^M is a CONTROL-M. When you type this, it will look like below command. In UNIX, you can escape a control character by proceeding it with a CONTROL-V.

:%s/^M//g

The :%s is a basic search and replace command in VI. Above command tells VI to replace all the ^M character globally with nothing and to escape ^M we need to type ^V.

Enjoy

Manisha

Creates the appropriate HSSFWorkbook / XSSFWorkbook from the given InputStream

I waited enough for Sandip to write this blog since it was his find :).

Here is the problem we had, we have an application where user can upload the excels and our application process them. Our users were uploading 2007 formatted excel file in .xls format or vices-verse. Microsoft excel 2007 is backward compatible but our POI libraries sometime gets confuse. DOM parser is smart enough to handle but when it comes to SAX we need to know the exact format of the file loaded. i.e. just checking the file extension might not be enough. So we have to look into input stream to find the exact loaded excel format of the file.

As usual I looked over the internet and could not find anything post how to do it. Since Sandip is lot more savvy with Apache POI he started looking into the Apace POI source code and found WorkbookFactory class (Factory for creating the appropriate kind of Workbook) has a help method which takes the inputStream and based on that creates an appropriate object. Look at the code below.

============================
/**
 * Creates the appropriate HSSFWorkbook / XSSFWorkbook from
 *  the given InputStream.
 * Your input stream MUST either support mark/reset, or
 *  be wrapped as a {@link PushbackInputStream}!
 */
public static Workbook create(InputStream inp) throws IOException, InvalidFormatException {
    // If clearly doesn't do mark/reset, wrap up
    if(! inp.markSupported()) {
        inp = new PushbackInputStream(inp, 8);
    }
   
    if(POIFSFileSystem.hasPOIFSHeader(inp)) {
        return new HSSFWorkbook(inp);
    }
    if(POIXMLDocument.hasOOXMLHeader(inp)) {
        return new XSSFWorkbook(OPCPackage.open(inp));
    }
    throw new IllegalArgumentException("Your InputStream was neither an OLE2 stream, nor an OOXML stream");
}

============================

Good find Sandip
Manisha

JSP/Tomcat 6 "org.apache.jasper.JasperException: java.lang.ArrayIndexOutOfBoundsException"

We noticed "org.apache.jasper.JasperException: java.lang.ArrayIndexOutOfBoundsException" on our JSP pages in the morning. I have copied the entire exception stack trace below for reference.

My first guess was the jsp pages have some issue where we are trying to access some index which has not been assigned. But we realized it's the server which ran our of space. so the first exception was "

java.io.IOException: There is not enough space in the file system.".

Why tomcat has to throw such an non intuitive error message? Why?  Not sure if it's tomcat's fault or the underneath framework is messing up.

==============================================================
org.apache.jasper.JasperException: java.lang.ArrayIndexOutOfBoundsException
 org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:522)
 org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:416)
 org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:342)
 org.apache.jasper.servlet.JspServlet.service(JspServlet.java:267)
 javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 sun.reflect.GeneratedMethodAccessor209.invoke(Unknown Source)
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
 java.lang.reflect.Method.invoke(Method.java:600)
 org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:269)
 java.security.AccessController.doPrivileged(AccessController.java:284)
 javax.security.auth.Subject.doAsPrivileged(Subject.java:727)
 org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:301)
 org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:162)

root cause
java.lang.ArrayIndexOutOfBoundsException
 java.io.OutputStreamWriter.write(OutputStreamWriter.java:250)
 java.io.Writer.write(Writer.java:152)
 org.apache.log4j.helpers.CountingQuietWriter.write(CountingQuietWriter.java:44)
 org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:308)
 org.apache.log4j.RollingFileAppender.subAppend(RollingFileAppender.java:236)
 org.apache.log4j.WriterAppender.append(WriterAppender.java:159)
 org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:230)
 org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:65)
 org.apache.log4j.Category.callAppenders(Category.java:203)
 org.apache.log4j.Category.forcedLog(Category.java:388)
 org.apache.log4j.Category.error(Category.java:319)
 com.jpmc.tao.common.logging.Logger.error(Logger.java:167)
 com.jpmc.tao.presentation.controller.FrontController.process(FrontController.java:61)
 org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)
 javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
 javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
 java.lang.reflect.Method.invoke(Method.java:600)
 org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:269)
 java.security.AccessController.doPrivileged(AccessController.java:284)
 javax.security.auth.Subject.doAsPrivileged(Subject.java:727)
 org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:301)
 org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:162)
 java.security.AccessController.doPrivileged(AccessController.java:251)
 java.security.AccessController.doPrivileged(AccessController.java:251)
 org.apache.jasper.runtime.PageContextImpl.doForward(PageContextImpl.java:706)
 org.apache.jasper.runtime.PageContextImpl.access$1000(PageContextImpl.java:71)
 org.apache.jasper.runtime.PageContextImpl$11.run(PageContextImpl.java:664)
 java.security.AccessController.doPrivileged(AccessController.java:251)
 org.apache.jasper.runtime.PageContextImpl.forward(PageContextImpl.java:662)
 org.apache.jsp.default_jsp._jspService(default_jsp.java:54)
 org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
 javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:374)
 org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:342)
 org.apache.jasper.servlet.JspServlet.service(JspServlet.java:267)
 javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
 sun.reflect.GeneratedMethodAccessor209.invoke(Unknown Source)
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
 java.lang.reflect.Method.invoke(Method.java:600)
 org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:269)
 java.security.AccessController.doPrivileged(AccessController.java:284)
 javax.security.auth.Subject.doAsPrivileged(Subject.java:727)
 org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:301)
 org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:162)
==============================================================

Any thoughts?
Manisha

How to be a Technical Leader

This is one question which is bothering me from a while. Honestly there are bunch of questions and this is the root of all.

1) What is leadership? Bossing over people?
2) How can I be leader and be technical at the same time?
3) Can I be a leader without becoming like those other people?

To me leadership should create an environment in which people feel empowered. An environment where one is free to talk, free to act and ask anything. Since each person is unique, can one style be application to entire team. Should leader choose different style of leadership for different teams? The biggest question to me is WHAT IS LEADERSHIP? Now a days I hear servant leadership term but what if your team is not capable to solve problems on their own. Will this style of leadership work in the mentioned scenario? I would assume no. I did not have very sweet experience.

To me a leader should be a good motivator. A leader should understand the problem and manage flow of ideas and manage quality too. Above all team should have FAITH in the leader and his leadership. There are lots of other factors which might come in play as a leader. May in next blog :)

Manisha

Remove all unused imports with one click

I am almost month old in my new company. The project I am working has 5K approx warnings and majority of them are because of unused imports and unused variables (gurrrrr). So I was looking for someway to fix these minor things using some tool and guess what I found eclipse "Organize Imports (Ctrl+Shift+O) in most eclipse" can be executed at package or project level. Bingoooooooooo resolved around half of warning. Feeling better inside :)

Choose the package -> Right click -> Source -> Organize Imports.

Now to the next job of finding the tool for unused variables. Wish me luck
Manisha

Java Performance- Cookbook Part II

This is the continuation of my previous blog post. Last post I shared my chapter 2 notes and here we go with chapter 3.

This chapter provides an overview of the "Hotspot Java Virtual Machine" (VM) architecture and it's major component. There are three major components of the HotSpot VM. 
1) VM Runtime
2) JIT Compiler
3) and A Memory Manager (Runtime)

HotSpot VM High Level Architecture

HotSpot VM has a JIT compiler, client or server, is pluggable, as is the choice of garbage collector: Serial GC, Throughput, Concurrent or G1. VM also has a runtime (HotSpot VM Runtime) which provides services and common APIs to the JIT compilers and garbage collector. Runtime also provides basic functionality to the VM such as a launcher, thread management, Java Native Interface and so on.

VM yields high scalability by generating dynamic optimization. In other words it makes optimization decisions while the Java application running and generates high performing native machine instruction targeted for the underlying system architecture.

32-bit VMs are limited to 4 GB memory. It is important to note that the actual Java heap space available for 32-bit VM may be further limited depending on the underlying OS. For instance Windows OS the maximum Java heap available to the VM is around 1.5 GB. For Linux OS it is around 2.5 to 3.0 GB. The actual maximums vary due to memory address space consumed by both a given Java application and JVM version.

64-bit VM allows these systems to utilize more memory. However, with 64-bit VMs come a performance penalty due to an increase in size of the internal HotSpot VM's representation of Java objects, called Ordinary Object Pointers (OOPS), which have an increase in the width from 32-bit to 64-bit. 

Increase in width results in fewer OOPS being available on a CPU cache line. The decrease in CPU cache efficiency results in about 8% to 15 % performance degradation compare to 32-bit JVM. A new feature called compressed oops, which is enabled with the -XX:+UseCompressedOops VM command line option, can yield 32-bit JVM performance with the benefit of larger 64-bit Java Heaps. Java application realize better performance with a 64-bit VM using compressed oops than they achieve with 32-bit VM. Application executes faster with improved CPU cache utilization.

HotSport VM Runtime

VM Runtime encompasses many responsibilities, including parsing of command line arguments, VM life cycle, class loading, byte code interpreter, exception handling, synchronization, thread management etc.

a) Command Line Options
VM Runtime parses command line options and configures the HotSpot VM based on those options. There are 3 main categories of command line options.

1) Standard Options - These options are expected to be accepted by all JVM implementation and stable between releases.
2) Non-standard Options - These options are not required to be supported by all JVM and begin with a -X option.
3) Developer Options - It begins with -XX prefix.

b) VM Life Cycle
The HotSpot VM Runtime is responsible for launching and shutting-down the HotSpot VM. The component that starts the VM is called the "launcher" (java, javaw, javaws (java web start)). Below are the steps which launcher executes to start the VM.


1) Parse command line options, such as -client or -server, determines the JIT compiler to load.
2) Establish the Java Heap size and JIT Compiler Type.
3) Establish environment varibales such as LD_LIBRARY_PATH and CLASSPATH.
4) Look for MAIN-Class name from command line or from JAR's manifest file.
5) Create VM using the standard JNI method (JNI_CreateJavaVM).
6) Once the VM is created and initialized, the Java Main-class is loaded and launcher gets the Java main methods attributes from the java main-class.
7) Java main method is invoked in the VM using JNI method CallStaticVoidMethod passing it the marshaled arguments from command line.

Once the java program completes it's execution both method's and program's exit status must be passed back to their caller. The Java main method is detached from the VM using JNI's method DetachCurrentThread and  thread count decrements so the JNI can safely invoke DestroyJavaVM method.

Sorry could not finish typing my notes.. There are few lot more important stuff which I would love to remember from this chapter. More to come soon.

Manisha

Java Performance- Cookbook Part I

I was involve in lots of performance related issues, so thought of reading "Java Performance" book to serve my clients better.
The Java Performance book is nice but umpteen times I felt it covers lots of things very superficially, having a link for further reading would have been nicer in my situation. Overall thumps up and would recommend every Java developer to have a copy on his/her bookshelf.

Things which I would like to remember (my notes) from Chapter 2 - Operating System Performance.

It's very important to understand the concepts before jumping deeper i.e. What is Performance Monitoring, Performance profiling and performance tuning. We will also look little bit into operating system monitoring command line and GUI tools.

Performance Monitoring

Performance monitoring is an act of collecting or observing performance data, from an application having performance issues with no sufficient information or clues to potential root cause.  This can be performed in any environment.

Performance Profiling
 Performance profiling in similar to monitoring activity with more narrow focus. Profiling is rarely done in Production environment. This is often an act that follows a monitoring activity that indicate some kind of performance issue.

Performance Tuning
 Performance tuning is an act of changing tune-ables, source code or configure attribute(s) for the purpose of improving application responsiveness or throughput. 

To reach highest performance we do need to understand, how CPU resources are utilized. Our goal should be to reduce kernel or system CPU utilization as much as possible i.e High kernel or system CPU utilization can be an indication of shared resource contention or a large number of interaction between I/O devices.

Stall
The stall is a scenario where CPU is reported as being utilized even though the CPU may be waiting for data to be fetched from memory. Stalls occurs any time the CPU executes an instruction and the data being operated on by the instruction is not readily available in CPU register or cache. For applications that are compute intensive, user has to observe IPC ( Instructions per clock , The number of CPU instructions per CPU clock cycle) or CPI ( Cycles per instructions, The number of CPU clock cycles per CPU instruction) to calculate it's wait. The target is to reduce number of stalls or improve the CPU's cache utilization so fewer CPU clock cycles are wasted waiting for data to be fetched from memory.

Lets look into few monitoring tools for Linux OS. The book has described tools for other operating systems, so if you need please reference the book :).

Monitoring CPU utilization on Linux
GNOME System Monitor tool can be used to monitor CPU utilization graphically. The tool can be launched with "gnome-system-monitor" command. Few Linux distributions may also include xosview tool.  One of the additional features of xosview in CPU utilization is further broken down into user CPU, kernel or system CPU and idle CPU.

Linux also provide vmstat and mpstat command line tools to monitor CPU utilization. 

VMSTAT
vmstat reports the summary of all CPU utilization , across all virtual processors (The number of virtual processors is the number of hardware threads on the system. It is also the value returned by the Java API, Runtime.availableProcessors() ), data collected since the system has last been booted.

MPSTAT
mpstat offers a tabular view of CPU utilization for each virtual processor. Most Linux distribution requires an installation of the sysstat package to use mpstat. Using mpstat to observe, per Virtual processor, CPU utilization can be useful in identifying whether an application has threads that tend to consume larger percentages of CPU cycles than other threads or whether application threads tend to utilize the same percentage of CPU cycles. The latter observed behavior usually  suggests an application that may scale better.

CPU Scheduler Run Queue
CPU scheduler run queue tell if the system is being saturated with work.   Run depth queue 3 or 4 times greater than the number of virtual processors over an extended time period should be considered an observation that requires immediate attention or action.
vmstat tool's first column reports the run queue depth. The number reported is the actual number of lightweighted processed in the run queue. Java programmers can realize better performance through choosing more efficient algorithms or data structure. In other words, explore alternative approaches that will result in fewer CPU cycles necessary to run the application such as reducing garbage collection frequency or alternative algorithms that result in fewer CPU instructions to execute the same work.

In addition to CPU utilization there are attributes of system's memory that should be monitored, such as paging or swapping activity, locking and voluntary and involuntary context switching along with thread migration activity.  Performance issues can be noticed for a JVM based application, if swapping activity is monitored.

Monitoring Memory Utilization on Linux
Swapping occurs when there is more memory being consumed by application running on the system than there is physical memory available. To deal with this situation, a system is usually configured with an area called swap space.  vmstat tool's can be used to monitor the system for swapping activity. top command and /proc/meminfo file can be used too.  The columns in vmstat to monitor are "si" and "so", which represents  the amount of memory paged-in and amount of memory paged-out.  In addition, the "free" column reports the amount of available free memory. It's important to observe the free memory when low and high paging activity is occurring at the same time.

Monitoring Lock Contention on Linux
pidstat command from sysstat package can be used to monitor lock contention. "pidstat -w" reports voluntary context switches in "cswch/s" column (The numbers are for all virtual processes.
The book has nice formula to calculate the percentage of clock cycles wasted). General guideline of 3% o 5% of clock cycles spent in voluntary context switches implies a Java application that may be suffering from lock contention. 

Monitoring Network I/O Utilization on Linux
Linux has netstat command line tool bundled with it's sysstat package. The netstat tool does not report network utilization. It reports numbers of packet sent and received per second along with errors and collisions. i.e. You can make out that network traffic is occurring but whether the network is 100% utilized or 1% utilized, is difficult to tell from netstat output.
A nicstat tool provides meaningful data under %Util column (The network interface utilization). We have to download and compile the source code before being able to use it on Linux.

Monitoring Disk I/O Utilization on Linux
Disk utilization is most useful monitoring statistic for understanding application's (writing to log, accessing database etc.). It is a measure of active disk I/O time.  iostat command tool can be used to monitor disk utilization.

If improved disk utilization is required several strategies may help.

At hardware and OS level
1. a faster storage device
2. Spreading file systems across multiple disks
3. Tuning the os to cache larger amounts of file system data structures

At application level:
1.  Reducing the number of read and write operations using buffered input and output streams or caching data structure into the application. Buffered data structured are available in the JDK that can easily be utilized.
2) Use non-blocking Java NIO (Grizzly project) instead of blocking java.net.Socket.  

SAR Command Line Tool on Linuz
With sar, you can select which data to collect such as user CPU utilization, system or kernel CPU utilization, number of system calls, memory paging and disk I/O statistics etc. Observing data collected over a longer period of time can help identifying trends that may provide early indications of pending performance concerns.

My notes are over :) for chapter 2. I read man pages for more information. Tomorrow will share notes for chapter 3.

Happy Reading
Manisha

SpEL: I am loving it

Sorry, as usual, got distracted from cucumber to SpEL (Spring Expression Language). I was discussing about SpEL with a friend other day, so thought of reading little bit more about it.

I should accept I am simply blown away and not sure why I was not using it all the while. Starting from “literal expression”-to-“expression templating”. All the expression usage will boost JAVA developer’s productivity.

My favorites so are:
1) Elvis Operator: The Elvis operator is a shortening of the ternary operator syntax. i.e. in ternary operator we write code like below

String name = "Elvis Presley";
String displayName = name != null ? name : "Unknown";

Using Elvis Operator it can be written like

ExpressionParser parser = new SpelExpressionParser();
Inventor tesla = new Inventor("Nikola Tesla", "Serbian");
StandardEvaluationContext context = new StandardEvaluationContext(tesla);
String name = parser.parseExpression("Name?:'Elvis Presley'")
.getValue(context, String.class); System.out.println(name); // Nikola Tesla

The Elvis operator can be used to apply default values in expressions

2) Safe Navigation operator: The Safe Navigation operator is used to avoid a NullPointerException. Here are some example code from Spring Documentation.

ExpressionParser parser = new SpelExpressionParser();  

Inventor tesla = new Inventor("Nikola Tesla", "Serbian"); tesla.setPlaceOfBirth(new PlaceOfBirth("Smiljan"));  

StandardEvaluationContext context = new StandardEvaluationContext(tesla);  

String city= parser.parseExpression("PlaceOfBirth?.City").getValue(context, String.class); 

System.out.println(city); // Smiljan  

tesla.setPlaceOfBirth(null);  city = parser.parseExpression("PlaceOfBirth?.City").getValue(context, String.class);  

System.out.println(city); // null - does not throw NullPointerException!!!

3) Collection Selection: This is my favorite, don’t get me wrong all of the ones I have listed are great but this is super. Selection is a powerful expression language feature that allows you to transform some source collection into another by selecting from its entries. Selection is possible upon both lists and maps

Selection uses the syntax ?[selectionExpression]. This will filter the collection and return a new collection containing a subset of the original elements. For example, selection would allow us to easily get a list of Serbian inventors.

List list = (List)parser.parseExpression           

                ("Members.?[Nationality == 'Serbian']")

                                              .getValue(societyContext);

4) Collection Projection: Projection allows a collection to drive the evaluation of a sub-expression and the result is a new collection. The syntax for projection is ![projectionExpression]. Most easily understood by example, suppose we have a list of inventors but want the list of cities where they were born. Effectively we want to evaluate 'placeOfBirth.city' for every entry in the inventor list. Using projection:

// returns [ 'Smiljan', 'Idvor' ]
List placesOfBirth = (List)parser.parseExpression("Members.![placeOfBirth.city]");

Thanks
Manisha

Java implementation of Cucumber

I know how we use to struggle setting up cuke4duke, which was Ruby implementation of Cucumber. Installing JRuby, gems and something or other were bound to go wrong.

Good news is java implementation for cucumber is released. (http://aslakhellesoy.com/post/20006051268/cucumber-jvm-1-0-0). I am yet to try it but after reading the article, I am super excited to use it.

More on this soon
Manisha

Return to AB tool

Once I had HAProxy up and running, it was time to do some performance test. I had two things in my mind for this round.

1) Making sure HAProxy is handling configuration changes gracefully, without breaking any existing connections.
2) Comparing the performance of server with and without HAProxy.

Started marching toward first goal. In HAProxy documentation there is a "-sf" option for soft restart. I started AB tool with 50000 request and while AB tool was busy firing those request to HAProxy, I took out one server from proxy setting and restarted the server using -sf option. I noticed AB tool gave me "connection reset by peer error". To understand what is happening and why my server is sending reset package to AB tool, I decided to capture the tcpdump and analyze it in wireshark.

I was surprised that to see that it's AB tool who is sending reset package to server. i.e. if I fired 100 requests using AB tool then I noticed 100 tcp connections with reset flag and 100 with fin, ack flag. Believe me at this point I am totally confused and not sure why the heck AB tool is behaving in such a way. I still don't know. Then I decided to use JMeter. Did similar exercise and the tcp dump was much cleaner. I noticed only 100 connection with fin,ack flag and no connections in reset mode. Much better.

So started looking to my original issue of dropping connection since JMeter log was also showing connection refused when I tried doing configuration swap. "option redispatch", "option httpchk" and "retries" options saved my day. After specifying these options I was not getting Connection refused any more while doing the changes in the configuration.

One down one more to go. Since I noticed tcp reset flag weirdness with AB tool thought it would be better to use JMeter for second round. Without HAProxy performance of my service was 4500+ req/sec using AB tool. I wanted to capture same thing with JMeter and it reported back 1800+ req/sec.

Ohhh Ohhh. Something wrong again. Started reading about JMeter and found out that JMeter’s HTTP client is slower than that of AB. You can use JMeter to find out if a change to your webapp, your tomcat installation or your JVM, accelerated or slows the response times of the webpage however you cannot use JMeter to determine the server’s maximum number of requests per second that it can successfully serve because of JMeter’s HTTP client appears to be slower than Tomcat’s server code.

So had to come back to my old love AB.
Manisha

Cause of "Connect reset by peer" error message using AB Tool

One more interesting finding about AB Tool (Apache Benchmarking tool) on linux.

During one of my recent runs I notices a very weird behavior. I was getting "Connection reset by peer" error message consistently when I was firing 15000 concurrent request with 90000 total number of request. I started getting successful response back when I fired 15000 concurrent request with 30000 total number of request. (i.e. ab -c 15000 -n 30000 vs. ab -c 15000 -n 90000) . I fired the 15k and 30k combination in a loop 100 times and every time I got successful result. I wanted to know why I am getting connection reset when I am firing 90K request in one process vs firing much more across multiple process.

I observed "Possible SYN flooding on port 80. Sending cookies" error message in the redhat's kernal log(/var/log/message). Since AB tool was firing lots of request the OS considered it as threat. :0.

I disabled tcp_syncookie by firing sysctl -w net.ipv4.tcp_syncookies=0 command and was being able to fire request much more than 90K. There are other parameters which you can be modify to increase the syncookie limit and time wait limit which might be helpful incase you don't have layers of firewall protecting your server. I will try mentioning them in my next blog.

Hope it helps

CXF Rest service renders JSON with @ for attributes

In my current project I am exposing lots of Rest Services in Java using CXF. All the service should be able to render response in json and xml format. This can very easily be done by just using a annotation(@Produces({ MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON })) on the exposed service method. To verify the service is rendering correct response based on passed header type (accept:application/json or accept:application/xml).

I used JsonView firefox plugin. (I like this plugin since you get formatted JSON back vs one line response which you get if you will use Poster firefox plugin).

I noticed that JSON output has extra '@' symbol for attributes. i.e.

......"Programs": {"Channel": {
"@ID": "E",
"TopName": "The Fresh Prince of Bel-Air",
"BottomName": "Bang the Drum Ashley",
"StartTime": {
"@utc": "1325741400000",
"$": "2012-01-05T05:30:00"
},...........

Then after googling I found that CXF default JSON render is Jettison which produced this sort of output. (Why Jettison behaves in such a way is still mystery. I have to it figure out sooner than later.)

If you will replace the default JSON renderer Jettison with Jackson problem will be resolved and it' very simple to provide your custom Provider. I added following lines into my spring.xml

<bean id="jacksonObjectMapper" class="org.codehaus.jackson.map.ObjectMapper">
<property name="serializationConfig.dateFormat">
<bean class="java.text.SimpleDateFormat">
<constructor-arg value="EEE, d MMM yyyy HH:mm:ss z" />
</bean>
</property>
</bean>

<bean id="jsonProvider" class="org.codehaus.jackson.jaxrs.JacksonJaxbJsonProvider">
<property name="mapper" ref="jacksonObjectMapper" />
</bean>

and added the provider into my server declaration i.e

<jaxrs:server id="xService" address="/x">
<jaxrs:serviceBeans>
<ref bean="xServices" />
</jaxrs:serviceBeans>
<jaxrs:providers>
  <ref bean="jsonProvider"/>
</jaxrs:providers>
</jaxrs:server&g

After adding Jackson as my provider. I redeployed my project and JSON was coming without @ sign. :))))

"programList": [{
"isHD": true,
"isClosedCaptioned": false,
"length": 120,
"description": "",
},
"startTime": {
"textValue": "2012-01-05T17:00:00",
"utcStartTime": 1325782800000 ....

-Enjoy
Manisha