If there is a leak in our application code, JVM should run out of heap space & throw an out of memory error.
In our case the heap size is within the set max value of 2GB but the java process memory usage is going upto 12GB.
Googling with the following seemed to suggest possibilities.
java RSS memory
In that I found the following interesting
do you know java heap and native heap ? the -Xms and -Xmx are setting only for java heap.
If you doubt there may be a native heap memory leak of jvm itself, you can use a toolkit named valgrind to find it.
By the way , you should also check if there is jni invoking in jboss itself and you application , the jvm is writted by c++ and a little c, when you are using jdk api, it will cost memory in jvm process memory address space and java heap, and some java code will allocate memory outside the jvm heap area, such as java.nio.ByteBuffer.allocateDirect.
You should also add -XX:+UseCompressedOops option to your jboss startup command line when you are using sun hotspot jvm 6u14 - 6u23, after 6u23, the option is open default.
At last, there is a article in oracle support, the main points are following, hope to be helpful to you.
Reasons for reaching this limit include:
• Memory leaks in calls to native libraries
• Loading many native libraries that fill up process address space
• Suboptimal JVM configurations due to changed memory requirements for an application
• Direct Buffers that are allocated outside of the garbage-collected heap
• Mapped Buffers, created by mapping a region of a file into memory
• mmap files using native implementations
• Any native resources that will tie up the address space