Since switching from Java 6 to Java 7 our application's memory foot print has increased significantly. Diagnosing the problem lead us to the java compiler. Our application generates, compiles and accesses source code at run-time via the JSR-000199 compiler API. We create a new instance of the compiler and class loader for every compilation, which may be sub-optimal but that is the way it is for now.
The behaviour we observer is:
in Java 6 our application uses under 300MB of memory and runs consistent.
in Java 7 the application uses as much memory as is available, plus a little bit extra outside the heap. For example on a 7.5GB heap on a 8GB system, the java process is killed by oom-killer after allocating over 8.8GB (8GB physical +1 GB swap).
The main object using up the space is the SharedNameTable object. One of the theories we were throwing around was that between Java 6 to Java 7 a week reference was changed to a soft reference, resulting in object being kept in memory for longer.
We are looking for any pointers which could explain our observation and maybe even some suggested option to get Java 7 to free up resources and utilise less memory. I fully appreciate that what we are doing is quite non-standard and would like to avoid getting into the reasons of how we got there.
There are some confusing things here.
Why are you letting the oom-killer kill your process? This would mean the Java program has been allowed more heap than it can actually use, so instead of java throwing a OutOfMemoryError, you kill the process?
Changing a weak reference to soft reference would still not exhaust the memory, at least if you had set your heap correctly.
As for the unSharedTable option, it doesn't look like you can easily switch it. Where is that method called from anyways? Who passes in the options?
The reason oom-killer kills my java process is because the process is using more than all the available memory. The system has got 8GB or RAM and 1GB of swap space. I set the heap to 7.5GB and 256MB permgen, leaving me with 256MB for the OS and 1GB of swap space for just in case. I appreciate the Java process will allocate more than the heap and permgen combined, but I didn't expect it to go over by 1.3 GB. Shortly before oom-killer did its job the java process was using around 8.8GB.
Maybe you can shed some light on how best to size the heap in this configuration.