This discussion is archived
10 Replies Latest reply: Nov 24, 2011 11:37 PM by 900093 RSS

High Java memory consumption.

900093 Newbie
Currently Being Moderated
Hello,

We are developing a solution using Servoy, which is a database server framework built in java. We have now 2 Servoy instances running on Debian linux servers, one with 64bit, both of them running java 1.6 update 26. We are having lot of crashes of the server with this error log:

-------
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 32776 bytes for Chunk::new
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (allocation.cpp:317), pid=12359, tid=1777961872
#
# JRE version: 6.0_26-b03
# Java VM: Java HotSpot(TM) Server VM (20.1-b02 mixed mode linux-x86 )

--------------- T H R E A D ---------------

Current thread (0x092c8400): JavaThread "C2 CompilerThread1" daemon [_thread_in_native, id=12368, stack(0x69f18000,0x69f99000)]

Stack: [0x69f18000,0x69f99000], sp=0x69f95fe0, free space=503k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x7248b0]
-------

We checked both the heap memory and the memory used by java in this case and we notice a highly increase of the total java memory, while the heap memory is resonable:

------
JVM Information
java.vm.name=Java HotSpot(TM) Server VM
java.version=1.6.0_26
java.vm.info=mixed mode
java.vm.vendor=Sun Microsystems Inc.

Operating System Information
os.name=Linux
os.version=2.6.29-xs5.5.0.17
os.arch=i386

System Information
Heap memory: allocated=141440K, used=105555K, max=699072K
None Heap memory: allocated=49312K, used=49018K, max=180224K
------
root 16388 2.5 43.0 2829220 1811772 ? Sl 20:11 3:43 java -Djava.awt.headless=true -Xmx768m -Xms128m -XX:MaxPermSize=128m -classpath .:lib/ohj-jewt.jar:lib/MRJAdapter.jar:lib/compat141.ja
------

Right now we are running it on the 64bit machine with -Xmx256m -Xms64m, and the memory is the same as on the 32bit one.

We also tried different memory configurations, but the result is the same. Java goes up to more the 2GB of memory used, while the heap is about 100MB - 400MB. In the output above you can see 1.8GB used, but this is at startup, after few hours it goes to more then 2GB - 2.5GB, then it crashes in at most few days, sometimes in few hours or even less.
  • 1. Re: High Java memory consumption.
    gimbal2 Guru
    Currently Being Moderated
    Okay, but why are you not taking this up with the Servoy customer service?
  • 2. Re: High Java memory consumption.
    900093 Newbie
    Currently Being Moderated
    We are talking with them for more then one month now. Since the heap memory is OK they claim is not their problem, but there might be a problem with the java machine. Because as I shown in the log messages, the memory used by java is too high, while the heap is OK.
    We also discussed with other people that are developing using Servoy and the heap memory for their solutions looks similar with ours, while they have a much low memory for the java process, as the OS displays.
    We also thought that there might be some leaks in the app, but those leaks would be visible in the heap memory.
    So, to make myself clear again, the heap memory is fine, but overall, java uses to much memory from the OS, and at a point, when the app needs more memory, it doesn't have any left, so it crashes.
    Even at startup, the java process consumes more then 1GB of memory, and after few queries to the server, while the heap remains in a 200MB limit, the java process memory goes up to 2GB.
  • 3. Re: High Java memory consumption.
    gimbal2 Guru
    Currently Being Moderated
    The bastards. Well then, do you have another database driven application that is not built with their technology that you can deploy on the same machine? If that works fine then I remain pointing my finger at their stuff.
    We also discussed with other people that are developing using Servoy and the heap memory for their solutions looks similar with ours, while they have a much low memory for the java process, as the OS displays.
    It would be more interesting to learn which OS, which JVM version and which servlet container / application server they use. That opens up the possibility for you to test in a similar environment to see if that changes anything.
    We also thought that there might be some leaks in the app, but those leaks would be visible in the heap memory.
    Not if the leak is in a native module (DLL, so), that is memory managed outside of the JVM.


    To get any more information you'll have to get to work with some kind of profiler to try and figure out where the drain is coming from I'm afraid.
  • 4. Re: High Java memory consumption.
    900093 Newbie
    Currently Being Moderated
    Can the profiler see more then the heap? OK, I'll try to profile the server.

    Servoy uses Tomcat 5.

    The OS we run on is Debian. The 64bit machine is this:

    JVM Information
    java.vm.name=Java HotSpot(TM) 64-Bit Server VM
    java.version=1.6.0_26
    java.vm.info=mixed mode
    java.vm.vendor=Sun Microsystems Inc.

    Operating System Information
    os.name=Linux
    os.version=2.6.32-5-amd64
    os.arch=amd64

    and the current memory usage is this:

    System Information
    Heap memory: allocated=253440K, used=228114K, max=253440K
    None Heap memory: allocated=112512K, used=112223K, max=180224K

    , while the java process memory is this:

    root 11629 6.9 44.4 2105640 1829808 hvc0 Sl 11:42 14:18 java -Djava.awt.headless=true -Xmx256m -Xms64m -XX:MaxPermSize=128m -classpath .:lib/ohj-jewt.jar:lib/MRJAdapter.jar:lib/compat141.jar:lib/commons-codec.jar:lib/commons-httpclient.jar:lib/activation.jar:lib/antlr.jar:lib/commons-collections.jar:lib/commons-dbcp.jar:lib/commons-fileupload-1.2.1.jar:lib/commons-io-1.4.jar:lib/commons-logging.jar:lib/commons-pool.jar:lib/dom4j.jar:lib/help.jar:lib/jabsorb.jar:lib/hibernate3.jar:lib/j2db.jar:lib/j2dbdev.jar:lib/jdbc2_0-stdext.jar:lib/jmx.jar:lib/jndi.jar:lib/js.jar:lib/jta.jar:lib/BrowserLauncher2.jar:lib/jug.jar:lib/log4j.jar:lib/mail.jar:lib/ohj-jewt.jar:lib/oracle_ice.jar:lib/server-bootstrap.jar:lib/servlet-api.jar:lib/wicket-extentions.jar:lib/wicket.jar:lib/wicket-calendar.jar:lib/slf4j-api.jar:lib/slf4j-log4j.jar:lib/joda-time.jar:lib/rmitnl.jar:lib/networktnl.jar com.servoy.j2db.server.ApplicationServer

    About another app, I'm confused, because we do have another java app that uses much less:

    root      1149  0.3  2.1 843312 86988 ?        Sl   10:16   0:58 java -Xmx512m -Xms128m -cp noaa_server.zip server.WebServer

    and for this one, the heap is this:

    Memory: total: 129892352, free: 114666408, maximum: 518979584

    So yes, this one looks better. It uses 15MB from heap and 80MB in total from ram.

    But still, I don't get it, how can the process memory be affected while the heap is much low?

    Edited by: 897090 on Nov 14, 2011 6:35 AM

    Edited by: 897090 on Nov 14, 2011 6:39 AM

    Edited by: 897090 on Nov 14, 2011 6:40 AM
  • 5. Re: High Java memory consumption.
    gimbal2 Guru
    Currently Being Moderated
    Like I said, when the memory is reserved in native code. That makes it exceptionally hard to track down.

    Anyone else besides me care to give an opinion on this?
  • 6. Re: High Java memory consumption.
    852650 Newbie
    Currently Being Moderated
    FWIW, I've noticed something similar with Solr 3.4 and Java 6u29 on RedHat 5.6: I set -Xmx32G and it takes 337 GB (!) of virtual memory. It used to obey the maximum with 6u26.
  • 7. Re: High Java memory consumption.
    900093 Newbie
    Currently Being Moderated
    Hi guys,

    Just switch to java7 and the system memory consumption decreased a lot. Have a look:

    root 20886 0.8 15.7 1397524 332020 ? Sl Nov17 5:31 ./jre1.7.0_01/bin/java -Djava.awt.headless=true -Xmx1024m -Xms128m -XX:MaxPermSize=128m -classpath .:lib/ohj-jewt.jar:lib/MRJAdapter.ja

    How can someone explain this. Shall I understand that there is a leak in the java6 u26. I'll try also with other java6 updates just out of curiosity to see if it's only that one or not.
  • 8. Re: High Java memory consumption.
    gimbal2 Guru
    Currently Being Moderated
    It might be an issue in the Linux JVM, who knows. It might be an issue of the JRE in combination with a specific Linux distribution. You can't really know unless you can find something about it in the Linux forums / bug reports. It might be worth making a bug report about to Oracle, if you have luck with the bug reporting tool.
  • 9. Re: High Java memory consumption.
    jschellSomeoneStoleMyAlias Expert
    Currently Being Moderated
    bogdan wrote:
    then it crashes in at most few days, sometimes in few hours or even less.
    If you are certain that there are absolutely no native (jni) code in anything that you are running then that is, by definition, a jvm bug.
    If any of your code, in any way, use native code then that is more likely source of the problem.

    If you are doing anything except setting the memory via command line options, such as GC options, then altering those might make the problem go away. It would however still be a vm bug (if there is no native code.)
  • 10. Re: High Java memory consumption.
    900093 Newbie
    Currently Being Moderated
    About native code I don't know what to say, but I just asked the people from servoy if their framework does it. I hope to get an answer soon. Definitely, I'm not calling any native code, or other system libraries.

    It might be indeed either a solely java bug or java + linux. I don't know if I mention this earlier but the problem appeared after an update to the linux kernel. I don't know if at that time the java was updated, but I'll try to find out. This might be helpful information. The thing is that the behavior is the same for both 32 bit and 64 bit machines with java6u26.

    I tried with java6u29 and java7 and they work great. Actually on the production server we are running for almost 4 days with java7 and it's stable, no crash, no slowdown, no restart in this period, and with less maximum memory. If it's going to last for a week then I trust it will go on fine.

    I'll keep updated!

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points