3 Replies Latest reply: Feb 9, 2010 1:31 PM by jschellSomeoneStoleMyAlias RSS

    Memory leak on the native side

    843798
      Hello,

      I am hoping someone here can offer some troubleshooting advice, as I am completely stumped. I am running JBoss 5.1 with JDK 1.6_u18 (same problem with u17 and u16 too)


      - 32-bit, Linux (RHEL 5).
      - min/max heap setting of 1024M
      - permgen max of 256M
      - Thread stack size of 128K
      - No JNI

      My problem: The memory footprint of the JVM slowly grows until it hits the 3G OS limit. This takes about 8 hours under moderate load. At this time, it of course dies as it has no more addressable memory left.

      Here is the strange part: I have used every possible memory debugging tool (jmap, Eclipse MAT, etc) and nothing looks out of the ordinary in my Java heap. Thread count stays at a reasonable 350 threads, Java heap size stabilizes at about 500M. For the first hour or so, the JVM footprint stays at about 1.7G, which makes sense. After that it starts to slowly grow until in exceeds the 3G limit.

      What can I do to figure out where the leak is occurring? There is clearly some native resource that is being allocated but not freed. As I indicated, all the Java analysis tools report a healthy, stable heap and thread count.

      Thanks in advance.

      Jon
        • 1. Re: Memory leak on the native side
          jschellSomeoneStoleMyAlias
          Do you have any JNI or 3rd party libraries that use JNI?
          Are you using any custom class loaders?
          Have you in any way, via command line options, adjusted the VM behavior and specifically the GC behavior?
          The Sun VM used to have a problem with allocated Thread instances that were never started. I think that might have been fixed though.

          As a debugging aid try reducing the vm max heap size significantly. That should cause it to fail sooner.
          • 2. Re: Memory leak on the native side
            843798
            Thanks for the reply.

            I have confirmed with -verbose:jni that the only JNI libraries getting loaded are those belonging to the JDK. My application does make heavy use of the ProcessClassLoader from the Jboss JBPM library, but classes all seem to be unloading normally, and my Permgen usage stays very low and stable. Is there anything I should look at in regards to this class loader? I have tried both a lack of GC parameters and specified the concurrent mark sweep collector, with the same results.
            • 3. Re: Memory leak on the native side
              jschellSomeoneStoleMyAlias
              jwmfe wrote:
              and my Permgen usage stays very low and stable. Is there anything I should look at in regards to this class loader?
              First part suggests it is not a class loader problem.