This discussion is archived
8 Replies Latest reply: Feb 16, 2011 12:40 PM by jschellSomeoneStoleMyAlias RSS

[JNI beginner] watching memory allocation by native code

jduprez Pro
Currently Being Moderated
Hello,

I have a JSE application that uses JNI to access legacy functionality.
While setting up the app's launching parameters, in particular the memory settings, I realize that I have no information as to how much memory the native code allocates. Indeed, I don't even have a proof that the native code is not leaking memory!
I have used both -Xloggc and commercial profling tools in the past, to check my Java apps for memory leaks. But I had not realized back then, that these tools were not giving me stats about anything called "native" ("native heap area", or whatever).

If I am to believe the Oracle documentation (http://www.oracle.com/technetwork/java/javase/memleaks-137499.html#gbyvk), there needs special (non-Java) tooling to measure memory consumption due to allocations in the native code.
- Am I correct that the memory allocated by the native code is not reported by the Java-level tools (e.g. HotSpot's -Xloggc flag)?

Indeed, so far I don't need to trace each allocation spot or allocated object, I just need to monitor the total allocated size, and check it does not grow over time.
In particular, if that total is already included in the total heap as reported by the loggc, and I can witness that the total doesn't drift, I'm fine.

Thanks in advance.

J.

Edited by: jduprez on Feb 15, 2011 5:05 PM
  • 1. Re: [JNI beginner] watching memory allocation by native code
    tschodt Pro
    Currently Being Moderated
    jduprez wrote:
    Indeed, so far I don't need to trace each allocation spot or allocated object, I just need to monitor the total allocated size, and check it does not grow over time.
    In particular, if that total is already included in the total heap as reported by the loggc, and I can witness that the total doesn't drift, I'm fine.
    I would look for a way to keep track of malloc(), free(), indeed
    Google ( [url http://www.google.com/search?q=c+lib+malloc+free+hooks]c lib malloc free hooks ) looks promising.
  • 2. Re: [JNI beginner] watching memory allocation by native code
    PHCharbonneau Explorer
    Currently Being Moderated
    Hi J.

    I agree with tschodt as keeping track of the malloc(), free() is key.

    If you are using AIX IBM VM, you can also easily monitor the native memory allocation (malloc operations) via the svmon OS command.

    ## Native Heap memory footprint for a 32-bit IBM VM
    svmon -P <java process pid> -m -r | grep 'working storage'

    ## Complete 32-bit VM 256 MB segment view
    svmon -P <java process pid> -m –r

    We have many AIX production systems in your environment and svmon has proven itself to be quite useful at this point to keep track on the native memory footprint of our Java EE applications; including confirming some native memory leak issues we were facing in production.

    Regards,
    P-H
    http://javaeesupportpatterns.blogspot.com/
  • 3. Re: [JNI beginner] watching memory allocation by native code
    jduprez Pro
    Currently Being Moderated
    tschodt wrote:
    jduprez wrote:
    Indeed, so far I don't need to trace each allocation spot or allocated object, I just need to monitor the total allocated size, and check it does not grow over time.
    In particular, if that total is already included in the total heap as reported by the loggc, and I can witness that the total doesn't drift, I'm fine.
    I would look for a way to keep track of malloc(), free(), indeed
    Google ( [url http://www.google.com/search?q=c+lib+malloc+free+hooks]c lib malloc free hooks ) looks promising.
    Thanks.
    However, well, that's the approach suggested already in the article I linked to in the OP.
    Plus, I don't need the detailed stats, just a live total to monitor for drift.
    Sorry if this sounds like I'm complaining against tools for which I pay nothing, but I'm surprised the (otherwise vastly instrumented) JDK does not monitor that. I mean, I don't expect the JDK to trace individual "native objects" in code it hasn't compiled itself. But monitoring the amount of bytes seems reachable (mmm, I have probably done too much Java for all those years, are the malloc/free function statically linked into a compiled C DLL)?

    Anyway, I don't have control over the whole native code: we're developping a JNI "wrapper" DLL that itself calls the services of a 3rd-party DLL (of which I don't have the source). If system tests reveal unexpected memory usage, I don't plan to investigate what happens in the 3rd-party library, I'll just ask the vendor to investigate (but I would need reasonable evidence that the memory usage grows, and not because of the Java-allocated memory, to have some leverage on the 3rd-party team).

    On the other hand, I have control over the "wrapper" part, and I may instrument that one using techniques such as you suggested (rant: another good reason why I deem Java make people more productive).

    Reards,

    J.
  • 4. Re: [JNI beginner] watching memory allocation by native code
    jduprez Pro
    Currently Being Moderated
    Hello,
    thanks for the svmon tip.
    I'm not on AIX, but on Windows and Linux. I'll see whether my Linux distro features such a tool (at least it has ps -aux that lists the RSS per-PID).

    Regards,

    J.
  • 5. Re: [JNI beginner] watching memory allocation by native code
    PHCharbonneau Explorer
    Currently Being Moderated
    Hi jduprez,

    That is correct, at a minimum, using ps –aux, you can still monitor your process size. Since the Java Heap is predictable size, any grow of your Java process overtime can definitely point to a Native Memory leak.

    We had one Native Memory leak issue about 2 years ago on Linux OS and 64-bit Java VM. We could see (via ps command) our Java process growing overtime. The root cause ended-up being a native memory leak triggered from a monitoring tool agent that was installed.

    Regards,
    P-H
  • 6. Re: [JNI beginner] watching memory allocation by native code
    tschodt Pro
    Currently Being Moderated
    PHCharbonneau wrote:
    The root cause ended-up being a native memory leak triggered from a monitoring tool agent that was installed.
    The irony would be complete if the monitoring tool agent was monitoring memory. o_O
  • 7. Re: [JNI beginner] watching memory allocation by native code
    PHCharbonneau Explorer
    Currently Being Moderated
    tschodt, so true :)

    Part of the agent role was in fact memory monitoring; but its primary role was Java / Java EE components profiling.

    Still, it did re-inforce the fact that we still need to monitor the monitorig tools :)

    Regards,
    P-H
  • 8. Re: [JNI beginner] watching memory allocation by native code
    jschellSomeoneStoleMyAlias Expert
    Currently Being Moderated
    jduprez wrote:
    I don't expect the JDK to trace individual "native objects" in code it hasn't compiled itself. But monitoring the amount of bytes seems reachable (mmm, I have probably done too much Java for all those years, are the malloc/free function statically linked into a compiled C DLL)?
    How is the VM shared library going to replace/intercept the malloc methods when the VM starts up?

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points