1 Reply Latest reply on Feb 11, 2014 10:01 AM by René van Wijk

    JBoss EAP 6 On JRockit - Memory Leak

    paata_lom

      hello team.

       

      I have memory leak problem on jboss and jrockit.

       

      My Environment :

       

      1. OS :          

      CentOS release 6.4 (Final)
      

       

      2. JRockit :     

       java version "1.6.0_45"
           Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
           Oracle JRockit(R) (build R28.2.7-7-155314-1.6.0_45-20130329-0641-linux-x86_64, compiled mode)
      

       

      3. Application Server:

       JBoss EAP 6.2.0.GA 
      

       

      4. Application

       Large EJB Application (100 and more EJB Beans (Stateless, Stateful,  MDB, Timers and so on)
      

       

       

      Everything works fine on older application server versions (4.3 , 4.2)

      But now I have Problem

       

      Of course I know that problem is new version - and i have discussion on JBoss forums.

       

      but guys I have question about jrockit with you:

       

      What is the option "Other" in memory ??

       

      [jboss@jboss-new bin]$ jrcmd 17114  print_memusage
      17114:
      Total mapped                       8457864KB           (reserved=2983100KB)
      -              Java heap              3145728KB           (reserved=0KB)
      -              GC tables            105232KB          
      -          Thread stacks       46412KB           (#threads=138)
      -          Compiled code       1048576KB           (used=12257KB)
      -               Internal                   1480KB          
      -                     OS       170324KB          
      -                  Other       3639056KB          
      -            Classblocks         10496KB           (malloced=9631KB #28393)
      -        Java class data       289536KB           (malloced=192391KB #133697 in 28393 classes)
      - Native memory tracking     1024KB           (malloced=294KB #10)
      
      
      [jboss@jboss-new bin]$ 
      

       

      This size increases every time - and took entire amount of RAM.

       

       

      Thank in Advance.

       

      Regards,

      Paata Lominadze

        • 1. Re: JBoss EAP 6 On JRockit - Memory Leak
          René van Wijk

          Not sure what the 'other' is, but it is probably best shown by using an example. By using displayMap we can display a memory map of various JVM subsystems and libraries that are loaded, including third-party libraries.

           

          ./jrcmd 4523 print_memusage displayMap

          Total mapped                  3984796KB           (reserved=2978740KB)

          -              Java heap       524288KB           (reserved=0KB)

          -              GC tables        17548KB         

          -          Thread stacks        20276KB           (#threads=39)

          -          Compiled code      1048576KB           (used=14224KB)

          -               Internal         1672KB         

          -                     OS       146924KB         

          -                  Other      2092648KB         

          -            Classblocks         7424KB           (malloced=7381KB #20064)

          -        Java class data       124416KB           (malloced=124411KB #91048 in 20064 classes)

          - Native memory tracking         1024KB           (malloced=118KB #10)

          +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

              OS                          *java    r x 0x0000000000400000.(     76KB)

              OS                          *java    rw  0x0000000000612000.(      4KB)

              OS                        *[heap]    rw  0x00000000007c1000.(    132KB)

             INT                           Poll    r   0x000000007fffe000 (      4KB)

             INT                         Membar    rw  0x000000007ffff000.(      4KB)

             MSP              Classblocks (1/2)    rw  0x00000000df8c0000 (   6912KB)

             MSP              Classblocks (2/2)    rw  0x00000000dff80000 (    512KB)

            HEAP                      Java heap    rw  0x00000000e0000000.( 524288KB)

              OS                    *ld-2.12.so    r x 0x0000003664400000.(    128KB)

              OS                    *ld-2.12.so    r   0x000000366461f000 (      4KB)

              OS                    *ld-2.12.so    rw  0x0000003664620000 (      4KB)

              OS                   **ld-2.12.so    rw  0x0000003664621000.(      4KB)

          ...

             OS           *gconv-modules.cache    r   0x00007f8f2e4a0000 (     28KB)

          THREAD                     Stack 4630    rwx 0x00007f8f2e4a7000 (      8KB)

          THREAD                     Stack 4630        0x00007f8f2e4a9000 (     12KB)

          THREAD                     Stack 4630    rwx 0x00007f8f2e4ac000 (    244KB)

             MSP         Java class data (5/37)    rw  0x00007f8f2e4e9000 (  14336KB)

          ...

             MSP         Java class data (9/37)    rw  0x00007f8f2fa40000 (   5888KB)

                                                   rw  0x00007f8f30000000 (    188KB)

                                                       0x00007f8f3002f000 (  65348KB)

                                                   rw  0x00007f8f34000000 (    132KB)

                                                       0x00007f8f34021000 (  65404KB)

                                                   rw  0x00007f8f38000000 (    412KB)

                                                       0x00007f8f38067000.(  65124KB)

             MSP        Java class data (10/37)    rw  0x00007f8f3c034000 (  34048KB)

                                                   rw  0x00007f8f3e174000 (   8200KB)

             MSP        Java class data (11/37)    rw  0x00007f8f3e976000 (    256KB)

              OS                     *libhpi.so    rw  0x00007f8fb37fc000 (      8KB)

              OS                    **libhpi.so    rw  0x00007f8fb37fe000 (      4KB)

            CODE                  Compiled code    rwx 0x00007f8fb37ff000 (     64KB)

            CODE                  Compiled code    rwx 0x00007f8fb380f000 (    128KB)

            CODE                  Compiled code    rwx 0x00007f8fb382f000 (     64KB)

          ...

            MSP        Java class data (37/37)    rw  0x00007f8ff83a1000 (    512KB)

              GC Modified Union Set (committed)    rw  0x00007f8ff8421000 (    132KB)

              GC                     Card table    rw  0x00007f8ff8442000 (   1024KB)

              GC        Object bits (committed)    rw  0x00007f8ff8542000 (   8196KB)

           

          Here

          - thread is thread related (such as thread stacks)

          - int, internal use (such as pointer pages)

          - heap, chunk used by JRockit for the Java heap

          - os, mapped directly from the operating system, such as third party DLLs or shared objects

          - msp, memory space. a memory space is a native heap with a specific purpose, for example native memory allocation inside the JVM

          - gc, garbage collection related, for example live bits

          - code, compiled code

           

          The 'other' memory space looks to me (from the blank entries in the above print-out) like they are a memory pages to are still not used. When the JVM starts it mappes an amount of memory. To my knowledge JRockit uses mmap (mmap(2) - Linux manual page), which creates a new mapping in the virtual address space. JRockit reserves an amount of memory (Java Heap (heap where your object instances go) + its own runtime (all the others)).

           

          To see where the growth is in the various memory spaces, you can use 'print_memusage baseline', after which you can execute print_memusage again, for example,

           

          ./jrcmd 4523 print_memusage scale=M

          4523:

          Total mapped                     3896MB      +4MB (reserved=2905MB -3MB)

          -              Java heap          512MB           (reserved=0MB)

          -              GC tables           17MB         

          -          Thread stacks           19MB           (#threads=39)

          -          Compiled code         1024MB           (used=14MB +1MB)

          -               Internal            1MB         

          -                     OS          143MB         

          -                  Other         2043MB         

          -            Classblocks            7MB           (malloced=7MB #20596 +532)

          -        Java class data          126MB      +4MB (malloced=125MB +4MB #93640 +2592 in 20596 classes)

          - Native memory tracking            1MB           (malloced=0MB #20 +10)

           

          Note the additional column that prints out the difference in memory usage in relation to the baseline. You can also monitor native allocations by using trace_alloc_sites, memory allocations can then be displayed with different levels of detail using the level argument.

           

          ./jrcmd 4523 print_memusage trace_alloc_sites=true

          4523:

          Total mapped                  3989660KB   +4864KB (reserved=2974732KB -4008KB)

          -              Java heap       524288KB           (reserved=0KB)

          -              GC tables        17548KB         

          -          Thread stacks        20276KB           (#threads=39)

          -          Compiled code      1048576KB           (used=15265KB +1040KB)

          -               Internal         1672KB         

          -                     OS       146924KB         

          -                  Other      2092648KB         

          -            Classblocks         7680KB    +256KB (malloced=7669KB +287KB #20596 +532)

          -        Java class data       129024KB   +4608KB (malloced=128967KB +4555KB #93640 +2592 in 20596 classes)

          - Native memory tracking         1024KB           (malloced=236KB +118KB #20 +10)

           

          ./jrcmd 4523 print_memusage level=3

          4523:

          Total mapped                  3989660KB   +4864KB (reserved=2974732KB -4008KB)

          -              Java heap       524288KB           (reserved=0KB)

          -              GC tables        17548KB         

          -          Thread stacks        20276KB           (#threads=39)

          -          Compiled code      1048576KB           (used=15265KB +1040KB)

          -               Internal         1672KB         

          -                     OS       146924KB         

          -                  Other      2092648KB         

          -            Classblocks         7680KB    +256KB (malloced=2KB -7379KB #4 -20060) Not fully traced

          -        Java class data       129024KB   +4608KB (malloced=26KB -124385KB #16 -91032 in 20596 classes) Not fully traced.

          - Native memory tracking         1024KB           (malloced=118KB #10) Not fully traced.

               gather_memorymap_database                     memtrace.c: 206         91KB     +91KB (#1 +1)

                     gather_memory_usage                  osal_mspace.c:5142          7KB      +7KB (#4 +4)

            msGatherMSpacesUsageDatabase                  osal_mspace.c:6128          2KB      +2KB (#1 +1)

            msGatherMSpacesUsageDatabase                  osal_mspace.c:6134         16KB     +16KB (#1 +1)

           

          Note this is more on the JVM level, in your case in might be beneficial to investigate what is happening on the java heap. By using print_object_summary you can get insight how memory on the heap is used on a per-class basis. To get to the bottom of where the memory leak is you can use the memory-leak-detector (an example of its use can be found here Middleware Snippets: Fast, Faster, JRockit). You can also obtain a heapdump that can be analyzed by using for example MAT (see for an example here Middleware Snippets: Visualizing Class Loading). To obtain a heapdump you can run the command, for example,

           

          [weblogic@machine1 bin]$ ./jrcmd 4523 runsystemgc full=true fullcompact=true

          4523:

           

           

          [weblogic@machine1 bin]$ ./jrcmd 4523 hprofdump filename=/home/weblogic/dump.hprof

          4523:

          Wrote dump to /home/weblogic/dump.hprof

           

          Note that this first issues a full GC by using the runsystemgc command.