2 Replies Latest reply: Jul 10, 2009 7:05 AM by 843829 RSS

    JVM CRASH->java.lang.OutOfMemoryError --> Out of swap space? --> REVISITED

    843829
      Hello all

      I am experiencing a JVM CRASH problem with my application deployed on
      JBoss.
      After searching a lot and reading many posts / reported bugs etc I think it's a
      good idea to start a fresh discussion about the topic!

      The platform:
      JDK version: jdk1.6.0_12
      OS version: Windows Server 2003
      Server R2, SP2 Enterprise Edition
      RAM: 4GB
      Virtual Memory: 2GB (paging file size for all drives)
      JBoss version: jboss-5.0.0.GA

      VM arguments:
      -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -Dorg.jboss.resolver.warning=true
      -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000
      The scenario:
      I have my Java EE application deployed on JBoss server. After every 2/3/4 days
      I am experiencing that the Java process gets terminated abruptly. I have experienced
      this 10 times now!
      I have checked the various generated hs_err_pidxxxx.log files. I have following
      observations

      1. I have found that 70% times the JVM got crashed because of
      *# java.lang.OutOfMemoryError: requested 2621440 bytes for GrET in
      C:\BUILD_AREA\jdk6_12\hotspot\src\share\vm\utilities\growableArray.cpp. Out of
      swap space?*
      *#*
      *# Internal Error (allocation.inline.hpp:42), pid=2500, tid=5040*
      *# Error: GrET in C:\BUILD_AREA\jdk6_12\hotspot\src\share\vm\utilities\growableArray.cpp*

      2. And all other cases (30% times) the crash is due to
      *# java.lang.OutOfMemoryError: requested 929600 bytes for Chunk::new. Out of
      swap space?*
      *#*
      *# Internal Error (allocation.cpp:218), pid=1908, tid=3420*
      *# Error: Chunk::new*

      I am surprised to see these two different errors for the JVM crash!
      I am attaching the logs for both the types of crashes here.
      What made me more suspicious is the following section.


      Error type 1
      Heap
       PSYoungGen      total 114112K, used 1233K [0x5c810000, 0x639d0000, 0x639d0000)
        eden space 112128K, 0% used [0x5c810000,0x5c810000,0x63590000)
        from space 1984K, 62% used [0x637e0000,0x63914500,0x639d0000)
        to   space 2176K, 0% used [0x63590000,0x63590000,0x637b0000)
       PSOldGen        total 932096K, used 182888K [0x239d0000, 0x5c810000, 0x5c810000)
        object space 932096K, 19% used [0x239d0000,0x2ec6a2a0,0x5c810000)
       PSPermGen       total 68864K, used 67172K [0x039d0000, 0x07d10000, 0x239d0000)
        object space 68864K, 97% used [0x039d0000,0x07b69158,0x07d10000)
      Error type 2
      Heap
       PSYoungGen      total 79424K, used 36671K [0x5c810000, 0x639d0000, 0x639d0000)
        eden space 63360K, 32% used [0x5c810000,0x5dc3daa8,0x605f0000)
        from space 16064K, 99% used [0x605f0000,0x615924f0,0x615a0000)
        to   space 27904K, 0% used [0x61e90000,0x61e90000,0x639d0000)
       PSOldGen        total 932096K, used 195081K [0x239d0000, 0x5c810000, 0x5c810000)
        object space 932096K, 20% used [0x239d0000,0x2f8526c8,0x5c810000)
       PSPermGen       total 69632K, used 69519K [0x039d0000, 0x07dd0000, 0x239d0000)
        object space 69632K, 99% used [0x039d0000,0x07db3c78,0x07dd0000)
      I am seeing PermGen space is almost used 100%. But I am also
      seeing that the PermGen space is just 69MB!
      Whereas I have given option -XX:MaxPermSize=512m to the JVM.
      I am not able to understand why JVM is not using all 512MB of
      specified space!
      Is there something which I am missing?

      Well now I have specified -XX: PermSize=512m option to VM. Let's see...


      Does this means all these crashes are because of FULL
      PermGen space? Or there is something which I am missing (though many posts and
      bug reports says that it's a bug in JVM)

      Please note that I am NOT
      performing any load test on the server. Simply means there is no load on server!

      *I would really very
      appreciate if the experts here shade some light on this!*


      Thanks & Best Regards,

      ~PP

      Edited by: preetam_pict on Jul 10, 2009 11:57 AM
        • 1. Re: JVM CRASH->java.lang.OutOfMemoryError --> Out of swap space? --> REVISITED
          843829
          error type 1
          #
          # An unexpected error has been detected by Java Runtime Environment:
          #
          # java.lang.OutOfMemoryError: requested 2621440 bytes for GrET in C:\BUILD_AREA\jdk6_12\hotspot\src\share\vm\utilities\growableArray.cpp. Out of swap space?
          #
          #  Internal Error (allocation.inline.hpp:42), pid=5316, tid=4232
          #  Error: GrET in C:\BUILD_AREA\jdk6_12\hotspot\src\share\vm\utilities\growableArray.cpp
          #
          # Java VM: Java HotSpot(TM) Server VM (11.2-b01 mixed mode windows-x86)
          # If you would like to submit a bug report, please visit:
          #   http://java.sun.com/webapps/bugreport/crash.jsp
          #
          
          ---------------  T H R E A D  ---------------
          
          Current thread (0x64000800):  VMThread [stack: 0x641f0000,0x64240000] [id=4232]
          
          Stack: [0x641f0000,0x64240000]
          [error occurred during error reporting (printing stack bounds), id 0x80000001]
          
          VM_Operation (0x666dfc14): ParallelGCSystemGC, mode: safepoint, requested by thread 0x67cd7800
          
          
          ---------------  P R O C E S S  ---------------
          
          Java Threads: ( => current thread )
            0x79ecb400 JavaThread "http-172.16.2.116-80-7" daemon [_thread_blocked, id=5816, stack(0x65420000,0x65470000)]
            0x6486ec00 JavaThread "WorkManager(2)-26" daemon [_thread_blocked, id=4188, stack(0x7c530000,0x7c580000)]
            0x7e270400 JavaThread "WorkManager(2)-25" daemon [_thread_blocked, id=1400, stack(0x7c4e0000,0x7c530000)]
            0x73c06400 JavaThread "WorkManager(2)-24" daemon [_thread_blocked, id=5592, stack(0x7c490000,0x7c4e0000)]
            0x67c8d400 JavaThread "WorkManager(2)-23" daemon [_thread_blocked, id=204, stack(0x7c440000,0x7c490000)]
            0x6522cc00 JavaThread "WorkManager(2)-22" daemon [_thread_blocked, id=3932, stack(0x7c3f0000,0x7c440000)]
            0x68964c00 JavaThread "WorkManager(2)-21" daemon [_thread_blocked, id=3480, stack(0x7c3a0000,0x7c3f0000)]
            0x79bdb400 JavaThread "WorkManager(2)-20" daemon [_thread_blocked, id=3400, stack(0x7c2b0000,0x7c300000)]
            0x740ba400 JavaThread "WorkManager(2)-19" daemon [_thread_blocked, id=2844, stack(0x7c260000,0x7c2b0000)]
            0x6480f400 JavaThread "WorkManager(2)-18" daemon [_thread_blocked, id=5684, stack(0x7c210000,0x7c260000)]
            0x78881c00 JavaThread "WorkManager(2)-17" daemon [_thread_blocked, id=2432, stack(0x7c1c0000,0x7c210000)]
            0x684c3400 JavaThread "WorkManager(2)-16" daemon [_thread_blocked, id=5552, stack(0x7c120000,0x7c170000)]
            0x75689c00 JavaThread "WorkManager(2)-15" daemon [_thread_blocked, id=3420, stack(0x7c0d0000,0x7c120000)]
            0x7a56f400 JavaThread "WorkManager(2)-14" daemon [_thread_blocked, id=4044, stack(0x77d40000,0x77d90000)]
            0x67bda400 JavaThread "WorkManager(2)-13" daemon [_thread_blocked, id=2808, stack(0x77cf0000,0x77d40000)]
            0x73da3400 JavaThread "WorkManager(2)-12" daemon [_thread_blocked, id=6084, stack(0x77b20000,0x77b70000)]
          
               ======================           MANY THREADS HERE      ====================== 
          
            0x002a7c00 JavaThread "DestroyJavaVM" [_thread_blocked, id=4828, stack(0x008c0000,0x00910000)]
            0x640a9400 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=2108, stack(0x64420000,0x64470000)]
            0x640a6800 JavaThread "CompilerThread1" daemon [_thread_blocked, id=6040, stack(0x643d0000,0x64420000)]
            0x640a1400 JavaThread "CompilerThread0" daemon [_thread_blocked, id=5460, stack(0x64380000,0x643d0000)]
            0x6409f800 JavaThread "Attach Listener" daemon [_thread_blocked, id=3340, stack(0x64330000,0x64380000)]
            0x6409e800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=6128, stack(0x642e0000,0x64330000)]
            0x64095400 JavaThread "Finalizer" daemon [_thread_blocked, id=1848, stack(0x64290000,0x642e0000)]
            0x64094000 JavaThread "Reference Handler" daemon [_thread_blocked, id=340, stack(0x64240000,0x64290000)]
          
          Other Threads:
          =>0x64000800 VMThread [stack: 0x641f0000,0x64240000] [id=4232]
            0x640ab400 WatcherThread [stack: 0x64470000,0x644c0000] [id=2416]
          
          VM state:at safepoint (normal execution)
          
          VM Mutex/Monitor currently owned by a thread:  ([mutex/lock_event])
          [0x002a6ca0] UNKNOWN - owner thread: 0x64000800
          [0x002a7100] UNKNOWN - owner thread: 0x67cd7800
          
          Heap
           PSYoungGen      total 114112K, used 1233K [0x5c810000, 0x639d0000, 0x639d0000)
            eden space 112128K, 0% used [0x5c810000,0x5c810000,0x63590000)
            from space 1984K, 62% used [0x637e0000,0x63914500,0x639d0000)
            to   space 2176K, 0% used [0x63590000,0x63590000,0x637b0000)
           PSOldGen        total 932096K, used 182888K [0x239d0000, 0x5c810000, 0x5c810000)
            object space 932096K, 19% used [0x239d0000,0x2ec6a2a0,0x5c810000)
           PSPermGen       total 68864K, used 67172K [0x039d0000, 0x07d10000, 0x239d0000)
            object space 68864K, 97% used [0x039d0000,0x07b69158,0x07d10000)
          
          Dynamic libraries:
          0x00400000 - 0x00424000      C:\Program Files\Java\jdk1.6.0_12\bin\java.exe
          0x7c800000 - 0x7c8c2000      C:\WINDOWS\system32\ntdll.dll
               ======================           MANY LIBS HERE      ====================== 
          0x76f90000 - 0x770da000      C:\Program Files\Java\jdk1.6.0_12\jre\bin\awt.dll
          0x6d310000 - 0x6d337000      C:\WINDOWS\system32\WINSPOOL.DRV
          0x77210000 - 0x77349000      C:\WINDOWS\system32\ole32.dll
          0x770e0000 - 0x771e3000      C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.3790.3959_x-ww_D8713E55\comctl32.dll
          0x77da0000 - 0x77df2000      C:\WINDOWS\system32\SHLWAPI.dll
          
          VM Arguments:
          jvm_args: -Djemail.config.file=C:\jboss-5.0.0.GA\server\default\descriptors\jemail-116.config -Dvelocity.config.file=C:\jboss-5.0.0.GA\server\default\descriptors\velocity-116.properties -Dprogram.name=run.bat -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.endorsed.dirs=C:\jboss-5.0.0.GA\lib\endorsed 
          java_command: org.jboss.Main -b 172.16.2.116
          Launcher Type: SUN_STANDARD
          
          Environment Variables:
          JAVA_HOME=C:\Program Files\Java\jdk1.6.0_12
          CLASSPATH=C:\JPW/lib/JavaProcessWatchdog.jar;C:\JPW/lib/log4j-1.2.15.jar;
          PATH=C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\WinSCP\;C:\Program Files\TortoiseSVN\bin;C:\Program Files\Java\jdk1.6.0_12\bin;C:\Sun\SDK\bin
          USERNAME=Administrator
          OS=Windows_NT
          PROCESSOR_IDENTIFIER=x86 Family 6 Model 23 Stepping 7, GenuineIntel
          
          
          
          ---------------  S Y S T E M  ---------------
          
          OS: Windows Server 2003 family Build 3790 Service Pack 2
          
          CPU:total 4 (4 cores per cpu, 1 threads per core) family 6 model 7 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3
          
          Memory: 4k page, physical 2097151k(2097151k free), swap 4194303k(4194303k free)
          
          vm_info: Java HotSpot(TM) Server VM (11.2-b01) for windows-x86 JRE (1.6.0_12-b04), built on Jan 17 2009 09:52:33 by "java_re" with MS VC++ 7.1
          
          time: Tue Jul 07 21:16:27 2009
          elapsed time: 100823 seconds
          • 2. Re: JVM CRASH->java.lang.OutOfMemoryError --> Out of swap space? --> REVISITED
            843829
            error type 2

            #
            # An unexpected error has been detected by Java Runtime Environment:
            #
            # java.lang.OutOfMemoryError: requested 929600 bytes for Chunk::new. Out of swap space?
            #
            #  Internal Error (allocation.cpp:218), pid=1908, tid=3420
            #  Error: Chunk::new
            #
            # Java VM: Java HotSpot(TM) Server VM (11.2-b01 mixed mode windows-x86)
            # If you would like to submit a bug report, please visit:
            #   http://java.sun.com/webapps/bugreport/crash.jsp
            # The crash happened outside the Java Virtual Machine in native code.
            # See problematic frame for where to report the bug.
            #
            
            ---------------  T H R E A D  ---------------
            
            Current thread (0x640a6800):  JavaThread "CompilerThread1" daemon [_thread_in_native, id=3420, stack(0x643d0000,0x64420000)]
            
            Stack: [0x643d0000,0x64420000]
            [error occurred during error reporting (printing stack bounds), id 0x80000001]
            
            
            Current CompileTask:
            C2:3769      org.apache.jasper.xmlparser.ParserUtils.convert(Lorg/apache/jasper/xmlparser/TreeNode;Lorg/w3c/dom/Node;)Lorg/apache/jasper/xmlparser/TreeNode; (202 bytes)
            
            
            ---------------  P R O C E S S  ---------------
            
            Java Threads: ( => current thread )
              0x64ed8400 JavaThread "ZipFile Lock Reaper" daemon [_thread_blocked, id=4188, stack(0x77210000,0x77260000)]
              0x69438400 JavaThread "http-172.16.2.116-80-11" daemon [_thread_in_Java, id=7000, stack(0x7fa20000,0x7fa70000)]
              0x65004400 JavaThread "http-172.16.2.116-80-6" daemon [_thread_in_native, id=7140, stack(0x7fd90000,0x7fde0000)]
              0x72b72c00 JavaThread "WorkManager(2)-98" daemon [_thread_blocked, id=4632, stack(0x7fe80000,0x7fed0000)]
              0x7b4f3c00 JavaThread "WorkManager(2)-97" daemon [_thread_blocked, id=7844, stack(0x7fe30000,0x7fe80000)]
              0x7def5400 JavaThread "WorkManager(2)-96" daemon [_thread_blocked, id=4680, stack(0x7fde0000,0x7fe30000)]
              0x6e693400 JavaThread "WorkManager(2)-95" daemon [_thread_blocked, id=7440, stack(0x7fd40000,0x7fd90000)]
              0x7482a400 JavaThread "WorkManager(2)-94" daemon [_thread_blocked, id=6696, stack(0x7fcf0000,0x7fd40000)]
              0x69a0cc00 JavaThread "WorkManager(2)-93" daemon [_thread_blocked, id=6652, stack(0x7fca0000,0x7fcf0000)]
              0x709f8c00 JavaThread "WorkManager(2)-92" daemon [_thread_blocked, id=6824, stack(0x7fc50000,0x7fca0000)]
              0x659ecc00 JavaThread "WorkManager(2)-91" daemon [_thread_blocked, id=3796, stack(0x7fc00000,0x7fc50000)]
              0x653d4400 JavaThread "WorkManager(2)-90" daemon [_thread_blocked, id=4516, stack(0x7fbb0000,0x7fc00000)]
              0x64efac00 JavaThread "WorkManager(2)-89" daemon [_thread_blocked, id=2244, stack(0x7fb60000,0x7fbb0000)]
              0x78235400 JavaThread "WorkManager(2)-88" daemon [_thread_blocked, id=5348, stack(0x7fb10000,0x7fb60000)]
              0x66c6d400 JavaThread "WorkManager(2)-87" daemon [_thread_blocked, id=5512, stack(0x7fac0000,0x7fb10000)]
              0x79f6b400 JavaThread "WorkManager(2)-86" daemon [_thread_blocked, id=3860, stack(0x7fa70000,0x7fac0000)]
            
                 ======================           MANY THREADS HERE      ====================== 
            
              0x67760400 JavaThread "JBoss System Threads(1)-1" daemon [_thread_in_native, id=5448, stack(0x66560000,0x665b0000)]
              0x67651400 JavaThread "Timer-Log4jService" daemon [_thread_blocked, id=3752, stack(0x66470000,0x664c0000)]
              0x6499a400 JavaThread "Timer-0" daemon [_thread_blocked, id=960, stack(0x64bd0000,0x64c20000)]
              0x002a7c00 JavaThread "DestroyJavaVM" [_thread_blocked, id=4132, stack(0x008c0000,0x00910000)]
              0x640a9400 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=1092, stack(0x64420000,0x64470000)]
            =>0x640a6800 JavaThread "CompilerThread1" daemon [_thread_in_native, id=3420, stack(0x643d0000,0x64420000)]
              0x640a1400 JavaThread "CompilerThread0" daemon [_thread_blocked, id=4084, stack(0x64380000,0x643d0000)]
              0x6409fc00 JavaThread "Attach Listener" daemon [_thread_blocked, id=3096, stack(0x64330000,0x64380000)]
              0x6409e800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=5916, stack(0x642e0000,0x64330000)]
              0x64095800 JavaThread "Finalizer" daemon [_thread_blocked, id=4480, stack(0x64290000,0x642e0000)]
              0x64094000 JavaThread "Reference Handler" daemon [_thread_blocked, id=5744, stack(0x64240000,0x64290000)]
            
            Other Threads:
              0x64000c00 VMThread [stack: 0x641f0000,0x64240000] [id=4116]
              0x640ab400 WatcherThread [stack: 0x64470000,0x644c0000] [id=584]
            
            VM state:not at safepoint (normal execution)
            
            VM Mutex/Monitor currently owned by a thread: None
            
            Heap
             PSYoungGen      total 79424K, used 36671K [0x5c810000, 0x639d0000, 0x639d0000)
              eden space 63360K, 32% used [0x5c810000,0x5dc3daa8,0x605f0000)
              from space 16064K, 99% used [0x605f0000,0x615924f0,0x615a0000)
              to   space 27904K, 0% used [0x61e90000,0x61e90000,0x639d0000)
             PSOldGen        total 932096K, used 195081K [0x239d0000, 0x5c810000, 0x5c810000)
              object space 932096K, 20% used [0x239d0000,0x2f8526c8,0x5c810000)
             PSPermGen       total 69632K, used 69519K [0x039d0000, 0x07dd0000, 0x239d0000)
              object space 69632K, 99% used [0x039d0000,0x07db3c78,0x07dd0000)
            
            Dynamic libraries:
            0x00400000 - 0x00424000      C:\Program Files\Java\jdk1.6.0_12\bin\java.exe
            0x7c800000 - 0x7c8c2000      C:\WINDOWS\system32\ntdll.dll
            0x77e40000 - 0x77f42000      C:\WINDOWS\system32\kernel32.dll
            0x7d1e0000 - 0x7d27c000      C:\WINDOWS\system32\ADVAPI32.dll
            
                 ======================           MANY LIBS HERE      ====================== 
            
            0x76b80000 - 0x76cca000      C:\Program Files\Java\jdk1.6.0_12\jre\bin\awt.dll
            0x6d3a0000 - 0x6d3c7000      C:\WINDOWS\system32\WINSPOOL.DRV
            0x76cd0000 - 0x76e09000      C:\WINDOWS\system32\ole32.dll
            0x77090000 - 0x77193000      C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.3790.3959_x-ww_D8713E55\comctl32.dll
            0x77da0000 - 0x77df2000      C:\WINDOWS\system32\SHLWAPI.dll
            
            VM Arguments:
            jvm_args: -Djemail.config.file=C:\jboss-5.0.0.GA\server\default\descriptors\jemail-116.config -Dvelocity.config.file=C:\jboss-5.0.0.GA\server\default\descriptors\velocity-116.properties -Dprogram.name=run.bat -Xms1024m -Xmx1024m -XX:MaxPermSize=512m -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.endorsed.dirs=C:\jboss-5.0.0.GA\lib\endorsed 
            java_command: org.jboss.Main -b 172.16.2.116
            Launcher Type: SUN_STANDARD
            
            Environment Variables:
            JAVA_HOME=C:\Program Files\Java\jdk1.6.0_12
            CLASSPATH=C:\JPW/lib/JavaProcessWatchdog.jar;C:\JPW/lib/log4j-1.2.15.jar;
            PATH=C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\WinSCP\;C:\Program Files\TortoiseSVN\bin;C:\Program Files\Java\jdk1.6.0_12\bin;C:\Sun\SDK\bin
            USERNAME=Administrator
            OS=Windows_NT
            PROCESSOR_IDENTIFIER=x86 Family 6 Model 23 Stepping 7, GenuineIntel
            
            
            
            ---------------  S Y S T E M  ---------------
            
            OS: Windows Server 2003 family Build 3790 Service Pack 2
            
            CPU:total 4 (4 cores per cpu, 1 threads per core) family 6 model 7 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3
            
            Memory: 4k page, physical 2097151k(2097151k free), swap 4194303k(4194303k free)
            
            vm_info: Java HotSpot(TM) Server VM (11.2-b01) for windows-x86 JRE (1.6.0_12-b04), built on Jan 17 2009 09:52:33 by "java_re" with MS VC++ 7.1
            
            time: Thu Jul 09 08:00:40 2009
            elapsed time: 119670 seconds