1 Reply Latest reply: Feb 6, 2014 5:48 AM by Sharmela-Oracle RSS

    JVM crashes on GC Worker Thread - Error Message: Illegal memory access

    user10055316

      Hello All

       

      I am running Oracle JRockit(R) R28.2.3-13-149708-1.6.0_31-20120327-1523-windows-ia32, recently we started to see JVM crashes on our Regression environment. Everytime, When I look at the dump file, there are no OutOfMemoryErrors and no StackOverFlowErrors.

      But, whenever it crashes, it is always on GC Worker Thread (thread number do vary). Spent a lot of time investigating, no solutions yet. Hence approaching this expert forum, any help would be much appreciated, Please share your opinion.


      Thanks

      Kannan

       

      Here is the dump file text:

       

      ===== BEGIN DUMP =============================================================
      JRockit dump produced after 0 days, 03:04:31 on Fri Oct 25 20:01:22 2013

          **************************************************************************
          *  If you see this dump, please go to                                    *
          *  http://download.oracle.com/docs/cd/E15289_01/go2troubleshooting.html  *
          *  for troubleshooting information.                                      *
          **************************************************************************

      Additional information is available in:
        D:\Apps\EchoCluster\bin\jrockit.5436.dump
        D:\Apps\EchoCluster\bin\jrockit.5436.mdmp

      Error Message: Illegal memory access. [54]
      Exception Rec: EXCEPTION_ACCESS_VIOLATION (c0000005) at 0x00D24BDF - memory at 0x00000048 could not be read.
      Minidump     : Wrote mdmp. Size is 2362MB
      SafeDllMode  : -1
      Version      : Oracle JRockit(R) R28.2.3-13-149708-1.6.0_31-20120327-1523-windows-ia32
      CPU          : Intel  SSE SSE2 SSE3 SSSE3 SSE4.1 SSE4.2 Intel64
      Number CPUs  : 8
      Tot Phys Mem : 34359271424 (32767 MB)
      OS version   : Microsoft Windows 2008 R2 version 6.1 Service Pack 1 (Build 7601) (64-bit)
      Hypervisor   : Unknown (but probably running on a hypervisor)
      Thread System: Windows Threads
      Java locking : Lazy unlocking enabled (class banning) (transfer banning)
      State        : JVM is running (Main thread has finished)
      Command Line : <modified: I have removed the lengthy command line parameters>
      Repository   : C:\Users\SVCUKL~1\AppData\Local\Temp\2013_10_25_16_56_55_5436
      java.home    : C:\Program Files (x86)\Java\JRockit\jre
      j.class.path : ..\\lib\wrappertest.jar;..\\lib\wrapper.jar;..\\lib\echo.jar;..\\lib\coherence.jar;..\\lib\coherence-jpa.jar;..\\lib\eclipselink.jar;..\\lib\eclipselink-jpa-modelgen_2.4.2.v20130514-5956486.jar;..\\lib\javax.persistence_1.0.0.jar;..\\lib\javax.persistence_2.0.5.v201212031355.jar;..\\lib\ojdbc6.jar;..\\lib\commons-lang3-3.1.jar;..\\lib\ojdbc5.jar;..\\config\cluster\pof;..\;..\\lib\log4j;..\\lib\log4j-1.2.17.jar;..\\lib\apache-log4j-extras-1.1.jar;..\\lib\monitoring-framework-1.10.0.jar;..\\lib\slf4j-api-1.7.2.jar;..\\lib\slf4j-simple-1.7.2.jar;..\\lib\tibrvj.jar;..\\lib\tibrvnative.jar;..\\lib\tibrv\;..\\config\cluster\server\log4j;..\\lib\coherence-rest.jar;..\\lib\jackson-all-1.8.1.jar;..\\lib\jersey-core-1.8.jar;..\\lib\jersey-json-1.8.jar;..\\lib\jersey-server-1.8.jar;..\\config\cluster\pof\coherence-rest-config.xml;..\\lib\coherence-loadbalancer.jar;..\\lib\commonj.jar;..\\lib\jmscommon.jar;..\\lib\jndi.jar;..\\lib\jta.jar;..\\lib\junit-4.10.jar;..\\lib\wrapperdemo.jar;..\\lib\vtd-xml.jar
      j.lib.path   : lib;..\\lib\tibrv\;..\\lib\
      JAVA_HOME    : C:\Program Files (x86)\Java\JRockit
      _JAVA_OPTIONS: <not set>
      PATH         : C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Bank\ODCReal;C:\Program Files\Python26\;C:\Program Files\Reuters\Rendez-vous\bin;C:\Windows\system32\config\systemprofile\AppData\Roaming\Python\Scripts;C:\Program Files (x86)\Bank\ODCReal\;C:\Program Files\SysinternalsSuite;C:\Program Files (x86)\Windows Resource Kits\Tools;C:\Windows\sysint;C:\ORACLE11G32\product\11.2.0\client_1\bin;C:\JAVA_637\BIN;C:\JAVA_637\jre\bin\client;C:\Program Files (x86)\Python26;
      StackOverFlow: 0 StackOverFlowErrors have occured
      OutOfMemory  : 0 OutOfMemoryErrors have occured
      C Heap       : Good; no memory allocations have failed
      GC Strategy  : Mode: pausetime, with strategy: genconcon (basic strategy: genconcon)
      GC Status    : OC is not running. Last finished OC was OC#1.
                   : YC is currently running. This is YC#111.
      YC Promotion : This YC has been able to promote all found objects so far
      YC History   : Ran 52 YCs before OC#1.
                   : Started 59 YCs since last OC.
      Heap Holes   : 1 hole(s), at 0x009F0000 - 0x7FFF0000
      Heap         : 0x009B0000 - 0xFFFB0000  (Size: 2048 MB)
      Compaction   : (no compaction area)
      Allocation   : TLA-min: 2048, TLA-preferred: 16384 TLA-waste limit: 2048
      NurseryList  : 0x7FFF0000 - 0xE4A95BD0
      KeepArea     : 0x83BEFFC0 - 0xE4A95BD0
      KA Markers   : [ 0x827EFFD8,  0x83BEFFC0 , 0xE4A95BD0 ]
      Forbidden A  : (none)
      Previous KA  : 0x827EFFD8 - 0x83BEFFC0
      Previous FA  : (none)
      CompRefs     : References are 32-bit.

      Registers (from ThreadContext: 0x0BA2FA08:
         eax = 80d813d8    ecx = 00000003    edx = 00000000    ebx = 0ba2fddc
         esp = 0ba2fcf0    ebp = 7fff0000    esi = 0ba2fd6c    edi = 02df0000
          es = 0000002b     cs = 00000023     ss = 0000002b     ds = 0000002b
          fs = 00000053     gs = 0000002b
         eip = 00d24bdf eflags = 00010286

      Loaded modules:
      (* denotes the module where the exception occured)
      00400000-00434fff  C:\Program Files (x86)\Java\JRockit\bin\java.exe
      77000000-7717ffff  C:\Windows\SysWOW64\ntdll.dll
      75750000-7585ffff  C:\Windows\syswow64\kernel32.dll
      76980000-769c6fff  C:\Windows\syswow64\KERNELBASE.dll
      754d0000-7556ffff  C:\Windows\syswow64\ADVAPI32.dll
      74d60000-74e0bfff  C:\Windows\syswow64\msvcrt.dll
      750b0000-750c8fff  C:\Windows\SysWOW64\sechost.dll
      76810000-768fffff  C:\Windows\syswow64\RPCRT4.dll
      74b60000-74bbffff  C:\Windows\syswow64\SspiCli.dll
      74b50000-74b5bfff  C:\Windows\syswow64\CRYPTBASE.dll
      7c340000-7c395fff  C:\Program Files (x86)\Java\JRockit\jre\bin\msvcr71.dll
      00bf0000-00ef2fff *C:\Program Files (x86)\Java\JRockit\jre\bin\jrockit\jvm.dll
      75860000-75894fff  C:\Windows\syswow64\WS2_32.dll
      758a0000-758a5fff  C:\Windows\syswow64\NSI.dll
      74230000-74261fff  C:\Windows\system32\WINMM.dll
      76ad0000-76bcffff  C:\Windows\syswow64\USER32.dll
      75020000-750affff  C:\Windows\syswow64\GDI32.dll
      76fd0000-76fd9fff  C:\Windows\syswow64\LPK.dll
      74ee0000-74f7cfff  C:\Windows\syswow64\USP10.dll
      74e80000-74edffff  C:\Windows\system32\IMM32.DLL
      753f0000-754bbfff  C:\Windows\syswow64\MSCTF.dll
      73750000-7379bfff  C:\Windows\system32\apphelp.dll
      10000000-1003cfff  C:\Program Files (x86)\Java\JRockit\jre\bin\jrosal.dll
      001a0000-001d2fff  C:\Program Files (x86)\Java\JRockit\jre\bin\jrutil.DLL
      74300000-7433bfff  C:\Windows\system32\pdh.dll
      743e0000-74400fff  C:\Windows\system32\ntmarta.dll
      74e30000-74e74fff  C:\Windows\syswow64\WLDAP32.dll
      73460000-7346afff  C:\Windows\System32\perfproc.dll
      738e0000-738e8fff  C:\Windows\System32\perfos.dll
      00540000-0056afff  C:\Program Files (x86)\Java\JRockit\jre\bin\jfr.dll
      6d870000-6d87bfff  C:\Program Files (x86)\Java\JRockit\jre\bin\verify.dll
      6d3d0000-6d3eefff  C:\Program Files (x86)\Java\JRockit\jre\bin\java.dll
      6d330000-6d337fff  C:\Program Files (x86)\Java\JRockit\jre\bin\hpi.dll
      75910000-7655afff  C:\Windows\system32\SHELL32.DLL
      75600000-75656fff  C:\Windows\syswow64\SHLWAPI.dll
      76560000-766bbfff  C:\Windows\syswow64\ole32.dll
      70c60000-70dfdfff  C:\Windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_6.0.7601.17514_none_41e6975e2bd6f2b2\comctl32.dll
      756c0000-7574efff  C:\Windows\syswow64\OLEAUT32.dll
      74bc0000-74d5cfff  C:\Windows\syswow64\SETUPAPI.dll
      75660000-75686fff  C:\Windows\syswow64\CFGMGR32.dll
      74e10000-74e21fff  C:\Windows\syswow64\DEVOBJ.dll
      74f90000-75012fff  C:\Windows\syswow64\CLBCatQ.DLL
      70a80000-70b74fff  C:\Windows\system32\propsys.dll
      6d8b0000-6d8befff  C:\Program Files (x86)\Java\jrockit\jre\bin\zip.dll
      6d600000-6d608fff  C:\Program Files (x86)\Java\jrockit\jre\bin\management.dll
      6d6b0000-6d6c2fff  C:\Program Files (x86)\Java\jrockit\jre\bin\net.dll
      74a20000-74a5bfff  C:\Windows\system32\mswsock.dll
      74820000-74824fff  C:\Windows\System32\wshtcpip.dll
      73960000-73965fff  C:\Windows\System32\wship6.dll
      74a70000-74a7ffff  C:\Windows\system32\NLAapi.dll
      74a60000-74a6ffff  C:\Windows\system32\napinsp.dll
      74850000-74893fff  C:\Windows\system32\DNSAPI.dll
      74840000-74847fff  C:\Windows\System32\winrnr.dll
      742e0000-742fbfff  C:\Windows\system32\IPHLPAPI.DLL
      742c0000-742c6fff  C:\Windows\system32\WINNSI.DLL
      74830000-74835fff  C:\Windows\system32\rasadhlp.dll
      733e0000-73417fff  C:\Windows\System32\fwpuclnt.dll
      0c900000-0c929fff  C:\Program Files (x86)\Java\jrockit\jre\bin\jmapi.dll
      0dc90000-0dcb1fff  C:\Program Files (x86)\Java\jrockit\jre\bin\orii.dll
      746c0000-746d5fff  C:\Windows\system32\CRYPTSP.dll
      74680000-746bafff  C:\Windows\system32\rsaenh.dll
      74290000-742a6fff  C:\Windows\system32\USERENV.dll
      74280000-7428afff  C:\Windows\system32\profapi.dll
      754c0000-754c4fff  C:\Windows\system32\psapi.DLL
      0e290000-0e2d3fff  D:\APPS\EchoCluster\lib\wrapper.dll
      74200000-7420cfff  C:\Windows\system32\dhcpcsvc6.DLL
      74210000-74221fff  C:\Windows\system32\dhcpcsvc.DLL
      6d6d0000-6d6d8fff  C:\Program Files (x86)\Java\jrockit\jre\bin\nio.dll
      71370000-71484fff  C:\Program Files (x86)\Java\JRockit\jre\bin\dbghelp.dll

      Stack:
      (* marks the word pointed to by the stack pointer)
      0ba2fcf0: 0ba2fd6c* 11558ea0  00846000  00d24e44  0ba2fd5c  0ba2fddc 
      0ba2fd08: 11558ea0  00000000  00d29cb0  0ba2fd6c  0ba2fddc  00000000 
      0ba2fd20: 00d29e5f  11558ea0  0ba2fd6c  1211f558  11558ea0  00000000 
      0ba2fd38: 1211f39c  00000000  851b0000  84fb0000  00000000  00ed2114

      Code:
      (* marks the word pointed to by the instruction pointer)
      00d24bac: e1830487  a8e8d31f  5f097401  333c5689  c3595dc0  212c152b 
      00d24bc4: 146800ed  5200ed21  093ee855  c483fffe  0fc0850c  00008984 
      00d24bdc: 8b108b00* ed33484a  03447a83  c8030474  788b1ceb  cfaf0f08 
      00d24bf4: 10014c8d  0e76ff85  8304528b  7c01447a  0001bd05  4e3b0000


      Scan Dump Helper:
      Processing the remset iterator at 0x0BA2FD6C.
      Processing cards from start card: 0x0B1D3000 to end card: 0x0B1D4000.
      The current dirty region is from heap address: 0x84FB0000 to: 0x84FB0200.
      The last cleaned card is 0x84FAFE00 and lastRefArray is 0x00000000.

      Processing references from object iterator at 0x0BA2FDDC.
      The reference that caused the crash is unknown (broken object).

      ObjRef.sourceObj = 0x0128CC22 -- ObjRef.targetObj = 0x00000000
      ObjRef.ref       = 0x0BA2FD50 -- ObjRef.kind      = fullref

       

      sourceObj: Seems to be bogus or broken.
        The object address points outside the heap.

      targetObj: Is NULL.
      Trying to locate who is pointing to the sourceObj.
        There seems to be no one on the heap referring to the sourceObj.

       

      Last optimized methods:
        #705 java/io/ObjectOutputStream.writeUTF(Ljava/lang/String;)V  9056.687-9056.701 0x13491B80-0x13491CFE 13.99 ms 192KB 
        #706 java/util/HashMap$ValueIterator.next()Ljava/lang/Object;  9374.233-9374.238 0x13491D00-0x13491E1E 4.78 ms 192KB 
        #707 java/io/ObjectStreamClass$FieldReflector.getObjFieldValues(Ljava/lang/Object;[Ljava/lang/Object;)V  9405.515-9405.524 0x13491E20-0x13492001 9.69 ms 192KB 
        #708 com/tangosol/coherence/component/net/management/model/localModel/ServiceModel.getPartitionsUnbalanced()I  9637.624-9637.773 0x13492020-0x13493E06 148.76 ms 2688KB 
        #709 com/tangosol/coherence/component/application/console/Coherence.getSafeCluster()Lcom/tangosol/coherence/component/util/SafeCluster;  9792.764-9792.896 0x13493E20-0x1349585C 131.83 ms 1984KB 
        #710 com/tangosol/run/xml/XmlHelper.trim(Ljava/lang/String;)Ljava/lang/String;  9933.140-9933.169 0x13495860-0x13495CAD 28.63 ms 384KB 
        #711 com/tangosol/util/SafeLinkedList.toArray([Ljava/lang/Object;)[Ljava/lang/Object;  10116.984-10117.172 0x13495CC0-0x13496D5B 188.04 ms 1280KB 
        #712 com/tangosol/coherence/component/net/TcpRing.getSelector()Ljava/nio/channels/Selector;  10525.859-10525.860 0x13496D60-0x13496D6E 1.29 ms 256KB 
        #713 java/util/WeakHashMap.getEntry(Ljava/lang/Object;)Ljava/util/WeakHashMap$Entry;  10880.654-10880.668 0x13496D80-0x134971B0 13.21 ms 448KB 
        #714 java/util/WeakHashMap.eq(Ljava/lang/Object;Ljava/lang/Object;)Z  10880.668-10880.668 0x134971C0-0x134971E1 0.38 ms 128KB 
      *#715 com/tangosol/coherence/component/util/daemon/queueProcessor/service/grid/ClusterService$MemberHeartbeat.write(Lcom/tangosol/io/WriteBuffer$BufferOutput;)V  10974.482-10974.483 0x13497200-0x13497224 0.42 ms 128KB 
        #700 java/util/HashMap$ValueIterator.<init>(Ljava/util/HashMap;Ljava/util/HashMap$1;)V  7609.065-7609.068 0x13491400-0x134914A8 2.84 ms 192KB 
        #701 com/tangosol/coherence/component/net/MemberSet.getBitSet(I)I  7660.780-7660.783 0x134914C0-0x134914EA 2.35 ms 128KB 
        #702 com/tangosol/util/SegmentedHashMap.getEntryInternal(Ljava/lang/Object;)Lcom/tangosol/util/SegmentedHashMap$Entry;  8745.015-8745.053 0x13491500-0x134918AE 38.58 ms 384KB 
        #703 com/tangosol/coherence/component/util/daemon/queueProcessor/Service.isRunning()Z  8780.124-8780.125 0x134918C0-0x134918E9 0.83 ms 64KB 
        #704 java/util/TreeMap.get(Ljava/lang/Object;)Ljava/lang/Object;  9028.359-9028.403 0x13491900-0x13491B7F 44.08 ms 320KB

       

      Thread:
      "(GC Worker Thread 4)" id=? idx=0x1c tid=5192 lastJavaFrame=0xFFFFFFFC
      Stack 0: start=0x0B9F0000, end=0x0BA30000, guards=0x0B9F3000 (ok), forbidden=0x0B9F1000

      Thread Stack Trace:
          at remSetIterPrepareNewRegion+143(remsetiter.c:354+2)@0x00D24BDF
          at remSetIterGetNext+228(remsetiter.c:504+11)@0x00D24E44
          at ycProcessRemSetIter+15(ycgc.c:231+15)@0x00D29CB0
          at ycProcessRemSet+286(ycgc.c:306+19)@0x00D29E5F
          at ycWorkerProcessRoots+556(ycgc.c:387+20)@0x00D2A16D
          at mmGCWorkerThread+166(gcthreads.c:828+16)@0x00D13D57
          at thread_stub+254(lifecycle.c:808+6)@0x00DC1A0F
          at BaseThreadInitThunk+17()@0x7576336A
          at RtlInitializeExceptionChain+98()@0x77039F72
          at RtlInitializeExceptionChain+53()@0x77039F45
          -- Java stack --

       

      Memory usage report:
      Could not mark thread stacks regions properly.
      Total mapped                  2499820KB           (reserved=73928KB)
      -              Java heap      2097152KB           (reserved=0KB)
      -              GC tables       139956KB         
      -          Thread stacks            0KB           (#threads=0)
      -          Compiled code         5568KB           (used=5003KB)
      -               Internal          776KB         
      -                     OS        41024KB         
      -                  Other       176432KB         
      -        Java class data        37888KB           (malloced=37827KB #39179 in 6926 classes)
                                                        Not tracing sites.

      - Native memory tracking         1024KB           (malloced=347KB #8)
                                                        Not tracing sites.


      Set the env variable TRACE_ALLOC_SITES=1 or use the print_memusage switch
      trace_alloc_sites=true to enable alloc site tracing.


      Warning! Discrepancy detected while collecting memory usage. Numbers can be wrong.

          **************************************************************************
          *  If you see this dump, please go to                                    *
          *  http://download.oracle.com/docs/cd/E15289_01/go2troubleshooting.html  *
          *  for troubleshooting information.                                      *
          **************************************************************************

      ===== END DUMP ===============================================================