3 Replies Latest reply: May 24, 2012 1:04 AM by 921316 RSS

    JRockit JVM has crashed with the dump file (at acGetOperand)

    921316
      Hi,

      JRockit JVM has crashed with the dump file below.


      ===== BEGIN DUMP =============================================================
      JRockit dump produced after 2 days, 04:51:44 on Fri Feb 10 12:05:04 2012

      Additional information is available in:
      C:\Program Files\D******\server\bin\jrockit.3628.dump
      C:\Program Files\D******\server\bin\jrockit.3628.mdmp
      If you see this dump, please open a support case with BEA and
      supply as much information as you can on your system setup and
      the program you were running. You can also search for solutions
      to your problem at http://forums.bea.com in
      the forum jrockit.developer.interest.general.

      Error Message: Illegal memory access. [54]
      Exception Rec: EXCEPTION_ACCESS_VIOLATION (00000000c0000005) at 0x000000001000247D - memory at 0xFFFFFFFFDFD35153 could not be read.
      Minidump : Wrote mdmp. Size is 3MB
      SafeDllMode : -1
      Version : BEA JRockit(R) R27.3.1-1-85830-1.5.0_11-20070716-1248-windows-x86_64
      GC Strategy : Mode: throughput. Currently using strategy: singleparpar
      GC Status : OC is not running. Last finished OC was OC#9275.
      : YC is not running. Last finished YC was YC#20.
      OC History : Strategy genparpar was used for OC#9266 to OC#9267.
      : Strategy singleparpar was used for OC#9268 to OC#9269.
      : Strategy genparpar was used for OC#9270 to OC#9271.
      : Strategy singleparpar was used for OC#9272 to OC#9273.
      : Strategy genparpar was used for OC#9274 to OC#9275.
      YC History : Ran 0 YCs before OC#9271.
      : Ran 0 YCs before OC#9272.
      : Ran 0 YCs before OC#9273.
      : Ran 0 YCs before OC#9274.
      : Ran 0 YCs before OC#9275.
      : Ran 0 YCs since last OC.
      Heap : 0x0000000180000000 - 0x0000000480000000
      Compaction : 0x0000000240000000 - 0x0000000270000000
      CompRefs : References are uncompressed 64-bit.
      : Heap is not included in dumps.
      CPU : Intel Pentium III, model A (HT) SSE SSE2 SSE3 SSSE3 EM64T
      Number CPUs : 16
      Tot Phys Mem : 51528912896 (49141 MB)
      OS version : Microsoft Windows "Longhorn" version 6.1 Service Pack 1 (Build 7601) (64-bit)
      Thread System: Windows Threads
      State : JVM is running (Main thread has finished)
      Command Line : -server -Djava.ext.dirs=..\\system\\boot\\ext;..\\..\\jre\\lib\\ext -Djava.endorsed.dirs=..\system\boot\endorsed -Djava.system.class.loader=***..
      -XstrictFP -XgcPrio:throughput -Xms12288m -Xmx12288m -XXdumpSize:small -XnoOpt -Dsun.nio.cs.map=x-windows-iso2022jp/ISO-2022-JP -Xrs -Dsun.java.launcher=SUN_STANDARD ..
      java.home : C:\Program Files\D******\jre
      j.class.path : **..
      j.lib.path : **..
      JAVAOPTIONS: <not set>
      PATH : **..
      C Heap : Good; no memory allocations have failed
      StackOverFlow: 0 StackOverFlowErrors have occured
      OutOfMemory : 0 OutOfMemoryErrors have occured

      Registers (from ThreadContext: 0x000000005CBACCA0 / OS context: 0x000000005CBAD480):
      rax = 0000000000000000 rcx = 0000000000000002
      rdx = 000000005cbadf30 rbx = 000000005cbade70
      rsp = 000000005cbada40 rbp = 0000000082f0ecf0
      rsi = 000000005cbadef0 rdi = ffffffffdfd35153
      r8 = ffffffffdfd35153 r9 = 0000000000000001
      r10 = 0000000000000200 r11 = 0000000070b480f8
      r12 = 000000005cbade00 r13 = 000000005cbadef0
      r14 = 000000005cbade10 r15 = 00000000ffffffff
      cs = 0000000000000033 fs = 0000000000000053
      gs = 000000000000002b
      rip = 000000001000247d flags = 0000000000010246

      Stack:
      (* marks the word pointed to by the stack pointer)
      000000005cbada40: 0000000000000000* 000000005cbade70 000000005cbadef0 ffffffffdfd35153
      000000005cbada60: 00000000102868c8 000000001006e5e8 464646464646465b 0000000000000000
      000000005cbada80: 000000001c787295 000000001c76fe17 00000000ffffffff 000007fefdc48348
      000000005cbadaa0: 000000001c76fe17 0000000064964300 0000000000000002 0000000010002f7d
      000000005cbadac0: 0000000000000000 0000000064964300 000000005cbade70 ffffffffdfd35153
      000000005cbadae0: 000000000000001f 000000001026e263 000007fefdcd2ae0 000000001026e18d

      Code:
      (* marks the word pointed to by the instruction pointer)
      0000000010002418: f88348337402e883 7c8b48c033167404 484024748b484824 28c4834838245c8b
      0000000010002438: 247c8b48078b48c3 8b484024748b4848 c328c4834838245c 4848247c8b48078b
      0000000010002458: 245c8b484024748b b70fc328c4834838 8b4848247c8b4807 38245c8b48402474
      0000000010002478: 07b60fc328c48348* 748b4848247c8b48 4838245c8b484024 e8cb8b48c328c483
      0000000010002498: 8bcf8b48001c04e4 8b480004d0eae8d0 1c049fe8f88b48cb 8b027504f8834800
      00000000100024b8: 8b484024748b48ff 8b48c78b4838245c c328c4834848247c 245c894828ec8348

      Loaded modules:
      (* denotes the module causing the exception)
      0000000000400000-0000000000414fff C:\Program Files\D******\jre\bin\java.exe
      00000000775f0000-0000000077798fff C:\Windows\SYSTEM32\ntdll.dll
      00000000773d0000-00000000774eefff C:\Windows\system32\kernel32.dll
      000007fefd5f0000-000007fefd65afff C:\Windows\system32\KERNELBASE.dll
      000007feff5d0000-000007feff6aafff C:\Windows\system32\ADVAPI32.dll
      000007fefdc40000-000007fefdcdefff C:\Windows\system32\msvcrt.dll
      000007fefdce0000-000007fefdcfefff C:\Windows\SYSTEM32\sechost.dll
      000007fefd910000-000007fefda3cfff C:\Windows\system32\RPCRT4.dll
      0000000010000000-0000000010407fff C:\Program Files\D*****\jre\bin\jrockit\jvm.dll
      000007feec910000-000007feec94afff C:\Windows\system32\WINMM.dll
      00000000774f0000-00000000775e9fff C:\Windows\system32\USER32.dll
      000007feff890000-000007feff8f6fff C:\Windows\system32\GDI32.dll
      000007fefdc30000-000007fefdc3dfff C:\Windows\system32\LPK.dll
      000007fefdae0000-000007fefdba8fff C:\Windows\system32\USP10.dll
      000007fefdee0000-000007fefdf2cfff C:\Windows\system32\WS2_32.dll
      000007feff880000-000007feff887fff C:\Windows\system32\NSI.dll
      000007fefe040000-000007fefe06dfff C:\Windows\system32\IMM32.DLL
      000007fefdf30000-000007fefe038fff C:\Windows\system32\MSCTF.dll
      00000000777c0000-00000000777c6fff C:\Windows\system32\psapi.dll
      000007fefd390000-000007fefd3b4fff C:\Windows\system32\SspiCli.dll
      0000000000150000-0000000000160fff C:\Program Files\D******\jre\bin\verify.dll
      0000000000ea0000-0000000000ecbfff C:\Program Files\D******\jre\bin\java.dll
      000000005ca20000-000000005ca2afff C:\Program Files\D******\jre\bin\hpi.dll
      000000005dbb0000-000000005dbc5fff C:\Program Files\D******\jre\bin\zip.dll
      000000005e800000-000000005e81afff C:\Program Files\D******\jre\bin\net.dll
      000007fefcd70000-000007fefcdc4fff C:\Windows\system32\mswsock.dll
      000007fefc750000-000007fefc756fff C:\Windows\System32\wshtcpip.dll
      000007fefcd60000-000007fefcd66fff C:\Windows\System32\wship6.dll
      000007fefbed0000-000007fefbee4fff C:\Windows\system32\NLAapi.dll
      000007fef1b60000-000007fef1b74fff C:\Windows\system32\napinsp.dll
      000007fefcbf0000-000007fefcc4afff C:\Windows\system32\DNSAPI.dll
      000007fef1b00000-000007fef1b0afff C:\Windows\System32\winrnr.dll
      000007fefb980000-000007fefb9a6fff C:\Windows\system32\IPHLPAPI.DLL
      000007fefb940000-000007fefb94afff C:\Windows\system32\WINNSI.DLL
      000007fef1b10000-000007fef1b17fff C:\Windows\system32\rasadhlp.dll
      000007fefb840000-000007fefb892fff C:\Windows\System32\fwpuclnt.dll
      000007fefcdd0000-000007fefcde6fff C:\Windows\system32\CRYPTSP.dll
      000007fefcad0000-000007fefcb16fff C:\Windows\system32\rsaenh.dll
      000007fefc870000-000007fefc88dfff C:\Windows\system32\USERENV.dll
      000007fefd540000-000007fefd54efff C:\Windows\system32\profapi.dll
      000007fefd510000-000007fefd51efff C:\Windows\system32\CRYPTBASE.dll
      0000000000170000-0000000000175fff C:\Program Files\D******\jre\bin\rmi.dll
      0000000000250000-0000000000263fff C:\Program Files\D******\jre\bin\JdbcOdbc.dll
      000007fef04a0000-000007fef0550fff C:\Windows\system32\ODBC32.dll
      0000000074300000-0000000074337fff C:\Windows\system32\odbcint.dll
      000007fefcf20000-000007fefcf41fff C:\Windows\system32\bcrypt.dll
      000007fefca10000-000007fefca5bfff C:\Windows\system32\bcryptprimitives.dll
      000007fefd660000-000007fefd7c6fff C:\Windows\system32\CRYPT32.dll
      000007fefd5e0000-000007fefd5eefff C:\Windows\system32\MSASN1.dll
      000007fefc680000-000007fefc68bfff C:\Windows\system32\VERSION.dll
      000007feeee80000-000007feeeea7fff C:\Windows\system32\odbccp32.dll
      000007fefd120000-000007fefd12afff C:\Windows\system32\SECUR32.DLL
      000007fefc9d0000-000007fefc9d9fff C:\Windows\system32\credssp.dll
      000007fefcd00000-000007fefcd50fff C:\Windows\system32\msv1_0.DLL
      000007fefd050000-000007fefd063fff C:\Windows\system32\cryptdll.dll
      000007fefcb60000-000007fefcbb6fff C:\Windows\system32\schannel.DLL
      000007fefcf50000-000007fefcf9dfff C:\Windows\system32\ncrypt.dll
      000007fefd3d0000-000007fefd426fff C:\Windows\system32\apphelp.dll
      000007fee5c90000-000007fee5d16fff C:\Program Files\TSOL\DCMS\LogWriter\MsmqJava.dll
      000007feee490000-000007feee4cffff C:\Windows\system32\mqrt.dll
      000007fefe070000-000007fefe272fff C:\Windows\system32\ole32.dll
      000007fefe560000-000007fefe636fff C:\Windows\system32\OLEAUT32.dll
      000007fef1c00000-000007fef1c43fff C:\Windows\system32\mqsec.dll
      000007fefb3c0000-000007fefb3d5fff C:\Windows\system32\NETAPI32.dll
      000007fefca00000-000007fefca0bfff C:\Windows\system32\netutils.dll
      000007fefd070000-000007fefd092fff C:\Windows\system32\srvcli.dll
      000007fefb3a0000-000007fefb3b4fff C:\Windows\system32\wkscli.dll
      000007fefcbc0000-000007fefcbeffff C:\Windows\system32\LOGONCLI.DLL
      000007fefa650000-000007fefa663fff C:\Windows\system32\SAMCLI.DLL
      000007fefbde0000-000007fefbdebfff C:\Windows\system32\DSROLE.DLL
      000007feface0000-000007fefacfcfff C:\Windows\system32\SAMLIB.dll
      000000000ffe0000-000000000ffebfff C:\Program Files\D******\jre\bin\nio.dll
      0000000062e10000-0000000062e12fff C:\Windows\system32\security.dll
      0000000074650000-00000000747a9fff C:\Program Files\D******\jre\bin\dbghelp.dll

      Rollforward Dump Helper:
           method=com/...; [000000001C76FDE0 - 000000001C76FE9F]
           ip=0x000000001C76FE17
           rf=0x000000005CBADEF0
           context=0x0000000082F0ECF0

      Rollforwarded Thread: "Thread-1062049" id=3729458 idx=0x88 tid=19268 lastJavaFrame=0x0000000062F0DA20
      at java/io/WinNTFileSystem.getBooleanAttributes(Ljava/io/File;)I(Native Method)
      at java/io/File.exists(File.java:702)
      at sun/misc/URLClassPath$FileLoader.getResource(URLClassPath.java:893)
      at sun/misc/URLClassPath$FileLoader.findResource(URLClassPath.java:871)
      at sun/misc/URLClassPath.findResource(URLClassPath.java:142)
      at java/net/URLClassLoader$2.run(URLClassLoader.java:362)
      at jrockit/vm/AccessController.doPrivileged(AccessController.java:232)
      at java/net/URLClassLoader.findResource(URLClassLoader.java:359)
      at java/lang/ClassLoader.getResource(ClassLoader.java:977)
      at java/lang/ClassLoader.getSystemResource(ClassLoader.java:1075)
      at java/lang/ClassLoader.getSystemResourceAsStream(ClassLoader.java:1181)
      at org/apache/xerces/parsers/SecuritySupport$6.run()Ljava/lang/Object;(Unknown Source)
      at jrockit/vm/AccessController.doPrivileged(AccessController.java:232)
      at jrockit/vm/AccessController.doPrivileged(AccessController.java:240)
      at org/apache/xerces/parsers/SecuritySupport.getResourceAsStream(Ljava/lang/ClassLoader;Ljava/lang/String;)Ljava/io/InputStream;(Unknown Source)
      at org/apache/xerces/parsers/ObjectFactory.findJarServiceProvider(Ljava/lang/String;)Ljava/lang/Object;(Unknown Source)
      at org/apache/xerces/parsers/ObjectFactory.createObject(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)Ljava/lang/Object;(Unknown Source)
      at org/apache/xerces/parsers/ObjectFactory.createObject(Ljava/lang/String;Ljava/lang/String;)Ljava/lang/Object;(Unknown Source)
      at org/apache/xerces/parsers/SAXParser.<init>(Lorg/apache/xerces/util/SymbolTable;Lorg/apache/xerces/xni/grammars/XMLGrammarPool;)V(Unknown Source)
      at org/apache/xerces/parsers/SAXParser.<init>()V(Unknown Source)
      at org/apache/xerces/jaxp/SAXParserImpl$JAXPSAXParser.<init>(Lorg/apache/xerces/jaxp/SAXParserImpl;)V(Unknown Source)
      at org/apache/xerces/jaxp/SAXParserImpl.<init>(Lorg/apache/xerces/jaxp/SAXParserFactoryImpl;Ljava/util/Hashtable;Z)V(Unknown Source)
      at org/apache/xerces/jaxp/SAXParserFactoryImpl.newSAXParser()Ljavax/xml/parsers/SAXParser;(Unknown Source)
      at com/**
      at com/**
      at com/**
      at java/lang/Thread.run(Thread.java:595)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace

      * 000000001C76FE13:     pop %rcx
           rsib
      000000001C76FE14:     pop %rbp
      000000001C76FE15:     pop %rbx
      000000001C76FE16:     ret
      * 000000001C76FE17:     mov 0x0(%rsp),%rbp
           rsib, [rsp+0x0]b
      000000001C76FE1B:     mov %rsi,%rbx


      Rollforwarded instructions:
      ======= jrockit/vm/Allocator.allocObjectOrArray(JIIZ)Ljava/lang/Object; [000000001C632210 - 000000001C632247]
      000000001C632239:     ret      ; ( 0:[00000000691B98B8]:0x1c6321b3 => )
      ======= jrockit/vm/Locks.fatLockSpin(J)I [000000001C9B4540 - 000000001C9B459B]
      000000001C9B4563:     dec %eax     ; ( 0:rax:0000000000000002 1:0000000000000000 => 0:rax:0000000000000001 )
      000000001C9B4565:     mov %eax,0x10(%rsp)     ; ( 0:rax:0000000000000001 => 0:[00000000634EE890]:0x00000001 )
      ======= java/util/Properties$LineReader.readLine()I [000000001C6941D0 - 000000001C694634]
      000000001C694334:     je 000000001C69435B     ; ( 0:000000001c69435b 1:flags:ZscPo => )
      ======= jrockit/vm/Reflect$IClassBlock.isAssignable(JJ)Z [000000001C6412F0 - 000000001C6413C9]
      000000001C6412FC:     cmp %rdx,%rax     ; ( 0:rax:00000000003dbeb0 1:rdx:00000000003dbeb0 => 0:flags:ZscPo )
      000000001C6412FF:     je 000000001C6413BD     ; ( 0:000000001c6413bd 1:flags:ZscPo => )
      ======= sun/nio/cs/ext/DoubleByteDecoder.decodeArrayLoop(Ljava/nio/ByteBuffer;Ljava/nio/CharBuffer;)Ljava/nio/charset/CoderResult; [000000001C66DC50 - 000000001C66E0E4]
      000000001C66DE55:     mov %rdx,0x38(%rsp)     ; ( 0:rdx:0000000264192ee8 => 0:[000000006591E178]:0x264192ee8 )
      000000001C66DE5A:     jmp 000000001C66DEA5     ; ( 0:000000001c66dea5 => )
      ======= jrockit/vm/Reflect$IClass.getNoCheck(J)Ljava/lang/Class; [000000001C643BB0 - 000000001C643BDD]
      000000001C643BC7:     cmp $0x0037d980,%eax     ; ( 0:rax:000000000037d980 1:000000000037d980 => 0:flags:ZscPo )
      000000001C643BCC:     jne 000000001C643BD0     ; ( 0:000000001c643bd0 1:flags:ZscPo => )


      "(GC Main Thread)" id=4 idx=0x10 tid=44608 lastJavaFrame=0xFFFFFFFFFFFFFFFC

      Stack 0: start=0x000000005CAB0000, end=0x000000005CBB0000, guards=0x000000005CAB4000 (ok), forbidden=0x000000005CAB1000
      Thread Stack Trace:
      at acGetOperand+285()@0x000000001000247D
      at rfExecute+4424()@0x000000001006E5E8
      at rfStop+432()@0x000000001020B5C0
      at rfLimitedFastForward+146()@0x000000001020BAD2
      at vmtSetRollForwardRetryLimit+235()@0x000000001007858B
      at vmtiSignalExitCritical+700()@0x0000000010078B2C
      at vmtSoftSuspend+696()@0x000000001007A008
      at tsRestartJavaExecutionForThread+499()@0x00000000102555F3
      at mmEnterOC+38()@0x000000001015E4D6
      at mmGCMainLoop+192()@0x0000000010040C30
      at tsiInitializeRandomizer+137()@0x000000001014FED9
      at tsCreateThreadObject+3346()@0x0000000010150FF2
      at ptGetSize+78()@0x00000000101DDADE
      at BaseThreadInitThunk+13()@0x00000000773E652D
      at RtlUserThreadStart+33()@0x000000007761C521
      -- Java stack --

      Additional information is available in:
      C:\Program Files\D******\server\bin\jrockit.3628.dump
      C:\Program Files\D******\server\bin\jrockit.3628.mdmp
      If you see this dump, please open a support case with BEA and
      supply as much information as you can on your system setup and
      the program you were running. You can also search for solutions
      to your problem at http://forums.bea.com in
      the forum jrockit.developer.interest.general.
      ===== END DUMP ===============================================================


      I would like to find out if this has been a known issue and if there is
      any workaround available which will solve this problem. As both
      "mmEnterOC" and "mmGCMainLoop" are found in its 'Thread Stack Trace:' my
      guess is that it is GC related problem and wonder if it can be prevented
      by specifying relevant JVM options such as '-Xgc:parallel' when
      starting up the application.

      Thank you for your kind attention.

      Hisashi SOGA

      Edited by: 918313 on 2012/06/18 2:16

      Edited by: 918313 on 2012/06/18 2:22

      Edited by: 918313 on 2012/06/18 17:08
        • 1. Re: JRockit JVM has crashed with the dump file (at acGetOperand)
          Arunbodap-Oracle
          Many bugs were reported for JVM crast at acGetOperand in JRockit R27.x
          But no bugs are reported exactly with the same stack trace.

          Regarding a workaround, the below stack is actually misleading us. It does not give us a clue about the reasons for the crash (whether it is happening due to concurrent GC or parallel GC or Compaction etc.), so we cannot really say what GC settings would avoid it.

          I recommend that we collect another textual dump if the crash occurs again and hope that the stack trace generated the next time will give us better insight into the root cause.

          Another suggestion would be to upgrade to R28.x
          • 2. Re: JRockit JVM has crashed with the dump file (at acGetOperand)
            Arunbodap-Oracle
            Many bugs were reported for JVM crast at acGetOperand in JRockit R27.x
            But no bugs are reported exactly with the same stack trace.

            Regarding a workaround, the below stack is actually misleading us. It does not give us a clue about the reasons for the crash (whether it is happening due to concurrent GC or parallel GC or Compaction etc.), so we cannot really say what GC settings would avoid it.

            I recommend that we collect another textual dump if the crash occurs again and hope that the stack trace generated the next time will give us better insight into the root cause.

            Another suggestion would be to upgrade to R28.x
            • 3. Re: JRockit JVM has crashed with the dump file (at acGetOperand)
              921316
              Thank you for your comment and apologize for this belated response.
              I would appreciate if you can point me to the aforementioned
              "acGetOperand bug reports", which by googling, I had no luck finding any
              of its relevant posts but this one per se as it contains the string
              "acGetOperand".
              We would like to review some of the acGetOperand bug reports which will
              help us determine upgrade to R28.
              Hope you are still following this thread.

              Kind regards,
              Hisashi SOGA