1 Reply Latest reply: Sep 16, 2010 9:39 AM by 843829 RSS

    JVM Crash in compiled code (1.6.0)

    843829
      Hello,
      in our productive environment we got some crashes of the JVM in the compiled code. This crashes don't happen in the test-environment and are unreproducible and look like a concurrent problem between two independent functions.
      The system is a Suse Enterprise 10. The crashes are in:
      J  java.lang.Thread.currentThread()Ljava/lang/Thread
      
      J  java.lang.Thread.isAlive()
      
      J  java.lang.Throwable.fillInStackTrace()Ljava/lang/Throwable
      We tried jdk1.6.0_18 and jdk1.6.0_11 and we use no native functions (except the functions used by the jdk).

      Any ideas what provoked this crash?

      bye
      Roland Spatzenegger

      excerpt from one hs_err log:
      #  SIGSEGV (0xb) at pc=0x00002aaaab414c1c, pid=6915, tid=1669724480
      #
      # Java VM: Java HotSpot(TM) 64-Bit Server VM (11.0-b16 mixed mode linux-amd64)
      # Problematic frame:
      # J  java.lang.Throwable.fillInStackTrace()Ljava/lang/Throwable;
      
      ---------------  T H R E A D  ---------------
      
      Current thread (0x00002aacc6c3ffe0):  JavaThread "Token-75650118-5085" daemon [_thread_in_native_trans, id=1549, stack(0x000000006375f000,0x0000000063860000)]
      
      siginfo:si_signo=SIGSEGV: si_errno=0, si_code=2 (SEGV_ACCERR), si_addr=0x00002aaaaabb8000
      
      Registers:
      RAX=0x000000006385e980, RBX=0x00002aac53c86e78, RCX=0x0000000000000ffc, RDX=0x0000000000000008
      RSP=0x000000006385e980, RBP=0x000000006385e9d0, RSI=0x00002aacc6c3e578, RDI=0x00002b13d3cf1968
      R8 =0x0000000000000ffc, R9 =0x00002aaaaabb7000, R10=0x00002aaaaabb7000, R11=0x0000000000000ffc
      R12=0x0000000000000000, R13=0x000000006385e9b0, R14=0x000000006385e980, R15=0x00002aacc6c3ffe0
      RIP=0x00002aaaab414c1c, EFL=0x0000000000010206, CSGSFS=0x0000000000000033, ERR=0x0000000000000007
        TRAPNO=0x000000000000000e
      
      Top of Stack: (sp=0x000000006385e980)
      0x000000006385e980:   00002aac53c86e78 0000000800000001
      0x000000006385e990:   00002aacc6c3ff00 000000006385eb78
      0x000000006385e9a0:   000000000000000e 00002aaa00000000
      0x000000006385e9b0:   0000000000000001 00002aaaae2f40b0
      0x000000006385e9c0:   0000000000000000 000000006385e9f0
      0x000000006385e9d0:   00002a9768cbee80 00002aaaab4a22c8
      0x000000006385e9e0:   00002aac53c86e78 00002aaaafe54448
      0x000000006385e9f0:   00002aacc6c3ffe0 00002aacc3794ea2
      0x000000006385ea00:   000000006385ea30 00002aaaae2f40b0
      0x000000006385ea10:   0000000000000000 000000006385eb48
      0x000000006385ea20:   00002aac53c858d8 00002aaaab1cadbb
      0x000000006385ea30:   00002aac53c86e78 00002aaaab9f0a18
      0x000000006385ea40:   00002aaabe265068 00002aac53c85860
      0x000000006385ea50:   00002aaabe265068 00002aac53c858d8
      0x000000006385ea60:   0012a5b3ec942001 00002aac53c86c40
      0x000000006385ea70:   00002aac53c86c70 00002aaabe3eb4e8
      0x000000006385ea80:   00002aaabe3faee0 00002aaabe45fd20
      0x000000006385ea90:   0000000000000000 00002aac53c85c90
      0x000000006385eaa0:   000000006385ead0 00002b13d380e606
      0x000000006385eab0:   00002aacc21f8130 00002b13d3cf80b0
      0x000000006385eac0:   0000000000000003 000000006385eca8
      0x000000006385ead0:   00002aaabe3eb458 00002aaaac961244
      0x000000006385eae0:   00002aac525f35a0 00002aac51fcc8d8
      0x000000006385eaf0:   00002aacc6c3e558 0000000000000000
      0x000000006385eb00:   000000006385ec80 00002b13d3980dbf
      0x000000006385eb10:   00002aaaab1683fb 00002aac1ae6c4f8
      0x000000006385eb20:   0000000000001fa0 000000006385ebc0
      0x000000006385eb30:   000000006385eba0 00002aaaab16b6c3
      0x000000006385eb40:   00002aaaab16b6c3 00002aac525f35a0
      0x000000006385eb50:   00002aaabe3eb458 00002aaabe3eb420
      0x000000006385eb60:   000000006385eb60 00002aaab173eb4e
      0x000000006385eb70:   000000006385ebd8 00002aaab173fcd0 
      
      Instructions: (pc=0x00002aaaab414c1c)
      0x00002aaaab414c0c:   81 e1 fc 0f 00 00 49 ba 00 70 bb aa aa 2a 00 00
      0x00002aaaab414c1c:   49 89 0c 0a 49 ba 88 4e d0 d3 13 2b 00 00 41 81 
      
      Stack: [0x000000006375f000,0x0000000063860000],  sp=0x000000006385e980,  free space=1022k
      Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
      J  java.lang.Throwable.fillInStackTrace()Ljava/lang/Throwable;
      
      Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
      J  java.lang.Throwable.fillInStackTrace()Ljava/lang/Throwable;
      J  de.m.e.core.util.StackTraceException.<init>(Ljava/lang/String;)V
      J  de.m.e.core.persistence.impl.HibernateSessionFactoryImpl.getOrCreateLocalSession()Lde/m/e/core/persistence/Session;
      J  de.m.e.application.process.engine.impl.MyProcessTokenImpl.getUser()Lde/m/e/application/organisation/User;
      j  de.m.e.application.process.engine.impl.ActivityHandler.run()V+142
      J  java.lang.Thread.run()V
      v  ~StubRoutines::call_stub
      
      ---------------  P R O C E S S  ---------------
      
      Java Threads: ( => current thread )
      =>0x00002aacc6c3ffe0 JavaThread "Token-75650118-5085" daemon [_thread_in_native_trans, id=1549, stack(0x000000006375f000,0x0000000063860000)]
        0x000000004011b3d0 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=6931, stack(0x000000004103a000,0x000000004113b000)]
        0x00002aacbfe80cf0 JavaThread "CompilerThread1" daemon [_thread_blocked, id=6930, stack(0x0000000040f39000,0x000000004103a000)]
        0x00002aacbfe7f300 JavaThread "CompilerThread0" daemon [_thread_blocked, id=6929, stack(0x0000000040e38000,0x0000000040f39000)]
        0x00002aacbfe7d980 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=6928, stack(0x0000000040d37000,0x0000000040e38000)]
        0x00002aacbfe5f6d0 JavaThread "Finalizer" daemon [_thread_blocked, id=6927, stack(0x0000000040c36000,0x0000000040d37000)]
        0x00002aacbfe5e010 JavaThread "Reference Handler" daemon [_thread_blocked, id=6926, stack(0x0000000040b35000,0x0000000040c36000)]
        0x00000000401122c0 JavaThread "main" [_thread_blocked, id=6916, stack(0x000000004012b000,0x000000004022c000)]
      
      Other Threads:
        0x00002aacbfe58d20 VMThread [stack: 0x0000000040a34000,0x0000000040b35000] [id=6925]
        0x00002aacbfec6fc0 WatcherThread [stack: 0x000000004143e000,0x000000004153f000] [id=6935]
      
      VM state:not at safepoint (normal execution)
      
      VM Mutex/Monitor currently owned by a thread: None
      
      Heap
       PSYoungGen      total 2221120K, used 1135520K [0x00002aac13790000, 0x00002aacaa210000, 0x00002aacbe230000)
        eden space 1985344K, 53% used [0x00002aac13790000,0x00002aac53c98738,0x00002aac8ca60000)
        from space 235776K, 34% used [0x00002aac8ca60000,0x00002aac91a3f988,0x00002aac9b0a0000)
        to   space 237504K, 0% used [0x00002aac9ba20000,0x00002aac9ba20000,0x00002aacaa210000)
       PSOldGen        total 1126784K, used 692004K [0x00002aaabe230000, 0x00002aab02e90000, 0x00002aac13790000)
        object space 1126784K, 61% used [0x00002aaabe230000,0x00002aaae85f9328,0x00002aab02e90000)
       PSPermGen       total 119104K, used 90464K [0x00002aaaae230000, 0x00002aaab5680000, 0x00002aaabe230000)
        object space 119104K, 75% used [0x00002aaaae230000,0x00002aaab3a88310,0x00002aaab5680000)
      
      VM Arguments:
      jvm_args: -Xmx8g -Xms1g -XX:+UseParallelGC -XX:MaxPermSize=256m -XX:-UseBiasedLocking -DE_VERSION=6.2.0 -Djava.net.preferIPv4Stack=true -Dhibernate.bytecode.provider=cglib
      java_command: de.m.e.server.EprotasServer
      
      uname:Linux 2.6.16.60-0.34-smp #1 SMP Fri Jan 16 14:59:01 UTC 2009 x86_64
      libc:glibc 2.4 NPTL 2.4 
      rlimit: STACK 8192k, CORE infinity, NPROC 266240, NOFILE 32767, AS infinity
      load average:0.09 0.16 0.16
      
      vm_info: Java HotSpot(TM) 64-Bit Server VM (11.0-b16) for linux-amd64 JRE (1.6.0_11-b03), built on Nov 10 2008 01:28:14 by "java_re" with gcc 3.2.2 (SuSE Linux)
        • 1. Re: JVM Crash in compiled code (1.6.0)
          843829
          Hello,
          after some debugging with gdb, I discovered that the three crashes happen in the same instruction (mov %rcx,(%r10,%rcx,1)) and code fragment.
          0x00002aaaabb73410:     and    %al,(%rdx)
          0x00002aaaabb73412:     add    %al,(%rax)
          0x00002aaaabb73414:     add    $0x0,%al
          0x00002aaaabb73416:     add    %al,(%rax)
          0x00002aaaabb73418:     mov    $0x2b71d348f0a0,%r10
          0x00002aaaabb73422:     rex.WB callq  *%r10
          0x00002aaaabb73425:     and    $0xff,%eax
          0x00002aaaabb7342b:     setne  %al
          0x00002aaaabb7342e:     movq   $0x5,0x220(%r15)
          0x00002aaaabb73439:     mov    %r15d,%ecx
          0x00002aaaabb7343c:     shr    $0x4,%ecx
          0x00002aaaabb7343f:     and    $0xffc,%ecx
          0x00002aaaabb73445:     mov    $0x2aaaaabb7000,%r10
          
          ----> 0x00002aaaabb7344f:     mov    %rcx,(%r10,%rcx,1)
          
          0x00002aaaabb73453:     mov    $0x2b71d38f0e88,%r10
          0x00002aaaabb7345d:     cmpl   $0x0,(%r10)
          0x00002aaaabb73464:     jne    0x2aaaabb73478
          0x00002aaaabb7346a:     cmpl   $0x0,0x30(%r15)
          0x00002aaaabb73472:     je     0x2aaaabb7349e
          0x00002aaaabb73478:     mov    %rax,-0x8(%rbp)
          0x00002aaaabb7347c:     mov    %r15,%rdi
          0x00002aaaabb7347f:     mov    %rsp,%r12
          0x00002aaaabb73482:     sub    $0x0,%rsp
          0x00002aaaabb73486:     and    $0xfffffffffffffff0,%rsp
          0x00002aaaabb7348a:     mov    $0x2b71d36c2e30,%r10 
          Is it possible that it's the same bug as described in:
          [http://bugs.sun.com/view_bug.do?bug_id=6811384] [http://bugs.sun.com/view_bug.do?bug_id=6885108]

          The jdk Versions with the problem are: 1.6.0_11 and 1.6.0_16
          (We didn't test it with 1.6.0_18 as previously mentioned, sorry)
          bye
          Roland

          Edited by: Cymric on Sep 16, 2010 7:37 AM