This discussion is archived
1 Reply Latest reply: Sep 16, 2010 7:39 AM by 843829 RSS

JVM Crash in compiled code (1.6.0)

843829 Newbie
Currently Being Moderated
Hello,
in our productive environment we got some crashes of the JVM in the compiled code. This crashes don't happen in the test-environment and are unreproducible and look like a concurrent problem between two independent functions.
The system is a Suse Enterprise 10. The crashes are in:
J  java.lang.Thread.currentThread()Ljava/lang/Thread

J  java.lang.Thread.isAlive()

J  java.lang.Throwable.fillInStackTrace()Ljava/lang/Throwable
We tried jdk1.6.0_18 and jdk1.6.0_11 and we use no native functions (except the functions used by the jdk).

Any ideas what provoked this crash?

bye
Roland Spatzenegger

excerpt from one hs_err log:
#  SIGSEGV (0xb) at pc=0x00002aaaab414c1c, pid=6915, tid=1669724480
#
# Java VM: Java HotSpot(TM) 64-Bit Server VM (11.0-b16 mixed mode linux-amd64)
# Problematic frame:
# J  java.lang.Throwable.fillInStackTrace()Ljava/lang/Throwable;

---------------  T H R E A D  ---------------

Current thread (0x00002aacc6c3ffe0):  JavaThread "Token-75650118-5085" daemon [_thread_in_native_trans, id=1549, stack(0x000000006375f000,0x0000000063860000)]

siginfo:si_signo=SIGSEGV: si_errno=0, si_code=2 (SEGV_ACCERR), si_addr=0x00002aaaaabb8000

Registers:
RAX=0x000000006385e980, RBX=0x00002aac53c86e78, RCX=0x0000000000000ffc, RDX=0x0000000000000008
RSP=0x000000006385e980, RBP=0x000000006385e9d0, RSI=0x00002aacc6c3e578, RDI=0x00002b13d3cf1968
R8 =0x0000000000000ffc, R9 =0x00002aaaaabb7000, R10=0x00002aaaaabb7000, R11=0x0000000000000ffc
R12=0x0000000000000000, R13=0x000000006385e9b0, R14=0x000000006385e980, R15=0x00002aacc6c3ffe0
RIP=0x00002aaaab414c1c, EFL=0x0000000000010206, CSGSFS=0x0000000000000033, ERR=0x0000000000000007
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x000000006385e980)
0x000000006385e980:   00002aac53c86e78 0000000800000001
0x000000006385e990:   00002aacc6c3ff00 000000006385eb78
0x000000006385e9a0:   000000000000000e 00002aaa00000000
0x000000006385e9b0:   0000000000000001 00002aaaae2f40b0
0x000000006385e9c0:   0000000000000000 000000006385e9f0
0x000000006385e9d0:   00002a9768cbee80 00002aaaab4a22c8
0x000000006385e9e0:   00002aac53c86e78 00002aaaafe54448
0x000000006385e9f0:   00002aacc6c3ffe0 00002aacc3794ea2
0x000000006385ea00:   000000006385ea30 00002aaaae2f40b0
0x000000006385ea10:   0000000000000000 000000006385eb48
0x000000006385ea20:   00002aac53c858d8 00002aaaab1cadbb
0x000000006385ea30:   00002aac53c86e78 00002aaaab9f0a18
0x000000006385ea40:   00002aaabe265068 00002aac53c85860
0x000000006385ea50:   00002aaabe265068 00002aac53c858d8
0x000000006385ea60:   0012a5b3ec942001 00002aac53c86c40
0x000000006385ea70:   00002aac53c86c70 00002aaabe3eb4e8
0x000000006385ea80:   00002aaabe3faee0 00002aaabe45fd20
0x000000006385ea90:   0000000000000000 00002aac53c85c90
0x000000006385eaa0:   000000006385ead0 00002b13d380e606
0x000000006385eab0:   00002aacc21f8130 00002b13d3cf80b0
0x000000006385eac0:   0000000000000003 000000006385eca8
0x000000006385ead0:   00002aaabe3eb458 00002aaaac961244
0x000000006385eae0:   00002aac525f35a0 00002aac51fcc8d8
0x000000006385eaf0:   00002aacc6c3e558 0000000000000000
0x000000006385eb00:   000000006385ec80 00002b13d3980dbf
0x000000006385eb10:   00002aaaab1683fb 00002aac1ae6c4f8
0x000000006385eb20:   0000000000001fa0 000000006385ebc0
0x000000006385eb30:   000000006385eba0 00002aaaab16b6c3
0x000000006385eb40:   00002aaaab16b6c3 00002aac525f35a0
0x000000006385eb50:   00002aaabe3eb458 00002aaabe3eb420
0x000000006385eb60:   000000006385eb60 00002aaab173eb4e
0x000000006385eb70:   000000006385ebd8 00002aaab173fcd0 

Instructions: (pc=0x00002aaaab414c1c)
0x00002aaaab414c0c:   81 e1 fc 0f 00 00 49 ba 00 70 bb aa aa 2a 00 00
0x00002aaaab414c1c:   49 89 0c 0a 49 ba 88 4e d0 d3 13 2b 00 00 41 81 

Stack: [0x000000006375f000,0x0000000063860000],  sp=0x000000006385e980,  free space=1022k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
J  java.lang.Throwable.fillInStackTrace()Ljava/lang/Throwable;

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J  java.lang.Throwable.fillInStackTrace()Ljava/lang/Throwable;
J  de.m.e.core.util.StackTraceException.<init>(Ljava/lang/String;)V
J  de.m.e.core.persistence.impl.HibernateSessionFactoryImpl.getOrCreateLocalSession()Lde/m/e/core/persistence/Session;
J  de.m.e.application.process.engine.impl.MyProcessTokenImpl.getUser()Lde/m/e/application/organisation/User;
j  de.m.e.application.process.engine.impl.ActivityHandler.run()V+142
J  java.lang.Thread.run()V
v  ~StubRoutines::call_stub

---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
=>0x00002aacc6c3ffe0 JavaThread "Token-75650118-5085" daemon [_thread_in_native_trans, id=1549, stack(0x000000006375f000,0x0000000063860000)]
  0x000000004011b3d0 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=6931, stack(0x000000004103a000,0x000000004113b000)]
  0x00002aacbfe80cf0 JavaThread "CompilerThread1" daemon [_thread_blocked, id=6930, stack(0x0000000040f39000,0x000000004103a000)]
  0x00002aacbfe7f300 JavaThread "CompilerThread0" daemon [_thread_blocked, id=6929, stack(0x0000000040e38000,0x0000000040f39000)]
  0x00002aacbfe7d980 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=6928, stack(0x0000000040d37000,0x0000000040e38000)]
  0x00002aacbfe5f6d0 JavaThread "Finalizer" daemon [_thread_blocked, id=6927, stack(0x0000000040c36000,0x0000000040d37000)]
  0x00002aacbfe5e010 JavaThread "Reference Handler" daemon [_thread_blocked, id=6926, stack(0x0000000040b35000,0x0000000040c36000)]
  0x00000000401122c0 JavaThread "main" [_thread_blocked, id=6916, stack(0x000000004012b000,0x000000004022c000)]

Other Threads:
  0x00002aacbfe58d20 VMThread [stack: 0x0000000040a34000,0x0000000040b35000] [id=6925]
  0x00002aacbfec6fc0 WatcherThread [stack: 0x000000004143e000,0x000000004153f000] [id=6935]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap
 PSYoungGen      total 2221120K, used 1135520K [0x00002aac13790000, 0x00002aacaa210000, 0x00002aacbe230000)
  eden space 1985344K, 53% used [0x00002aac13790000,0x00002aac53c98738,0x00002aac8ca60000)
  from space 235776K, 34% used [0x00002aac8ca60000,0x00002aac91a3f988,0x00002aac9b0a0000)
  to   space 237504K, 0% used [0x00002aac9ba20000,0x00002aac9ba20000,0x00002aacaa210000)
 PSOldGen        total 1126784K, used 692004K [0x00002aaabe230000, 0x00002aab02e90000, 0x00002aac13790000)
  object space 1126784K, 61% used [0x00002aaabe230000,0x00002aaae85f9328,0x00002aab02e90000)
 PSPermGen       total 119104K, used 90464K [0x00002aaaae230000, 0x00002aaab5680000, 0x00002aaabe230000)
  object space 119104K, 75% used [0x00002aaaae230000,0x00002aaab3a88310,0x00002aaab5680000)

VM Arguments:
jvm_args: -Xmx8g -Xms1g -XX:+UseParallelGC -XX:MaxPermSize=256m -XX:-UseBiasedLocking -DE_VERSION=6.2.0 -Djava.net.preferIPv4Stack=true -Dhibernate.bytecode.provider=cglib
java_command: de.m.e.server.EprotasServer

uname:Linux 2.6.16.60-0.34-smp #1 SMP Fri Jan 16 14:59:01 UTC 2009 x86_64
libc:glibc 2.4 NPTL 2.4 
rlimit: STACK 8192k, CORE infinity, NPROC 266240, NOFILE 32767, AS infinity
load average:0.09 0.16 0.16

vm_info: Java HotSpot(TM) 64-Bit Server VM (11.0-b16) for linux-amd64 JRE (1.6.0_11-b03), built on Nov 10 2008 01:28:14 by "java_re" with gcc 3.2.2 (SuSE Linux)
  • 1. Re: JVM Crash in compiled code (1.6.0)
    843829 Newbie
    Currently Being Moderated
    Hello,
    after some debugging with gdb, I discovered that the three crashes happen in the same instruction (mov %rcx,(%r10,%rcx,1)) and code fragment.
    0x00002aaaabb73410:     and    %al,(%rdx)
    0x00002aaaabb73412:     add    %al,(%rax)
    0x00002aaaabb73414:     add    $0x0,%al
    0x00002aaaabb73416:     add    %al,(%rax)
    0x00002aaaabb73418:     mov    $0x2b71d348f0a0,%r10
    0x00002aaaabb73422:     rex.WB callq  *%r10
    0x00002aaaabb73425:     and    $0xff,%eax
    0x00002aaaabb7342b:     setne  %al
    0x00002aaaabb7342e:     movq   $0x5,0x220(%r15)
    0x00002aaaabb73439:     mov    %r15d,%ecx
    0x00002aaaabb7343c:     shr    $0x4,%ecx
    0x00002aaaabb7343f:     and    $0xffc,%ecx
    0x00002aaaabb73445:     mov    $0x2aaaaabb7000,%r10
    
    ----> 0x00002aaaabb7344f:     mov    %rcx,(%r10,%rcx,1)
    
    0x00002aaaabb73453:     mov    $0x2b71d38f0e88,%r10
    0x00002aaaabb7345d:     cmpl   $0x0,(%r10)
    0x00002aaaabb73464:     jne    0x2aaaabb73478
    0x00002aaaabb7346a:     cmpl   $0x0,0x30(%r15)
    0x00002aaaabb73472:     je     0x2aaaabb7349e
    0x00002aaaabb73478:     mov    %rax,-0x8(%rbp)
    0x00002aaaabb7347c:     mov    %r15,%rdi
    0x00002aaaabb7347f:     mov    %rsp,%r12
    0x00002aaaabb73482:     sub    $0x0,%rsp
    0x00002aaaabb73486:     and    $0xfffffffffffffff0,%rsp
    0x00002aaaabb7348a:     mov    $0x2b71d36c2e30,%r10 
    Is it possible that it's the same bug as described in:
    [http://bugs.sun.com/view_bug.do?bug_id=6811384] [http://bugs.sun.com/view_bug.do?bug_id=6885108]

    The jdk Versions with the problem are: 1.6.0_11 and 1.6.0_16
    (We didn't test it with 1.6.0_18 as previously mentioned, sorry)
    bye
    Roland

    Edited by: Cymric on Sep 16, 2010 7:37 AM