4 Replies Latest reply: Jul 1, 2009 6:45 PM by 843829 RSS

    JVM crashes in CompilerThread CompileTask

    843829
      We got quite constant JVM crashes running some automated tests since we upgraded to 1.5.0_17. Note, that it seems like the JIT compiler is active even though the -Djava.compile=NONE flag is set. The crashes are always in the compile task when compiling the method org.apache.derby.impl.store.raw.data.StoredPage.logRow from the Derby project (recent stable version).

      The crashes seem to occur a little less, when the debug-options are not set.

      This has been tested with the sun-java5 package distributed with Debian Lenny and the same version (JDK) directly installed from the .bin-archive provided by Sun.

      The following is a shortened hs_err_<pid>.log. The SIGSEGV, problematic frame and the method being compiled by the CompileTask are always the same!

      The following thread has some similarities: http://forums.sun.com/thread.jspa?threadID=5280098 however, the referenced bugs are marked as fix released.

      Any help on how to track this down or fix this is greatly appreciated!

      Thanks in advance
      Kevin Göser
      #
      # An unexpected error has been detected by HotSpot Virtual Machine:
      #
      #  SIGSEGV (0xb) at pc=0x00007f9c94f68a40, pid=17857, tid=1087859024
      #
      # Java VM: Java HotSpot(TM) 64-Bit Server VM (1.5.0_17-b04 mixed mode)
      # Problematic frame:
      # V  [libjvm.so+0x286a40]
      #
      
      ---------------  T H R E A D  ---------------
      
      Current thread (0x00007f9c7c021c90):  JavaThread "CompilerThread1" daemon [_thread_in_native, id=17868]
      
      siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x0000000000000008
      
      Registers:
      RAX=0x0000000000000001, RBX=0x00007f9c7cf23c60, RCX=0x00007f9c7c021c90, RDX=0x00007f9c7cf23c70
      RSP=0x0000000040d73e58, RBP=0x0000000040d73ea0, RSI=0x0000000000000010, RDI=0x0000000000000000
      R8 =0x00007f9c7ce3f890, R9 =0x000000000000000b, R10=0x00007f9c7ca3d530, R11=0x00000000404d92d0
      R12=0x00000000404d92d0, R13=0x0000000040d74540, R14=0x0000000040d74f00, R15=0x00007f9c7cf23940
      RIP=0x00007f9c94f68a40, EFL=0x0000000000010293, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
        TRAPNO=0x000000000000000e
      
      Top of Stack: (sp=0x0000000040d73e58)
      0x0000000040d73e58:   00007f9c952526af 0000000040d73ea0
      0x0000000040d73e68:   00007f9c7cf23940 ffffffff40d73ea0
      0x0000000040d73e78:   00007f9c7cf23940 0000000000000027
      ...
      
      Instructions: (pc=0x00007f9c94f68a40)
      0x00007f9c94f68a30:   55 48 c1 ff 03 48 89 e5 89 f8 c9 c3 66 66 66 90
      0x00007f9c94f68a40:   48 8b 47 08 55 48 89 e5 c9 c3 66 66 90 66 66 90 
      
      Stack: [0x0000000040c76000,0x0000000040d77000),  sp=0x0000000040d73e58,  free space=1015k
      Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
      V  [libjvm.so+0x286a40]
      V  [libjvm.so+0x570c8c]
      .....
      
      Current CompileTask:
      opto: 13% !   org.apache.derby.impl.store.raw.data.StoredPage.logRow(IZI[Ljava/lang/Object;Lorg/apache/derby/iapi/services/io/FormatableBitSet;Lorg/apache/derby/iapi/services/io/DynamicByteArrayOutputStream;IBIII)I @ 511 (1279 bytes)
      
      
      ---------------  P R O C E S S  ---------------
      
      Java Threads: ( => current thread )
      ....
      =>0x00007f9c7c021c90 JavaThread "CompilerThread1" daemon [_thread_in_native, id=17868]
      ....
      
      Other Threads:
        0x00000000405188d0 VMThread [id=17860]
        0x00007f9c7c0253c0 WatcherThread [id=17870]
      
      VM state:not at safepoint (normal execution)
      
      VM Mutex/Monitor currently owned by a thread: None
      
      Heap
       def new generation   total 7616K, used 2279K [0x00007f9c835c0000, 0x00007f9c83e00000, 0x00007f9c851c0000)
        eden space 6784K,  21% used [0x00007f9c835c0000, 0x00007f9c83729f68, 0x00007f9c83c60000)
        from space 832K, 100% used [0x00007f9c83d30000, 0x00007f9c83e00000, 0x00007f9c83e00000)
        to   space 832K,   0% used [0x00007f9c83c60000, 0x00007f9c83c60000, 0x00007f9c83d30000)
       tenured generation   total 16776K, used 10180K [0x00007f9c851c0000, 0x00007f9c86222000, 0x00007f9c889c0000)
         the space 16776K,  60% used [0x00007f9c851c0000, 0x00007f9c85bb1060, 0x00007f9c85bb1200, 0x00007f9c86222000)
       compacting perm gen  total 27008K, used 26908K [0x00007f9c889c0000, 0x00007f9c8a420000, 0x00007f9c8ddc0000)
         the space 27008K,  99% used [0x00007f9c889c0000, 0x00007f9c8a4071a0, 0x00007f9c8a407200, 0x00007f9c8a420000)
      No shared spaces configured.
      
      ....
      
      VM Arguments:
      jvm_args: -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8000 ... (truncated)
      
      ---------------  S Y S T E M  ---------------
      
      OS:5.0
      
      uname:Linux 2.6.26-1-amd64 #1 SMP Mon Dec 15 17:25:36 UTC 2008 x86_64
      libc:glibc 2.7 NPTL 2.7 
      rlimit: STACK 8192k, CORE 0k, NPROC 14336, NOFILE 1024, AS infinity
      load average:3.60 3.75 3.11
      
      CPU:total 2 em64t
      
      Memory: 4k page, physical 1803480k(1175480k free), swap 497972k(248992k free)
      
      vm_info: Java HotSpot(TM) 64-Bit Server VM (1.5.0_17-b04) for linux-amd64, built on Nov 10 2008 01:04:27 by java_re with gcc 3.2.2 (SuSE Linux)
      Edited by: kgoeser on Feb 12, 2009 1:54 AM
        • 1. Re: JVM crashes in CompilerThread CompileTask
          843829
          We have many of these too. I believe there is a defect in the JIT compiler causing this. It is not easy to make a simple test condition to demonstrate it.

          What we have done is to exclude certain classes from JIT compilation - for example:

          -XX:CompileCommand=exclude,java/util/Arrays.mergeSort

          However, eventually the same error crops up on a different class.
          • 2. Re: JVM crashes in CompilerThread CompileTask
            843829
            We also are infected by the same issue...

            #
            # An unexpected error has been detected by HotSpot Virtual Machine:
            #
            # SIGSEGV (0xb) at pc=0x00002b06d0bcc712, pid=22067, tid=1101850944
            #
            # Java VM: Java HotSpot(TM) 64-Bit Server VM (1.5.0_17-b04 mixed mode)

            Still no solution yet.. but just wondering if it has to do anything with the memory configuration (GC)

            Logs shows below 99% for space..
            PSYoungGen total 307840K, used 145482K [0x00002b070a3b0000, 0x00002b071f900000, 0x00002b071f900000)
            eden space 266176K, 39% used [0x00002b070a3b0000,0x00002b07109164a8,0x00002b071a7a0000)
            from space 41664K, 99% used [0x00002b071a7a0000,0x00002b071d04c380,0x00002b071d050000)
            to space 41664K, 0% used [0x00002b071d050000,0x00002b071d050000,0x00002b071f900000)

            Let me know if you find anything...
            • 3. Re: JVM crashes in CompilerThread CompileTask
              843829
              I found that when I moved from JDK 1.5.0_16 to JDK1.5.0_17 that I would occasionally get a JVM crash in the ComplieTask during the deployment of our applicatioin EAR in JBoss.

              I found that this occured if I configured "-Xdebug -Xrunjdwp:server=y,suspend=n,transport=dt_socket,address=7777".
              If I left out these parameters I would not get a crash.
              • 4. Re: JVM crashes in CompilerThread CompileTask
                843829
                So why don't you report it after all?