0 Replies Latest reply: Feb 1, 2012 4:10 AM by Thomas Falkenberg RSS

    JRockit R28.2 crashes under load

    Thomas Falkenberg
      Hi,

      we experience crashes of our Jenkins after upgrading to JRockit R28.2.0-79-14677 from R28.1.

      Now:
      java version "1.6.0_29"
      Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
      Oracle JRockit(R) (build R28.2.0-79-146777-1.6.0_29-20111005-1807-linux-x86_64, compiled mode)

      Before:
      java version "1.6.0_20"
      Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
      Oracle JRockit(R) (build R28.1.0-123-138454-1.6.0_20-20101014-1350-linux-x86_64, compiled mode)


      Maybe this dump file already helps, I'll try to get a .core file too. I noticed there has been a similar issue with R28.0.1 (Oracle support [ID 1295504.1] / Bug 10418780) but it should have been fixed in R28.1.1. From the dump-file we have no stack overflow and different signal info (Signal info : si_signo=11, si_code=1 si_addr=0x2aaaf318a910) from the one in the bug description.


      ===== BEGIN DUMP =============================================================
      JRockit dump produced after 0 days, 17:14:01 on Fri Jan 27 09:47:08 2012

      **************************************************************************
      * If you see this dump, please go to *
      * http://download.oracle.com/docs/cd/E15289_01/go2troubleshooting.html *
      * for troubleshooting information. *
      **************************************************************************

      Additional information is available in:
      /opt/ci/jenkins/jenkins_slaves/slave1/jrockit.13327.dump
      No snapshot file (core dump) will be created because core dumps have been
      disabled. To enable core dumping, try "ulimit -c unlimited"
      before starting JRockit again.

      Error Message: Illegal memory access. [54]
      Signal info : si_signo=11, si_code=1 si_addr=0x2aaaf318a910
      Version : Oracle JRockit(R) R28.2.0-79-146777-1.6.0_29-20111005-1807-linux-x86_64
      CPU : Intel Westmere (HT) SSE SSE2 SSE3 SSSE3 SSE4.1 SSE4.2 Core Intel64
      Number CPUs : 24
      Tot Phys Mem : 50635268096 (48289 MB)
      OS version : CentOS release 5.5 (Final)
      Linux version 2.6.18-194.el5 (mockbuild@builder10.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-48)) #1 SMP Fri Apr 2 14:58:14 EDT 2010 (x86_64)
      Thread System: Linux NPTL
      LibC release : 2.5-stable
      Java locking : Lazy unlocking enabled (class banning) (transfer banning)
      State : JVM is running
      Command Line : -Xms256m -Xmx1280m -Djava.security.egd=file:/dev/./urandom -Duser.timezone=Europe/Berlin -Dsun.java.command=slave.jar -cp .:commons-codec.jar -jnlpCredentials xxx:xxx-jnlpUrl http://xxx/computer/slave1/slave-agent.jnlp -Dsun.java.launcher=SUN_STANDARD slave.jar -cp .:commons-codec.jar -jnlpCredentials xxx:xxx-jnlpUrl http://xxx/computer/slave1/slave-agent.jnlp
      Repository :
      java.home : /opt/java/jrockit1.6.0_29/jre
      j.class.path : slave.jar
      j.lib.path : /usr/java/packages/lib/amd64:/opt/java/jrockit1.6.0_29/jre/lib/amd64/jrockit:/opt/java/jrockit1.6.0_29/jre/lib/amd64:/opt/java/jrockit1.6.0_29/jre/../lib/amd64
      JAVA_HOME : /opt/java/jrockit1.6.0_29
      JAVAOPTIONS: <not set>
      LD_LIBRARY_PATH: /opt/java/jrockit1.6.0_29/jre/lib/amd64/jrockit:/opt/java/jrockit1.6.0_29/jre/lib/amd64:/opt/java/jrockit1.6.0_29/jre/../lib/amd64
      LD_ASSUME_KERNEL: <not set>
      LD_PRELOAD : <not set>
      StackOverFlow: 0 StackOverFlowErrors have occured
      OutOfMemory : 0 OutOfMemoryErrors have occured
      C Heap : Good; no memory allocations have failed
      GC Strategy : Mode: throughput, with strategy: genparpar (basic strategy: genparpar)
      GC Status : OC is not running. Last finished OC was OC#3.
      : YC is not running. Last finished YC was YC#23.
      YC Promotion : Last YC successfully promoted all objects
      YC History : Ran 9 YCs before OC#1.
      : Ran 8 YCs before OC#2.
      : Ran 6 YCs before OC#3.
      : Ran 0 YCs since last OC.
      Heap : 0xb0000000 - 0xc0000000 (Size: 256 MB)
      Compaction : (no compaction area)
      Allocation : TLA-min: 2048, TLA-preferred: 65536 TLA-waste limit: 2048
      NurseryList : 0xb0023e28 - 0xb81d3db0
      KeepArea : 0xb6159000 - 0xb81d3db0
      KA Markers : [ 0xb4198908,  0xb6159000 , 0xb81d3db0 ]
      Forbidden A : (none)
      Previous KA : 0xb3fdb580 - 0xb5f8dd70
      Previous FA : (none)
      CompRefs : References are compressed, with heap base 0x0 and shift 0.

      Registers (from ThreadContext: 0x42315a30:
      rax = 000000001e1e1b20 rcx = 00000000b9ff6810
      rdx = 000000001ebaeed0 rbx = 00000000b9ff6810
      rsp = 0000000042315e78 rbp = 00000000b8794d88
      rsi = 0000000042315ea8 rdi = 000000001e1e1d40
      r8 = 00000000b9f3c028 r9 = 0000000042315e88
      r10 = 0000000042315e88 r11 = 00002aaaaddef400
      r12 = 0000000042315e80 r13 = 00000000b0024608
      r14 = 000000001e1e1b20 r15 = 000000001e1e1b20
      cs = 0000000000000033 fs = 0000001400000000
      gs = 0014000000000000
      rip = 00002aaaf318a910 flags = 0000000000000202

      Loaded modules:
      (* denotes the module where the exception occured)
      0000000000400000-00000000004128a3 /opt/java/jrockit1.6.0_29/bin/java
      0000003a13000000-0000003a13001f93 /lib64/libdl.so.2
      0000003a13400000-0000003a13415a6f /lib64/libpthread.so.0
      0000003a12800000-0000003a1294d067 /lib64/libc.so.6
      0000003a12400000-0000003a1241bb7f /lib64/ld-linux-x86-64.so.2
      00002b462d7ed000-00002b462daf3c2b /opt/java/jrockit1.6.0_29/jre/lib/amd64/jrockit/libjvm.so
      00002b462ddce000-00002b462ddecd63 /opt/java/jrockit1.6.0_29/jre/lib/amd64/libjrosal.so
      00002b462dff0000-00002b462dffc553 /opt/java/jrockit1.6.0_29/jre/lib/amd64/libjrutil.so
      0000003a12c00000-0000003a12c8122f /lib64/libm.so.6
      0000003a17000000-0000003a17006e8b /lib64/librt.so.1
      00002aaaaabd6000-00002aaaaabdf64b /lib64/libnss_files.so.2
      00002aaaaade1000-00002aaaaaded003 /opt/java/jrockit1.6.0_29/jre/lib/amd64/libjfr.so
      00002aaaedbd3000-00002aaaedbdf337 /opt/java/jrockit1.6.0_29/jre/lib/amd64/libverify.so
      00002aaaedce2000-00002aaaedd0a15b /opt/java/jrockit1.6.0_29/jre/lib/amd64/libjava.so
      0000003a1ca00000-0000003a1ca140c7 /lib64/libnsl.so.1
      00002aaaede91000-00002aaaede975bf /opt/java/jrockit1.6.0_29/jre/lib/amd64/native_threads/libhpi.so
      00002aaaf162b000-00002aaaf1638bc3 /opt/java/jrockit1.6.0_29/jre/lib/amd64/libzip.so
      00002aaaf1cbf000-00002aaaf1cd1c87 /opt/java/jrockit1.6.0_29/jre/lib/amd64/libnet.so
      00002aaaf1dff000-00002aaaf1e02d73 /lib64/libnss_dns.so.2
      0000003a1ea00000-0000003a1ea10823 /lib64/libresolv.so.2
      00002aaaf339a000-00002aaaf33a097b /opt/java/jrockit1.6.0_29/jre/lib/amd64/libnio.so
      00002aaaf34a2000-00002aaaf34a7f63 /opt/java/jrockit1.6.0_29/jre/lib/amd64/liborii.so
      00002aaaf38a9000-00002aaaf38bde33 /tmp/hudson-remoting2651934831509873941/com/sun/jna/linux-amd64/libjnidispatch.so

      Stack:
      (* marks the word pointed to by the stack pointer)
      0000000042315e78: 00002aaaaddef405* 0000000042315e60 00002aaaf8004f70 00002aaaaddef400
      0000000042315e98: 000000001e1e1e98 0000000042315e80 00000000b9f3c028 000000001e1e1b20
      0000000042315eb8: 00000000b8794d88 00000000b9ff6810 00002aaaaddef34d 00000000b9ff6810
      0000000042315ed8: 00002aaaaddef305 00000000b9ff6815 00000000b8794d88 0000000000000030
      0000000042315ef8: 00002aaaaddef2d3 000000000000002f 00002aaaaddee9a7 00000000000000c8
      0000000042315f18: 00002aaaaddee970 00000000000000c8 00002aaaadb98b15 00000000b87755d8

      Code:
      (* marks the word pointed to by the instruction pointer)
      00002aaaf318a8b0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
      00002aaaf318a8d0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
      00002aaaf318a8f0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
      00002aaaf318a910: 0000000000000000* 0000000000000000 0000000000000000 0000000000000000
      00002aaaf318a930: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
      00002aaaf318a950: 0000000000000000 0000000000000000 0000000000000000 0000000000000000


      Last optimized methods:
      *#1 jrockit/vm/Allocator.allocObjectOrArray(JJIZ)Ljava/lang/Object; 6252.364-6252.369 0x2aaaade23000-0x2aaaade231e0 4.57 ms 256KB

      Thread:
      "Finalizer" id=7 idx=0x64 tid=13353 lastJavaFrame=0x42315e88
      Stack 0: start=0x422d4000, end=0x42318000, guards=0x422d9000 (ok), forbidden=0x422d7000

      Thread Stack Trace:
      at <unknown>(???.c)@0x2aaaf318a910
      -- Java stack --
      at com/sun/jna/Native.free(J)V(Native Method)
      at com/sun/jna/Memory.free(Memory.java:686)
      at com/sun/jna/Memory.dispose(Memory.java:144)
      at com/sun/jna/Memory.finalize(Memory.java:139)
      at jrockit/memory/Finalizer.doFinalize(Finalizer.java:29)
      at jrockit/memory/Finalizer.access$300(Finalizer.java:12)
      at jrockit/memory/Finalizer$4.run(Finalizer.java:186)
      at java/lang/Thread.run(Thread.java:662)
      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)
      -- end of trace

      Memory usage report:
      Total mapped 2734748KB (reserved=2283508KB)
      - Java heap 1310720KB (reserved=1048576KB)
      - GC tables 43852KB
      - Thread stacks 19492KB (#threads=35)
      - Compiled code 1048576KB (used=2934KB)
      - Internal 1416KB
      - OS 132764KB
      - Other 146952KB
      - Classblocks 1536KB (malloced=1459KB #4227)
      Not tracing sites.

      - Java class data 28416KB (malloced=28046KB #19675 in 4227 classes)
      Not tracing sites.

      - Native memory tracking 1024KB (malloced=90KB #10)
      Not tracing sites.


      Set the env variable TRACE_ALLOC_SITES=1 or use the print_memusage switch
      trace_alloc_sites=true to enable alloc site tracing.

      **************************************************************************
      * If you see this dump, please go to *
      * http://download.oracle.com/docs/cd/E15289_01/go2troubleshooting.html *
      * for troubleshooting information. *
      **************************************************************************

      ===== END DUMP ===============================================================


      Kind regards,
      Thomas