2 Replies Latest reply: Feb 17, 2012 6:22 PM by 263312 RSS

    repeated jrockit crash

    263312
      It doesn't happen too frequently and unfortunately I have no small reproducible test case, but I've experienced this crash a few times with the same signature.

      Is it a known bug? Any thoughts on how to proceed?

      I'll post the crash dumps as replies
        • 1. Re: repeated jrockit crash
          263312
          ===== BEGIN DUMP =============================================================
          JRockit dump produced after 0 days, 00:00:16 on Wed Jan 18 09:44:53 2012

          **************************************************************************
          * If you see this dump, please go to *
          * http://download.oracle.com/docs/cd/E15289_01/go2troubleshooting.html *
          * for troubleshooting information. *
          **************************************************************************

          Additional information is available in:
          /export1/cruise/jenkins/workspace/terracotta-session_trunk_tests_tomcat-7.0.16_JROCKIT_1.6_3.7.x/trunk/system-tests/target/temp/SynchWriteMultiThreadsTest/sandbox/server_1/jrockit.9896.dump
          No snapshot file (core dump) will be created because core dumps have been
          disabled. To enable core dumping, try "ulimit -c unlimited"
          before starting JRockit again.

          Error Message: Illegal memory access. [54]
          Signal info : si_signo=11, si_code=1 si_addr=0xfffffff8
          Version : Oracle JRockit(R) R28.1.4-7-144370-1.6.0_26-20110617-2130-linux-ia32
          CPU : AMD Opteron QC/Phenom SSE SSE2 SSE3 SSE4A AMD64
          Number CPUs : 8
          Tot Phys Mem : 6442627072 (6144 MB)
          OS version : SUSE Linux Enterprise Server 10 (x86_64)
          VERSION = 10
          PATCHLEVEL = 4
          Linux version 2.6.16.60-0.85.1-xen (geeko@buildhost) (gcc version 4.1.2 20070115 (SUSE Linux)) #1 SMP Thu Mar 17 11:45:06 UTC 2011 (x86_64)
          Hypervisor : Xen v3.4
          Thread System: Linux NPTL
          LibC release : 2.4-stable
          Java locking : Lazy unlocking enabled (class banning) (transfer banning)
          State : JVM is running
          Command Line : -XX:+HeapDumpOnOutOfMemoryError -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=63629 -Dcom.tc.session.debug.sessions=true -Drmi.registry.port=17295 -verbose:gc -Xverboselog:/export1/cruise/jenkins/workspace/terracotta-session_trunk_tests_tomcat-7.0.16_JROCKIT_1.6_3.7.x/trunk/system-tests/target/temp/SynchWriteMultiThreadsTest/sandbox/server_1-gc.log -Djvmroute=server_1 -Xms128m -Xmx192m -Djava.security.egd=file:/dev/./urandom -DNODE=server_1 -Dcatalina.home=/export1/cruise/.tc/appservers/tomcat-7.0.16 -Dcatalina.base=/export1/cruise/jenkins/workspace/terracotta-session_trunk_tests_tomcat-7.0.16_JROCKIT_1.6_3.7.x/trunk/system-tests/target/temp/SynchWriteMultiThreadsTest/sandbox/server_1 -Djava.io.tmpdir=/export1/cruise/jenkins/workspace/terracotta-session_trunk_tests_tomcat-7.0.16_JROCKIT_1.6_3.7.x/trunk/system-tests/target/temp/SynchWriteMultiThreadsTest/sandbox/server_1/temp -Dsun.java.launcher=SUN_STANDARD com.tc.lcp.CargoLinkedChildProcess org.apache.catalina.startup.Bootstrap 55239 /export1/cruise/jenkins/workspace/terracotta-session_trunk_tests_tomcat-7.0.16_JROCKIT_1.6_3.7.x/trunk/system-tests/target/temp/SynchWriteMultiThreadsTest/sandbox/server_1 start
          Repository : /export1/cruise/jenkins/workspace/terracotta-session_trunk_tests_tomcat-7.0.16_JROCKIT_1.6_3.7.x/trunk/system-tests/target/temp/SynchWriteMultiThreadsTest/sandbox/server_1/temp/2012_01_18_09_44_39_9896
          java.home : /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre
          j.class.path : /export1/cruise/.tc/appservers/tomcat-7.0.16/bin/tomcat-juli.jar:/export1/cruise/.tc/appservers/tomcat-7.0.16/bin/bootstrap.jar:/export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/lib/tools.jar:/export1/cruise/.m2/repository/org/terracotta/linked-child-process/1.0.6/linked-child-process-1.0.6.jar
          j.lib.path : /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386/jrockit:/export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386:/export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/../lib/i386:/shares/monkeyshare/setup/jdk/linux/hotspot1.6.0_30/jre/lib/i386/server:/shares/monkeyshare/setup/jdk/linux/hotspot1.6.0_30/jre/lib/i386:/shares/monkeyshare/setup/jdk/linux/hotspot1.6.0_30/jre/../lib/i386
          JAVA_HOME : /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre
          JAVAOPTIONS: <not set>
          LD_LIBRARY_PATH: /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386/jrockit:/export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386:/export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/../lib/i386:/shares/monkeyshare/setup/jdk/linux/hotspot1.6.0_30/jre/lib/i386/server:/shares/monkeyshare/setup/jdk/linux/hotspot1.6.0_30/jre/lib/i386:/shares/monkeyshare/setup/jdk/linux/hotspot1.6.0_30/jre/../lib/i386
          LD_ASSUME_KERNEL: <not set>
          LD_PRELOAD : <not set>
          StackOverFlow: 0 StackOverFlowErrors have occured
          OutOfMemory : 0 OutOfMemoryErrors have occured
          C Heap : Good; no memory allocations have failed
          GC Strategy : Mode: throughput, with strategy: genparpar (basic strategy: genparpar)
          GC Status : OC is not running. Last finished OC was OC#1.
          : YC is not running. Last finished YC was YC#9.
          YC Promotion : Last YC successfully promoted all objects
          YC History : Ran 0 YCs before OC#1.
          : Ran 9 YCs since last OC.
          Heap : 0xe9ebb000 - 0xf1ebb000 (Size: 128 MB)
          Compaction : (no compaction area)
          Allocation : TLA-min: 2048, TLA-preferred: 24576 TLA-waste limit: 2048
          NurseryList : 0xe9ec8958 - 0xecf8cef8
          KeepArea : 0xeb698720 - 0xec716a50
          KA Markers : [ 0xeb698720,  0xec716a50 , 0xecf8cef8 ]
          Forbidden A : (none)
          Previous KA : 0xec716a50 - 0xecf8cef8
          Previous FA : (none)
          CompRefs : References are 32-bit.

          Registers (from ThreadContext: 0xe663a7ac:
          eax = ffffffff ecx = e6618dc4 edx = 00000000 ebx = e663ab3c
          esp = e663aaa0 ebp = e663aad8 esi = fffffff8 edi = e663abe8
          es = 0000002b cs = 00000023 ss = 0000002b ds = 0000002b
          fs = 00000003 gs = 00000063
          eip = f74e100d eflags = 00200282

          Loaded modules:
          (* denotes the module where the exception occured)
          08048000-08057193 /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/bin/java
          ffffe000-ffffe7c7 /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/bin/java
          f7761000-f776304b /lib/libdl.so.2
          f774a000-f775cec7 /lib/libpthread.so.0
          f7608000-f774242d /lib/libc.so.6
          f7770000-f778b2d3 /lib/ld-linux.so.2
          f72ce000-f7590317 */export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386/jrockit/libjvm.so
          f72b1000-f72cb4eb /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386/libosal.so
          f72a5000-f72af5cf /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386/libutil.so
          f727e000-f72a2f37 /lib/libm.so.6
          f7274000-f727b0dc /lib/librt.so.1
          f6887000-f6893747 /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386/libjfr.so
          e9760000-e976a9bb /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386/libverify.so
          e973b000-e975df07 /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386/libjava.so
          e971b000-e972d20b /lib/libnsl.so.1
          f7767000-f776ce50 /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386/native_threads/libhpi.so
          e93c2000-e93d06e4 /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386/libzip.so
          e9734000-e9739a06 /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386/libmanagement.so
          e90cb000-e90de150 /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386/libnet.so
          e8d1c000-e8d24f93 /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386/libjmapi.so
          e8cb3000-e8cb8013 /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386/liborii.so
          e7ef2000-e7ef84e4 /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386/libnio.so

          Stack:
          (* marks the word pointed to by the stack pointer)
          e663aaa0: e663aac4* e6c66a30 0000000c f7632cfe e6c66af8 00000000
          e663aab8: 00000000 ffffffff ffffffff e6618dc4 04000009 ffffffff
          e663aad0: e663ab3c e663ab3c e663aae8 f74e10f5 e663ab3c 00000000
          e663aae8: e663ab28 f74e1507 e6c63cd8 f74e0df0 e663ab3c e663ab3c

          Code:
          (* marks the word pointed to by the instruction pointer)
          f74e0fdc: 1c4389ec 89f0458b 458b2043 0c4383f4 24438901 5b30c483
          f74e0ff4: 90c35d5e 097407a8 012843c7 8b000000 83c68901 7389f8e6
          f74e100c: a8068b14* 74c18901 fee08305 7983088b 40740344 8d48418b
          f74e1024: 43890604 90afeb18 0026748d 85e0558b 8b4174d2 4389e045


          Last optimized methods:
          No methods optimized.

          Thread:
          ""http-bio-14955"-exec-15" id=115 idx=0x1d8 tid=10067 lastJavaFrame=0xe663adbc
          Stack 0: start=0xe661a000, end=0xe663c000, guards=0xe661f000 (ok), forbidden=0xe661d000

          Thread Stack Trace:
          at findNext+253(refiter.c:409)@0xf74e100d
          at findNextToReturn+100(refiter.c:302)@0xf74e10f5
          at refIterFillFromFrame+342(refiter.c:123)@0xf74e1507
          at trProcessLocksForThread+73(roots.c:620)@0xf74ec4ea
          at get_all_locks+95(javalock.c:1265)@0xf73ea810
          at javaLockConvertLazyToThin+133(javalock.c:1760)@0xf73ea9f6
          at RJNI_jrockit_vm_Locks_checkAndTransferLazyLocked+486(javalock.c:2032)@0xf73eba27
          -- Java stack --
          at jrockit/vm/Locks.checkAndTransferLazyLocked(Ljava/lang/Object;)I(Native Method)
          at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:971)
          at jrockit/vm/Locks.monitorEnter(Locks.java:2179)
          at EDU/oswego/cs/dl/util/concurrent/ConcurrentReaderHashMap.getTableForReading()[LEDU/oswego/cs/dl/util/concurrent/ConcurrentReaderHashMap$Entry;(Unknown Source)
              at EDU/oswego/cs/dl/util/concurrent/ConcurrentReaderHashMap.get(Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
              at com/tc/object/PortabilityImpl.isPortableClass(PortabilityImpl.java:122)
              at com/tc/object/ClientObjectManagerImpl.isPortableClass(ClientObjectManagerImpl.java:1225)
              at com/tc/object/TCClassImpl.createPortableFields(TCClassImpl.java:341)
              at com/tc/object/TCClassImpl.<init>(TCClassImpl.java:119)
              at com/tc/object/TCClassFactoryImpl.createTCClass(TCClassFactoryImpl.java:95)
              at com/tc/object/TCClassFactoryImpl.getOrCreate(TCClassFactoryImpl.java:72)
              at com/tc/object/ClientObjectManagerImpl.lookup(ClientObjectManagerImpl.java:567)
              at com/tc/object/ClientObjectManagerImpl.lookupObject(ClientObjectManagerImpl.java:463)
              at com/tc/object/ClientObjectManagerImpl.lookupObject(ClientObjectManagerImpl.java:451)
              at com/tc/object/bytecode/ManagerImpl.lookupObject(ManagerImpl.java:484)
              at com/tc/object/bytecode/ManagerUtil.lookupObject(ManagerUtil.java:389)
              at com/terracotta/toolkit/collections/ConcurrentDistributedMapDso.lookupAndFaultIn(ConcurrentDistributedMapDso.java:708)
              at com/terracotta/toolkit/collections/ConcurrentDistributedMapDso.lookupAndFaultIn(ConcurrentDistributedMapDso.java:702)
              at com/terracotta/toolkit/collections/ConcurrentDistributedMapDso.get(ConcurrentDistributedMapDso.java:198)
              at com/terracotta/toolkit/collections/ConcurrentDistributedMapDsoArray.get(ConcurrentDistributedMapDsoArray.java:249)
              at org/terracotta/collections/ConcurrentDistributedMap.get(ConcurrentDistributedMap.java:220)
              at com/terracotta/session/SessionDataStore.find(SessionDataStore.java:145)
              at com/terracotta/session/TerracottaSessionManager.getSessionIfExists(TerracottaSessionManager.java:573)
              at com/terracotta/session/SessionRequest.getTerracottaSession(SessionRequest.java:150)
              at com/terracotta/session/SessionRequest.getSession(SessionRequest.java:109)
              at org/apache/catalina/connector/SessionRequest55.getSession(SessionRequest55.java:57)
              at org/apache/catalina/connector/SessionRequest55.getSessionInternal(SessionRequest55.java:105)
              at org/apache/catalina/authenticator/AuthenticatorBase.invoke(AuthenticatorBase.java:417)
              at org/terracotta/modules/tomcat/tomcat_7_0/SessionValve70.tcInvoke(SessionValve70.java:97)
              at org/terracotta/modules/tomcat/tomcat_7_0/SessionValve70.invoke(SessionValve70.java:82)
              at org/terracotta/session/ModernTomcatSessionValve.invoke(ModernTomcatSessionValve.java:66)
              at org/apache/catalina/core/StandardHostValve.invoke(StandardHostValve.java:164)
              at org/apache/catalina/valves/ErrorReportValve.invoke(ErrorReportValve.java:100)
              at org/apache/catalina/valves/AccessLogValve.invoke(AccessLogValve.java:563)
              at org/apache/catalina/core/StandardEngineValve.invoke(StandardEngineValve.java:118)
              at org/apache/catalina/connector/CoyoteAdapter.service(CoyoteAdapter.java:403)
              at org/apache/coyote/http11/Http11Processor.process(Http11Processor.java:301)
              at org/apache/coyote/http11/Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:162)
              at org/apache/coyote/http11/Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:140)
              at org/apache/tomcat/util/net/JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:309)
              at java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
              at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
              at java/lang/Thread.run(Thread.java:662)
              at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
              -- end of trace

          Memory usage report:
          Total mapped                   322484KB           (reserved=69932KB)
          -              Java heap       196608KB           (reserved=65536KB)
          -              GC tables         6588KB         
          -          Thread stacks        24204KB           (#threads=114)
          -          Compiled code         4480KB           (used=4366KB)
          -               Internal          968KB         
          -                     OS        25628KB         
          -                  Other        37640KB         
          -        Java class data        25344KB           (malloced=25087KB #35708 in 6424 classes)
                                                            Not tracing sites.

          - Native memory tracking         1024KB           (malloced=99KB #8)
                                                            Not tracing sites.


          Set the env variable TRACE_ALLOC_SITES=1 or use the print_memusage switch
          trace_alloc_sites=true to enable alloc site tracing.

              **************************************************************************
              *  If you see this dump, please go to                                    *
              *  http://download.oracle.com/docs/cd/E15289_01/go2troubleshooting.html  *
              *  for troubleshooting information.                                      *
              **************************************************************************

          ===== END DUMP ===============================================================                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
          • 2. Re: repeated jrockit crash
            263312
            ===== BEGIN DUMP =============================================================
            JRockit dump produced after 0 days, 00:00:23 on Sat Dec 31 11:02:53 2011

            **************************************************************************
            * If you see this dump, please go to *
            * http://download.oracle.com/docs/cd/E15289_01/go2troubleshooting.html *
            * for troubleshooting information. *
            **************************************************************************

            Additional information is available in:
            /export1/cruise/jenkins/workspace/terracotta-session_trunk_tests_jetty-7.4.4_JROCKIT_1.6_3.7.x/trunk/system-tests/target/temp/SynchWriteMultiThreadsTest/sandbox/server_1/jrockit.19791.dump
            No snapshot file (core dump) will be created because core dumps have been
            disabled. To enable core dumping, try "ulimit -c unlimited"
            before starting JRockit again.

            Error Message: Illegal memory access. [54]
            Signal info : si_signo=11, si_code=1 si_addr=0xfffffff8
            Version : Oracle JRockit(R) R28.1.4-7-144370-1.6.0_26-20110617-2130-linux-ia32
            CPU : AMD Opteron QC/Phenom SSE SSE2 SSE3 SSE4A AMD64
            Number CPUs : 4
            Tot Phys Mem : 6442450944 (6144 MB)
            OS version : Red Hat Enterprise Linux Server release 5.6 (Tikanga)
            Linux version 2.6.18-238.9.1.el5xen (mockbuild@x86-002.build.bos.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-50)) #1 SMP Fri Mar 18 12:53:42 EDT 2011 (x86_64)
            Hypervisor : Xen v3.4
            Thread System: Linux NPTL
            LibC release : 2.5-stable
            Java locking : Lazy unlocking enabled (class banning) (transfer banning)
            State : JVM is running (Main thread has finished)
            Command Line : -XX:+HeapDumpOnOutOfMemoryError -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=4964 -Dcom.tc.session.debug.sessions=true -Drmi.registry.port=13087 -verbose:gc -Xverboselog:/export1/cruise/jenkins/workspace/terracotta-session_trunk_tests_jetty-7.4.4_JROCKIT_1.6_3.7.x/trunk/system-tests/target/temp/SynchWriteMultiThreadsTest/sandbox/server_1-gc.log -Djava.security.egd=file:/dev/./urandom -Djetty.home=/export1/cruise/.tc/appservers/jetty-7.4.4 -Djetty.port=14976 -Djava.io.tmpdir=/export1/cruise/jenkins/workspace/terracotta-session_trunk_tests_jetty-7.4.4_JROCKIT_1.6_3.7.x/trunk/system-tests/target/temp/SynchWriteMultiThreadsTest/sandbox/work -Dsun.java.launcher=SUN_STANDARD com.tc.lcp.CargoLinkedChildProcess org.eclipse.jetty.start.Main 51660 /export1/cruise/jenkins/workspace/terracotta-session_trunk_tests_jetty-7.4.4_JROCKIT_1.6_3.7.x/trunk/system-tests/target/temp/SynchWriteMultiThreadsTest/sandbox/server_1 STOP.PORT=21626 STOP.KEY=secret OPTIONS=All /export1/cruise/jenkins/workspace/terracotta-session_trunk_tests_jetty-7.4.4_JROCKIT_1.6_3.7.x/trunk/system-tests/target/temp/SynchWriteMultiThreadsTest/sandbox/server_1/jetty.xml
            Repository : /export1/cruise/jenkins/workspace/terracotta-session_trunk_tests_jetty-7.4.4_JROCKIT_1.6_3.7.x/trunk/system-tests/target/temp/SynchWriteMultiThreadsTest/sandbox/work/2011_12_31_11_02_31_19791
            java.home : /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre
            j.class.path : /export1/cruise/.tc/appservers/jetty-7.4.4/start.jar:/export1/cruise/.m2/repository/org/terracotta/linked-child-process/1.0.6/linked-child-process-1.0.6.jar
            j.lib.path : /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386/jrockit:/export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386:/export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/../lib/i386:/shares/monkeyshare/setup/jdk/linux/hotspot1.6.0_27/jre/lib/i386/server:/shares/monkeyshare/setup/jdk/linux/hotspot1.6.0_27/jre/lib/i386:/shares/monkeyshare/setup/jdk/linux/hotspot1.6.0_27/jre/../lib/i386
            JAVA_HOME : /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26
            JAVAOPTIONS: <not set>
            LD_LIBRARY_PATH: /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386/jrockit:/export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386:/export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/../lib/i386:/shares/monkeyshare/setup/jdk/linux/hotspot1.6.0_27/jre/lib/i386/server:/shares/monkeyshare/setup/jdk/linux/hotspot1.6.0_27/jre/lib/i386:/shares/monkeyshare/setup/jdk/linux/hotspot1.6.0_27/jre/../lib/i386
            LD_ASSUME_KERNEL: <not set>
            LD_PRELOAD : <not set>
            StackOverFlow: 0 StackOverFlowErrors have occured
            OutOfMemory : 0 OutOfMemoryErrors have occured
            C Heap : Good; no memory allocations have failed
            GC Strategy : Mode: throughput, with strategy: genparpar (basic strategy: genparpar)
            GC Status : OC is not running. Last finished OC was OC#16.
            : YC is not running. Last finished YC was YC#141.
            YC Promotion : Last YC successfully promoted all objects
            YC History : Ran 0 YCs before OC#12.
            : Ran 0 YCs before OC#13.
            : Ran 0 YCs before OC#14.
            : Ran 2 YCs before OC#15.
            : Ran 6 YCs before OC#16.
            : Ran 2 YCs since last OC.
            Heap : 0xb6605000 - 0xbeabd000 (Size: 133 MB)
            Heap History : OC#8 - 0xba605000 (64 MB -> 77 MB; +13108 KB)
            : OC#12 - 0xbb2d2000 (77 MB -> 92 MB; +15732 KB)
            : OC#15 - 0xbc22f000 (92 MB -> 111 MB; +18876 KB)
            : OC#16 - 0xbd49e000 (111 MB -> 133 MB; +22652 KB)
            Compaction : (no compaction area)
            Allocation : TLA-min: 2048, TLA-preferred: 65536 TLA-waste limit: 2048
            NurseryList : 0xb6ca5008 - 0xbbb0bf68
            KeepArea : 0xb9be8168 - 0xbaf15878
            KA Markers : [ 0xb919cb58,  0xb9be8168 , 0xbaf15878 ]
            Forbidden A : (none)
            Previous KA : 0xb919cb58 - 0xb9be8168
            Previous FA : (none)
            CompRefs : References are 32-bit.

            Registers (from ThreadContext: 0xb24e83dc:
            eax = ffffffff ecx = b22baa1c edx = 00000000 ebx = b24e876c
            esp = b24e86d0 ebp = b24e8708 esi = fffffff8 edi = b24e8818
            es = 0000002b cs = b2fb0023 ss = 0000002b ds = 0000002b
            fs = 00000003 gs = 00000063
            eip = f7e2c00d eflags = 00000282

            Loaded modules:
            (* denotes the module where the exception occured)
            08048000-08057193 /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/bin/java
            ffffe000-ffffe7bf /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/bin/java
            00674000-006760fb /lib/libdl.so.2
            00659000-0066df17 /lib/libpthread.so.0
            004fe000-00650723 /lib/libc.so.6
            004df000-004f9457 /lib/ld-linux.so.2
            f7c19000-f7edb317 */export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386/jrockit/libjvm.so
            f7bfc000-f7c164eb /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386/libosal.so
            f7bf0000-f7bfa5cf /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386/libutil.so
            f7bc7000-f7beda23 /lib/libm.so.6
            f7bbe000-f7bc4ebf /lib/librt.so.1
            f6fd1000-f6fdd747 /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386/libjfr.so
            b42d6000-b42e09bb /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386/libverify.so
            b42b1000-b42d3f07 /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386/libjava.so
            007d4000-007e87a7 /lib/libnsl.so.1
            f7f55000-f7f5ae50 /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386/native_threads/libhpi.so
            b3e13000-b3e216e4 /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386/libzip.so
            b3b8e000-b3b93a06 /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386/libmanagement.so
            b3b79000-b3b8c150 /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386/libnet.so
            b37ca000-b37d2f93 /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386/libjmapi.so
            b3721000-b3726013 /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386/liborii.so
            b3607000-b360d4e4 /export1/cruise/jenkins/tools/JROCKIT_1.6/jrockit-1.6.0_26/jre/lib/i386/libnio.so

            Stack:
            (* marks the word pointed to by the stack pointer)
            b24e86d0: b24e86f4* 097f5508 0000000c 0000000c 00000002 f7e2bdf0
            b24e86e8: 00000000 ffffffff ffffffff b22baa1c 04000009 ffffffff
            b24e8700: b24e876c b24e876c b24e8718 f7e2c0f5 b24e876c 00000000
            b24e8718: b24e8758 f7e2c507 097f5270 f7e2bdf0 b24e876c b24e876c

            Code:
            (* marks the word pointed to by the instruction pointer)
            f7e2bfdc: 1c4389ec 89f0458b 458b2043 0c4383f4 24438901 5b30c483
            f7e2bff4: 90c35d5e 097407a8 012843c7 8b000000 83c68901 7389f8e6
            f7e2c00c: a8068b14* 74c18901 fee08305 7983088b 40740344 8d48418b
            f7e2c024: 43890604 90afeb18 0026748d 85e0558b 8b4174d2 4389e045


            Last optimized methods:
            No methods optimized.

            Thread:
            "qtp386208759-54" id=54 idx=0xdc tid=19853 lastJavaFrame=0xb24e89ec
            Stack 0: start=0xb24c8000, end=0xb24ea000, guards=0xb24cd000 (ok), forbidden=0xb24cb000

            Thread Stack Trace:
            at findNext+253(refiter.c:409)@0xf7e2c00d
            at findNextToReturn+100(refiter.c:302)@0xf7e2c0f5
            at refIterFillFromFrame+342(refiter.c:123)@0xf7e2c507
            at trProcessLocksForThread+73(roots.c:620)@0xf7e374ea
            at get_all_locks+95(javalock.c:1265)@0xf7d35810
            at javaLockConvertLazyToThin+133(javalock.c:1760)@0xf7d359f6
            at RJNI_jrockit_vm_Locks_checkAndTransferLazyLocked+486(javalock.c:2032)@0xf7d36a27
            -- Java stack --
            at jrockit/vm/Locks.checkAndTransferLazyLocked(Ljava/lang/Object;)I(Native Method)
            at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:971)
            at jrockit/vm/Locks.monitorEnter(Locks.java:2179)
            at EDU/oswego/cs/dl/util/concurrent/ConcurrentReaderHashMap.getTableForReading()[LEDU/oswego/cs/dl/util/concurrent/ConcurrentReaderHashMap$Entry;(Unknown Source)
                at EDU/oswego/cs/dl/util/concurrent/ConcurrentReaderHashMap.get(Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
                at com/tc/object/PortabilityImpl.isClassPhysicallyInstrumented(PortabilityImpl.java:144)
                at com/tc/object/PortabilityImpl.isPortableClass(PortabilityImpl.java:127)
                at com/tc/object/ClientObjectManagerImpl.isPortableClass(ClientObjectManagerImpl.java:1225)
                at com/tc/object/TCClassImpl.createPortableFields(TCClassImpl.java:341)
                at com/tc/object/TCClassImpl.<init>(TCClassImpl.java:119)
                at com/tc/object/TCClassFactoryImpl.createTCClass(TCClassFactoryImpl.java:95)
                at com/tc/object/TCClassFactoryImpl.getOrCreate(TCClassFactoryImpl.java:72)
                at com/tc/object/ClientObjectManagerImpl.lookup(ClientObjectManagerImpl.java:567)
                at com/tc/object/ClientObjectManagerImpl.lookupObject(ClientObjectManagerImpl.java:463)
                at com/tc/object/ClientObjectManagerImpl.lookupObject(ClientObjectManagerImpl.java:451)
                at com/tc/object/bytecode/ManagerImpl.lookupObject(ManagerImpl.java:484)
                at com/tc/object/bytecode/ManagerUtil.lookupObject(ManagerUtil.java:389)
                at com/terracotta/toolkit/collections/ConcurrentDistributedMapDso.lookupAndFaultIn(ConcurrentDistributedMapDso.java:708)
                at com/terracotta/toolkit/collections/ConcurrentDistributedMapDso.lookupAndFaultIn(ConcurrentDistributedMapDso.java:702)
                at com/terracotta/toolkit/collections/ConcurrentDistributedMapDso.get(ConcurrentDistributedMapDso.java:198)
                at com/terracotta/toolkit/collections/ConcurrentDistributedMapDsoArray.get(ConcurrentDistributedMapDsoArray.java:251)
                at org/terracotta/collections/ConcurrentDistributedMap.get(ConcurrentDistributedMap.java:220)
                at com/terracotta/session/SessionDataStore.find(SessionDataStore.java:145)
                at com/terracotta/session/TerracottaSessionManager.doGetSession(TerracottaSessionManager.java:671)
                at com/terracotta/session/TerracottaSessionManager.getSession(TerracottaSessionManager.java:562)
                at com/terracotta/session/SessionRequest.getTerracottaSession(SessionRequest.java:150)
                at com/terracotta/session/SessionRequest.getSession(SessionRequest.java:109)
                at com/tctest/webapp/servlets/SynchWriteMultiThreadsTestServlet.doGet(SynchWriteMultiThreadsTestServlet.java:18)
                at javax/servlet/http/HttpServlet.service(HttpServlet.java:707)
                at javax/servlet/http/HttpServlet.service(HttpServlet.java:820)
                at org/eclipse/jetty/servlet/ServletHolder.handle(ServletHolder.java:538)
                at org/eclipse/jetty/servlet/ServletHandler$CachedChain.doFilter(ServletHandler.java:1352)
                at com/terracotta/session/SessionFilter.doHttpFilter(SessionFilter.java:81)
                at com/terracotta/session/SessionFilter.doFilter(SessionFilter.java:65)
                at org/terracotta/session/BaseExpressSessionFilter.doFilter(BaseExpressSessionFilter.java:44)
                at org/terracotta/session/TerracottaJetty74xSessionFilter.doFilter(TerracottaJetty74xSessionFilter.java:13)
                at org/eclipse/jetty/servlet/ServletHandler$CachedChain.doFilter(ServletHandler.java:1323)
                at org/eclipse/jetty/servlet/ServletHandler.doHandle(ServletHandler.java:476)
                at org/eclipse/jetty/server/handler/ScopedHandler.handle(ScopedHandler.java:119)
                at org/eclipse/jetty/security/SecurityHandler.handle(SecurityHandler.java:480)
                at org/eclipse/jetty/server/session/SessionHandler.doHandle(SessionHandler.java:225)
                at org/eclipse/jetty/server/handler/ContextHandler.doHandle(ContextHandler.java:937)
                at org/eclipse/jetty/servlet/ServletHandler.doScope(ServletHandler.java:406)
                at org/eclipse/jetty/server/session/SessionHandler.doScope(SessionHandler.java:183)
                at org/eclipse/jetty/server/handler/ContextHandler.doScope(ContextHandler.java:871)
                at org/eclipse/jetty/server/handler/ScopedHandler.handle(ScopedHandler.java:117)
                at org/eclipse/jetty/server/handler/ContextHandlerCollection.handle(ContextHandlerCollection.java:247)
                at org/eclipse/jetty/server/handler/HandlerCollection.handle(HandlerCollection.java:149)
                at org/eclipse/jetty/server/handler/HandlerWrapper.handle(HandlerWrapper.java:110)
                at org/eclipse/jetty/server/Server.handle(Server.java:346)
                at org/eclipse/jetty/server/HttpConnection.handleRequest(HttpConnection.java:589)
                at org/eclipse/jetty/server/HttpConnection$RequestHandler.headerComplete(HttpConnection.java:1048)
                at org/eclipse/jetty/http/HttpParser.parseNext(HttpParser.java:601)
                at org/eclipse/jetty/http/HttpParser.parseAvailable(HttpParser.java:214)
                at org/eclipse/jetty/server/HttpConnection.handle(HttpConnection.java:411)
                at org/eclipse/jetty/io/nio/SelectChannelEndPoint.handle(SelectChannelEndPoint.java:535)
                at org/eclipse/jetty/io/nio/SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:40)
                at org/eclipse/jetty/util/thread/QueuedThreadPool$3.run(QueuedThreadPool.java:529)
                at java/lang/Thread.run(Thread.java:662)
                at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
                -- end of trace

            Memory usage report:
            Total mapped                  1217824KB           (reserved=944628KB)
            -              Java heap      1048576KB           (reserved=912672KB)
            -              GC tables        35084KB         
            -          Thread stacks        31284KB           (#threads=151)
            -          Compiled code         4352KB           (used=3823KB)
            -               Internal         1352KB         
            -                     OS         9272KB         
            -                  Other        59488KB         
            -        Java class data        27392KB           (malloced=27135KB #38974 in 6995 classes)
                                                              Not tracing sites.

            - Native memory tracking         1024KB           (malloced=120KB #8)
                                                              Not tracing sites.


            Set the env variable TRACE_ALLOC_SITES=1 or use the print_memusage switch
            trace_alloc_sites=true to enable alloc site tracing.

                **************************************************************************
                *  If you see this dump, please go to                                    *
                *  http://download.oracle.com/docs/cd/E15289_01/go2troubleshooting.html  *
                *  for troubleshooting information.                                      *
                **************************************************************************

            ===== END DUMP ===============================================================