3 Replies Latest reply: Jan 18, 2012 4:39 PM by jschellSomeoneStoleMyAlias RSS

    Hotspot JVM Crash during Compiler Optimization of Portal related JSPs

    user8321790
      Problem Statement_
      During the execution of Performance testing, Hotspot JVM Crashes are observed while the portal related jsps (tablecolumn.jsp) are optimized by the HotSpot Compiler.
      Occurrence_
      When: During the Performance test
      Test Duration: 2-3hrs
      JVM Crashes after an hour or so.
      Server Details_
      2 Weblogic Servers (Version 8.1 SP4) with 3 instances each
      Portal Server: SUN Portal Server 6.1 (Portal Server installed on top of Weblogic Server)
      JDK Version: j2sdk1.4.2_18
      Most of the times Stuck threads are observed at around the same time of the JVM Crash. Please find below the details from the hotspot error logs:
      Signal that caused the crash:* SIGSEGV (0xb) at pc=0xfeccd87c, pid=7177, tid=9
      Problematic frame:* V [libjvm.so+0xcd87c]
      Current thread (0x000f9910):* JavaThread "CompilerThread1" daemon [_thread_in_native, id=9]
      Current compiletask:* opto:3086!_jsps._etc._opt._SUNWps._desktop._dealerportal_en_US._DealershipManagementContainer._html._tablecolumn_jsp._jspService(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (1225 bytes)
      Stuck thread in the instance logs during the time of the crash*:<Dec 7, 2011 5:04:36 AM EST> <Error> <WebLogicServer> <gmacds645a> <CGDP-B-3> <weblogic.health.CoreHealthMonitor> <<WLS K
      ernel>> <> <BEA-000337> <ExecuteThread: '20' for queue: 'weblogic.kernel.Default' has been busy for "1,286" seconds working on the request "Http Request: /portal/dt", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>
      Few points to be noted:
      • Crash occurs always due to CompilerThread1 or CompilerThread0
      • Problematic Stack frame is always V [libjvm.so+0xcd87c]
      • Current compiletask during the jvm crash is always:
      opto:2990!_jsps._etc._opt._SUNWps._desktop._dealerportal_en_US._*+<<SomeContainer>>+*._html._tablecolumn_jsp._jspService(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (1225 bytes)
      Contents of the hs_err_pid7177.log File:
      #
      # An unexpected error has been detected by HotSpot Virtual Machine:
      #
      # SIGSEGV (0xb) at pc=0xfeccd87c, pid=7177, tid=9
      #
      # Java VM: Java HotSpot(TM) Server VM (1.4.2_18-b06 mixed mode)
      # Problematic frame:
      # V [libjvm.so+0xcd87c]
      #--------------- T H R E A D ---------------
      Current thread (0x000f9910): JavaThread "CompilerThread1" daemon [_thread_in_native, id=9]
      siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x00000004
      Registers:
      O0=0x019f46a0 O1=0x00000000 O2=0x00004000 O3=0x00002fcb
      O4=0x00000000 O5=0x0037f521 O6=0x88e7e928 O7=0xfeccd7dc
      G1=0x00381958 G2=0x00000000 G3=0x00000000 G4=0x00000000
      G5=0x00000000 G6=0x00000000 G7=0xff351000 Y=0x00000000
      PC=0xfeccd87c nPC=0xfeccd880
      Top of Stack: (sp=0x88e7e928)
      0x88e7e928: 01564880 00fe5b14 88e7ebdc 00fefecc
      0x88e7e938: 00fee9b0 01564881 88e7ec04 00feea10
      0x88e7e948: 88e7ebcc 88e7eb20 88e7eb20 88e7eb10
      0x88e7e958: 00000000 00381918 88e7e988 feccd420
      0x88e7e968: feccf8cc 000031d3 0000000a 031c8064
      0x88e7e978: 00000009 00d39e4c 00e10b88 00000026
      0x88e7e988: 0043a848 00000007 88e7ebdc 0064dc6c
      0x88e7e998: 0064dc0c 00fee9b0 88e7ec04 0064dc6c
      Instructions: (pc=0xfeccd87c)
      0xfeccd86c: c4 02 60 1c d0 06 20 58 93 28 a0 02 c8 02 00 09
      0xfeccd87c: c4 01 20 04 c4 00 a0 00 80 a0 a0 00 32 48 00 12
      Stack: [0x88c80000,0x88e80000), sp=0x88e7e928, free space=2042k
      Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
      V [libjvm.so+0xcd87c]
      V [libjvm.so+0xcd428]
      V [libjvm.so+0xcd428]
      V [libjvm.so+0xcd428]
      V [libjvm.so+0x1c206c]
      V [libjvm.so+0x20449c]
      V [libjvm.so+0x2028b0]
      V [libjvm.so+0x1ff044]
      V [libjvm.so+0x1fe808]
      V [libjvm.so+0x2aee5c]
      V [libjvm.so+0x26be98]
      V [libjvm.so+0x4bd138]
      Current CompileTask:
      opto:3086 ! jsps.etc._opt._SUNWps._desktop._dealerportal_en_US._DealershipManagementContainer._html._tablecolumn_jsp._jspService(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (1225 bytes)
      Other Threads:
      0x000efbf8 VMThread [id=2]
      0x000fb100 WatcherThread [id=10]
      VM state:not at safepoint (normal execution)
      VM Mutex/Monitor currently owned by a thread: None
      Heap
      def new generation total 318080K, used 96668K [0x89800000, 0x9d800000, 0x9d800000)
      eden space 308480K, 28% used [0x89800000, 0x8ed07168, 0x9c540000)
      from space 9600K, 100% used [0x9cea0000, 0x9d800000, 0x9d800000)
      to space 9600K, 0% used [0x9c540000, 0x9c540000, 0x9cea0000)
      tenured generation total 1245184K, used 789326K [0x9d800000, 0xe9800000, 0xe9800000)
      the space 1245184K, 63% used [0x9d800000, 0xcdad3900, 0xcdad3a00, 0xe9800000)
      compacting perm gen total 49920K, used 49680K [0xe9800000, 0xec8c0000, 0xf9800000)
      the space 49920K, 99% used [0xe9800000, 0xec884010, 0xec884200, 0xec8c0000)
      Dynamic libraries:
      0x00010000      /apps/weblogic/81_sp4/j2sdk1.4.2_18/bin/java
      0xff380000      /usr/lib/lwp/libthread.so.1
      0xff370000      /usr/lib/libdl.so.1
      0xff280000      /usr/lib/libc.so.1
      0xff3b0000      /usr/platform/SUNW,Sun-Fire-15000/lib/libc_psr.so.1
      0xfec00000      /apps/weblogic/81_sp4/j2sdk1.4.2_18/jre/lib/sparc/server/libjvm.so
      0xff230000      /usr/lib/libCrun.so.1
      0xfebe0000      /usr/lib/libsocket.so.1
      0xfeb00000      /usr/lib/libnsl.so.1
      0xfeab0000      /usr/lib/libm.so.1
      0xfebc0000      /usr/lib/libsched.so.1
      0xfea90000      /usr/lib/libmp.so.2
      0xfea50000      /apps/weblogic/81_sp4/j2sdk1.4.2_18/jre/lib/sparc/native_threads/libhpi.so
      0xfea30000      /usr/lib/nss_files.so.1
      0xfe9f0000      /apps/weblogic/81_sp4/j2sdk1.4.2_18/jre/lib/sparc/libverify.so
      0xfe990000      /apps/weblogic/81_sp4/j2sdk1.4.2_18/jre/lib/sparc/libjava.so
      0xfe970000      /apps/weblogic/81_sp4/j2sdk1.4.2_18/jre/lib/sparc/libzip.so
      0x86ed0000      /apps/weblogic/81_sp4/j2sdk1.4.2_18/jre/lib/sparc/libnet.so
      0x86d60000      /apps/weblogic/81_sp4/weblogic81/server/lib/solaris/libweblogicunix1.so
      0x86950000      /apps/weblogic/81_sp4/weblogic81/server/lib/solaris/libstackdump.so
      0x86860000      /apps/weblogic/81_sp4/weblogic81/server/lib/solaris/libmuxer.so
      0x86510000      /usr/ucblib/libucb.so.1
      0x85d20000      /usr/lib/libresolv.so.2
      0x83bc0000      /usr/lib/libelf.so.1
      0x864f0000      /apps/weblogic/81_sp4/j2sdk1.4.2_18/jre/lib/sparc/libnio.so
      0x839e0000      /usr/lib/librt.so.1
      0x839c0000      /usr/lib/libaio.so.1
      0x83860000      /usr/lib/libmd5.so.1
      0x83840000      /usr/lib/libsendfile.so.1
      0x836e0000      /apps/weblogic/81_sp4/j2sdk1.4.2_18/jre/lib/sparc/libioser12.so
      0x80c90000      /usr/lib/nss_dns.so.1
      0x7bc00000      /apps/oblix/netpoint/AccessServerSDK/oblix/lib/libobaccess.so
      0x80ad0000      ../../../lib/libpthread.so.1
      0x7bb00000      /apps/oblix/netpoint/AccessServerSDK/oblix/lib/libstdc++.so.5
      0x80ab0000      /apps/oblix/netpoint/AccessServerSDK/oblix/lib/libgcc_s.so.1
      0x7b400000      /apps/oblix/netpoint/AccessServerSDK/oblix/lib/libxmlengine.so
      VM Arguments:
      jvm_args: -XX:NewSize=320m -XX:MaxNewSize=320m -Xms1536m -Xmx1536m -XX:MaxPermSize=256m -Dweblogic.security.SSL.enforceConstraints=off -Djavax.xml.soap.MessageFactory=com.sun.xml.messaging.saaj.soap.ver1_1.SOAPMessageFactory1_1Impl -Djava.security.auth.login.config=/apps/weblogic/81_sp4/user_projects/domains/GDPPROD/.java.login.config -DLOG_COMPATMODE=Off -Ds1is.java.util.logging.config.class=com.sun.identity.log.s1is.LogConfigReader -Djava.protocol.handler.pkgs=com.iplanet.services.comm -Dcom.iplanet.am.serverMode=true -Dfile.encoding=UTF-8 -Dweblogic.Name=CGDP-B-3 -Dweblogic.management.username= -Dweblogic.management.password= -Dweblogic.management.server=http://10.13.52.81:7001 -Djava.security.policy=/apps/weblogic/81_sp4/weblogic81/server/lib/weblogic.policy
      java_command: weblogic.Server
      Launcher Type: SUN_STANDARD
      Environment Variables:
      JAVA_HOME=/apps/weblogic/81_sp4/j2sdk1.4.2_18
      PATH=/apps/weblogic/81_sp4/weblogic81/server/bin:/apps/weblogic/81_sp4/j2sdk1.4.2_18/jre/bin:/apps/weblogic/81_sp4/j2sdk1.4.2_18/bin:/usr/local/bin:/usr/local/bin:/usr/bin::/usr/sbin:/usr/local/bin:/usr/bin:/bin:/usr/dt/bin:/usr/openwin/bin:/opt/EMCpower/bin:/etc/emc/bin:/etc:.::.:
      LD_LIBRARY_PATH=/apps/weblogic/81_sp4/j2sdk1.4.2_18/jre/lib/sparc/server:/apps/weblogic/81_sp4/j2sdk1.4.2_18/jre/lib/sparc:/apps/weblogic/81_sp4/j2sdk1.4.2_18/jre/../lib/sparc:/usr/lib/mps/secv1:/usr/lib/mps/secv1:/usr/lib/mps/secv1:/apps/SunOne/SUNWam/lib:/apps/SunOne/SUNWam/ldaplib/ldapsdk:/apps/weblogic/81_sp4/server/lib:/apps/SunOne/SUNWps/lib:/usr/lib/lwp:/apps/SunOne/SUNWps/lib:/usr/lib/lwp:/apps/oblix/netpoint/AccessServerSDK/oblix/lib:/etc/emc/rsa/cst/lib:/apps/weblogic/81_sp4/weblogic81/server/lib/solaris:/apps/weblogic/81_sp4/weblogic81/server/lib/solaris/oci920_8
      SHELL=/usr/bin/ksh
      HOSTTYPE=sparc
      OSTYPE=solaris2.9
      MACHTYPE=sparc-sun-solaris2.9
      --------------- S Y S T E M ---------------

      OS: Solaris 9 12/03 s9s_u5wos_08b SPARC
      Copyright 2003 Sun Microsystems, Inc. All Rights Reserved.
      Use is subject to license terms.
      Assembled 21 November 2003

      uname:SunOS 5.9 Generic_122300-46 sun4u (T2 libthread)
      rlimit: STACK 8192k, CORE infinity, NOFILE 16000, AS infinity
      load average:0.56 0.52 0.55

      CPU:total 8 has_v8, has_v9, has_vis1, has_vis2, is_ultra3

      Memory: 8k page, physical 16777216k(2202056k free)

      vm_info: Java HotSpot(TM) Server VM (1.4.2_18-b06) for solaris-sparc, built on May 27 2008 23:45:13 by unknown with Workshop 5.2 compat=5
        • 1. Re: Hotspot JVM Crash during Compiler Optimization of Portal related JSPs
          user8321790
          In addition to the details above, Please find below the Java threads section from the hotspot error log file:

          --------------- P R O C E S S ---------------

          Java Threads: ( => current thread )
          0x01b54a88 JavaThread "LDAPConnThread 10.13.245.94:4636" daemon [_thread_blocked, id=3451]
          0x035df1d8 JavaThread "LDAPConnThread 10.13.245.94:4636" daemon [_thread_blocked, id=3421]
          0x005e49c8 JavaThread "LDAPConnThread 10.13.245.23:4636" daemon [_thread_blocked, id=3315]
          0x017678a8 JavaThread "LDAPConnThread 10.13.245.23:4636" daemon [_thread_blocked, id=3171]
          0x005e4788 JavaThread "LDAPConnThread 10.13.245.23:4636" daemon [_thread_blocked, id=3021]
          0x00d62778 JavaThread "LDAPConnThread 10.13.245.94:4636" daemon [_thread_blocked, id=3019]
          0x00f82ee0 JavaThread "LDAPConnThread 10.13.245.94:4636" daemon [_thread_blocked, id=3018]
          0x01f5df70 JavaThread "LDAPConnThread 10.13.245.94:4636" daemon [_thread_blocked, id=3017]
          0x0362abb8 JavaThread "LDAPConnThread 10.13.245.94:4389" daemon [_thread_blocked, id=3012]
          0x014bf888 JavaThread "LDAPConnThread 10.13.245.94:4389" daemon [_thread_blocked, id=3011]
          0x00fa5e30 JavaThread "LDAPConnThread 10.13.245.94:4389" daemon [_thread_blocked, id=3010]
          0x0108ea20 JavaThread "LDAPConnThread 10.13.245.94:4389" daemon [_thread_blocked, id=3009]
          0x01765fc0 JavaThread "LDAPConnThread 10.13.245.23:4636" daemon [_thread_blocked, id=3006]
          0x00fa5c88 JavaThread "LDAPConnThread 10.13.245.94:4389" daemon [_thread_blocked, id=2977]
          0x064bf6c0 JavaThread "LDAPConnThread 10.13.245.94:4389" daemon [_thread_blocked, id=2963]
          0x00d5f118 JavaThread "LDAPConnThread 10.13.245.94:4636" daemon [_thread_blocked, id=2948]
          0x0011b9a8 JavaThread "LDAPConnThread 10.13.245.94:4389" daemon [_thread_blocked, id=2935]
          0x01f5c9c0 JavaThread "LDAPConnThread 10.13.245.94:4636" daemon [_thread_blocked, id=2897]
          0x00e1b278 JavaThread "LDAPConnThread 10.13.245.94:4389" daemon [_thread_blocked, id=2840]
          0x00e1a698 JavaThread "LDAPConnThread 10.13.245.94:4636" daemon [_thread_blocked, id=2499]
          0x01793e90 JavaThread "LDAPConnThread 10.13.245.94:4389" daemon [_thread_blocked, id=1842]
          0x019c7c08 JavaThread "LDAPConnThread 10.13.245.94:4389" daemon [_thread_blocked, id=1841]
          0x01830418 JavaThread "Thread-173" daemon [_thread_blocked, id=364]
          0x010c8a20 JavaThread "LDAPConnThread 10.13.245.94:4636" daemon [_thread_blocked, id=287]
          0x010c44d8 JavaThread "Thread-172" daemon [_thread_blocked, id=286]
          0x019e9970 JavaThread "Thread-171" daemon [_thread_blocked, id=285]
          0x019e8d78 JavaThread "Thread-170" daemon [_thread_blocked, id=284]
          0x019e7b78 JavaThread "Thread-169" daemon [_thread_blocked, id=283]
          0x019e6f80 JavaThread "Thread-168" daemon [_thread_blocked, id=282]
          0x019e6388 JavaThread "Thread-167" daemon [_thread_blocked, id=281]
          0x019e5790 JavaThread "Thread-166" daemon [_thread_blocked, id=280]
          0x019e4bc8 JavaThread "Thread-165" daemon [_thread_blocked, id=279]
          0x0144eeb8 JavaThread "Thread-164" daemon [_thread_blocked, id=278]
          0x0144e2c0 JavaThread "Thread-163" daemon [_thread_blocked, id=277]
          0x0144d6c8 JavaThread "Thread-162" daemon [_thread_blocked, id=276]
          0x0144cad0 JavaThread "Thread-161" daemon [_thread_blocked, id=275]
          0x0144bed8 JavaThread "Thread-160" daemon [_thread_blocked, id=274]
          0x0144b2e0 JavaThread "Thread-159" daemon [_thread_blocked, id=273]
          0x0144a6e8 JavaThread "Thread-158" daemon [_thread_blocked, id=272]
          0x01449b58 JavaThread "Thread-157" daemon [_thread_blocked, id=271]
          0x00728538 JavaThread "Thread-156" daemon [_thread_blocked, id=270]
          0x00727940 JavaThread "Thread-155" daemon [_thread_blocked, id=269]
          0x00726d48 JavaThread "Thread-154" daemon [_thread_blocked, id=268]
          0x00726150 JavaThread "Thread-153" daemon [_thread_blocked, id=267]
          0x00725558 JavaThread "Thread-152" daemon [_thread_blocked, id=266]
          0x00724960 JavaThread "Thread-151" daemon [_thread_blocked, id=265]
          0x00723d98 JavaThread "Thread-150" daemon [_thread_blocked, id=264]
          0x00ab8da0 JavaThread "Thread-149" daemon [_thread_blocked, id=263]
          0x00ab81a8 JavaThread "Thread-148" daemon [_thread_blocked, id=262]
          0x00ab75b0 JavaThread "Thread-147" daemon [_thread_blocked, id=261]
          0x00ab69b8 JavaThread "Thread-146" daemon [_thread_blocked, id=260]
          0x00ab5dc0 JavaThread "Thread-145" daemon [_thread_blocked, id=259]
          0x00ab51c8 JavaThread "Thread-144" daemon [_thread_blocked, id=258]
          0x00ab45d0 JavaThread "Thread-143" daemon [_thread_blocked, id=257]
          0x03423d80 JavaThread "Thread-142" daemon [_thread_blocked, id=256]
          0x03423188 JavaThread "Thread-141" daemon [_thread_blocked, id=255]
          0x03422590 JavaThread "Thread-140" daemon [_thread_blocked, id=254]
          0x03421998 JavaThread "Thread-139" daemon [_thread_blocked, id=253]
          0x03420da0 JavaThread "Thread-138" daemon [_thread_blocked, id=252]
          0x0341fba0 JavaThread "Thread-137" daemon [_thread_blocked, id=251]
          0x04f46c58 JavaThread "Thread-136" daemon [_thread_blocked, id=250]
          0x04f46060 JavaThread "Thread-135" daemon [_thread_blocked, id=249]
          0x04f45468 JavaThread "Thread-134" daemon [_thread_blocked, id=248]
          0x04f44870 JavaThread "Thread-133" daemon [_thread_blocked, id=247]
          0x04f43d50 JavaThread "Thread-132" daemon [_thread_blocked, id=246]
          0x015ae7b8 JavaThread "Thread-131" daemon [_thread_blocked, id=245]
          0x015adbc0 JavaThread "Thread-130" daemon [_thread_blocked, id=244]
          0x015acfc8 JavaThread "Thread-129" daemon [_thread_blocked, id=243]
          0x015ac3d0 JavaThread "Thread-128" daemon [_thread_blocked, id=242]
          0x008cac70 JavaThread "Thread-127" daemon [_thread_blocked, id=241]
          0x008ca078 JavaThread "Thread-126" daemon [_thread_blocked, id=240]
          0x008c9480 JavaThread "Thread-125" daemon [_thread_blocked, id=239]
          0x008c8888 JavaThread "Thread-124" daemon [_thread_blocked, id=238]
          0x008c7cc0 JavaThread "Thread-123" daemon [_thread_blocked, id=237]
          0x06841f70 JavaThread "Thread-122" daemon [_thread_blocked, id=236]
          0x06841378 JavaThread "Thread-121" daemon [_thread_blocked, id=235]
          0x06840780 JavaThread "Thread-120" daemon [_thread_blocked, id=234]
          0x0683fc98 JavaThread "Thread-119" daemon [_thread_blocked, id=233]
          0x06353328 JavaThread "Thread-118" daemon [_thread_blocked, id=232]
          0x06352730 JavaThread "Thread-117" daemon [_thread_blocked, id=231]
          0x06351b38 JavaThread "Thread-116" daemon [_thread_blocked, id=230]
          0x06350f70 JavaThread "Thread-115" daemon [_thread_blocked, id=229]
          0x00fd0ae8 JavaThread "Thread-114" daemon [_thread_blocked, id=228]
          0x00fcfef0 JavaThread "Thread-113" daemon [_thread_blocked, id=227]
          0x00fcf2f8 JavaThread "Thread-112" daemon [_thread_blocked, id=226]
          0x00fce700 JavaThread "Thread-111" daemon [_thread_blocked, id=225]
          0x00ba5430 JavaThread "Thread-110" daemon [_thread_blocked, id=224]
          0x00ba4838 JavaThread "Thread-109" daemon [_thread_blocked, id=223]
          0x00ba3c40 JavaThread "Thread-108" daemon [_thread_blocked, id=222]
          0x00ba30e0 JavaThread "Thread-107" daemon [_thread_blocked, id=221]
          0x03b51c68 JavaThread "Thread-106" daemon [_thread_blocked, id=220]
          0x03b50ad0 JavaThread "Thread-105" daemon [_thread_blocked, id=219]
          0x03b4ffe8 JavaThread "Thread-104" daemon [_thread_blocked, id=218]
          0x015bd7f8 JavaThread "Thread-103" daemon [_thread_blocked, id=217]
          0x015bcc00 JavaThread "Thread-102" daemon [_thread_blocked, id=216]
          0x015bc008 JavaThread "Thread-101" daemon [_thread_blocked, id=215]
          0x015bb478 JavaThread "Thread-100" daemon [_thread_blocked, id=214]
          0x032c9ad0 JavaThread "Thread-99" daemon [_thread_blocked, id=213]
          0x032c8ed8 JavaThread "Thread-98" daemon [_thread_blocked, id=212]
          0x032c82e0 JavaThread "Thread-97" daemon [_thread_blocked, id=211]
          0x00ce7278 JavaThread "Thread-96" daemon [_thread_blocked, id=210]
          0x00ce6680 JavaThread "Thread-95" daemon [_thread_blocked, id=209]
          0x00ce5a88 JavaThread "Thread-94" daemon [_thread_blocked, id=208]
          0x00ce4e90 JavaThread "Thread-93" daemon [_thread_blocked, id=207]
          0x01959280 JavaThread "Thread-92" daemon [_thread_blocked, id=206]
          0x01958688 JavaThread "Thread-91" daemon [_thread_blocked, id=205]
          0x01957ac0 JavaThread "Thread-90" daemon [_thread_blocked, id=204]
          0x01956fd8 JavaThread "Thread-89" daemon [_thread_blocked, id=203]
          0x010b4fb0 JavaThread "Thread-88" daemon [_thread_blocked, id=202]
          0x010b43b8 JavaThread "Thread-87" daemon [_thread_blocked, id=201]
          0x010b37c0 JavaThread "Thread-86" daemon [_thread_blocked, id=200]
          0x010b2bf8 JavaThread "Thread-85" daemon [_thread_blocked, id=199]
          0x01f3ed48 JavaThread "Thread-84" daemon [_thread_blocked, id=198]
          0x01f3e150 JavaThread "Thread-83" daemon [_thread_blocked, id=197]
          0x01f3d5c0 JavaThread "Thread-82" daemon [_thread_blocked, id=196]
          0x00bb0ee0 JavaThread "Thread-81" daemon [_thread_blocked, id=195]
          0x00bb02e8 JavaThread "Thread-80" daemon [_thread_blocked, id=194]
          0x00baf6f0 JavaThread "Thread-79" daemon [_thread_blocked, id=193]
          0x00baeaf8 JavaThread "Thread-78" daemon [_thread_blocked, id=192]
          0x011d1e18 JavaThread "Thread-77" daemon [_thread_blocked, id=191]
          0x011d1220 JavaThread "Thread-76" daemon [_thread_blocked, id=190]
          0x011d0628 JavaThread "Thread-75" daemon [_thread_blocked, id=189]
          0x011cfb40 JavaThread "Thread-74" daemon [_thread_blocked, id=188]
          0x010634b8 JavaThread "Thread-73" daemon [_thread_blocked, id=187]
          0x010628c0 JavaThread "Thread-72" daemon [_thread_blocked, id=186]
          0x01061cc8 JavaThread "Thread-71" daemon [_thread_blocked, id=185]
          0x00a91528 JavaThread "Thread-70" daemon [_thread_blocked, id=184]
          0x00a90930 JavaThread "Thread-69" daemon [_thread_blocked, id=183]
          0x00a8fd38 JavaThread "Thread-68" daemon [_thread_blocked, id=182]
          0x00a8f250 JavaThread "Thread-67" daemon [_thread_blocked, id=181]
          0x035f7de0 JavaThread "Thread-66" daemon [_thread_blocked, id=180]
          0x035f71e8 JavaThread "Thread-65" daemon [_thread_blocked, id=179]
          0x035f65f0 JavaThread "Thread-64" daemon [_thread_blocked, id=178]
          0x035f59f8 JavaThread "Thread-63" daemon [_thread_blocked, id=177]
          0x003c1710 JavaThread "Thread-62" daemon [_thread_blocked, id=176]
          0x003c0b48 JavaThread "Thread-61" daemon [_thread_blocked, id=175]
          0x003c0060 JavaThread "Thread-60" daemon [_thread_blocked, id=174]
          0x0500b7b8 JavaThread "Thread-59" daemon [_thread_blocked, id=173]
          0x0500abc0 JavaThread "Thread-58" daemon [_thread_blocked, id=172]
          0x05009fc8 JavaThread "Thread-57" daemon [_thread_blocked, id=171]
          0x05009400 JavaThread "Thread-56" daemon [_thread_blocked, id=170]
          0x0168ca38 JavaThread "Thread-55" daemon [_thread_blocked, id=169]
          0x0168be40 JavaThread "Thread-54" daemon [_thread_blocked, id=168]
          0x0168b2e0 JavaThread "Thread-53" daemon [_thread_blocked, id=167]
          0x00f6d048 JavaThread "Thread-52" daemon [_thread_blocked, id=166]
          0x00f6c480 JavaThread "Thread-51" daemon [_thread_blocked, id=165]
          0x01c2f658 JavaThread "Thread-50" daemon [_thread_blocked, id=164]
          0x046125b8 JavaThread "Thread-49" daemon [_thread_blocked, id=163]
          0x04612410 JavaThread "Thread-48" daemon [_thread_blocked, id=162]
          0x03b42640 JavaThread "Thread-47" daemon [_thread_blocked, id=161]
          0x03b42498 JavaThread "Thread-46" daemon [_thread_blocked, id=160]
          0x00f6bad8 JavaThread "Thread-45" daemon [_thread_blocked, id=159]
          0x01c2e0a0 JavaThread "LDAPConnThread 10.13.245.23:4636" daemon [_thread_blocked, id=152]
          0x00a9d1d8 JavaThread "LDAPConnThread 10.13.245.23:4636" daemon [_thread_blocked, id=150]
          0x04d23c58 JavaThread "LDAPConnThread 10.13.245.23:4636" daemon [_thread_blocked, id=148]
          0x04d23860 JavaThread "LDAPConnThread 10.13.245.23:4636" daemon [_thread_blocked, id=146]
          0x002694d0 JavaThread "LDAPConnThread 10.13.245.23:4636" daemon [_thread_blocked, id=144]
          0x046df8f0 JavaThread "LDAPConnThread 10.13.245.23:4636" daemon [_thread_blocked, id=142]
          0x046df0f8 JavaThread "LDAPConnThread 10.13.245.23:4636" daemon [_thread_blocked, id=140]
          0x046ded00 JavaThread "LDAPConnThread 10.13.245.23:4636" daemon [_thread_blocked, id=138]
          0x046de9a0 JavaThread "LDAPConnThread 10.13.245.23:4636" daemon [_thread_blocked, id=136]
          0x046dd6b8 JavaThread "LDAPConnThread 10.13.245.23:4636" daemon [_thread_blocked, id=134]
          0x046dd470 JavaThread "LDAPConnThread 10.13.245.23:4636" daemon [_thread_blocked, id=132]
          0x01ba9b88 JavaThread "LDAPConnThread 10.13.245.23:4636" daemon [_thread_blocked, id=130]
          0x068549d8 JavaThread "Thread-41" daemon [_thread_blocked, id=115]
          0x00d93bd0 JavaThread "SSLListenThread.Default" [_thread_blocked, id=114]
          0x01a46300 JavaThread "ListenThread.Default" [_thread_blocked, id=113]
          0x019c35f8 JavaThread "Thread-40" daemon [_thread_blocked, id=112]
          0x011ecde0 JavaThread "Timer" daemon [_thread_blocked, id=110]
          0x010c1c58 JavaThread "Timer" daemon [_thread_blocked, id=109]
          0x015b1ed0 JavaThread "Timer" daemon [_thread_blocked, id=108]
          0x01b42b88 JavaThread "Thread-38" daemon [_thread_blocked, id=107]
          0x036344e0 JavaThread "Thread-37" daemon [_thread_blocked, id=106]
          0x01c305a8 JavaThread "Thread-36" daemon [_thread_blocked, id=105]
          0x008817b0 JavaThread "Thread-35" daemon [_thread_blocked, id=104]
          0x0087f390 JavaThread "Timer" daemon [_thread_blocked, id=102]
          0x01fa2c18 JavaThread "Timer" daemon [_thread_blocked, id=101]
          0x01fa1aa0 JavaThread "Thread-33" daemon [_thread_blocked, id=100]
          0x0045f8b8 JavaThread "Thread-32" daemon [_thread_blocked, id=99]
          0x008809a0 JavaThread "Thread-31" daemon [_thread_blocked, id=98]
          0x064028f8 JavaThread "Thread-30" daemon [_thread_blocked, id=97]
          0x00884860 JavaThread "Thread-29" daemon [_thread_blocked, id=96]
          0x03ae9df8 JavaThread "LDAPConnThread 10.13.245.94:4389" daemon [_thread_blocked, id=88]
          0x01be8990 JavaThread "LDAPConnThread 10.13.245.94:4389" daemon [_thread_blocked, id=87]
          0x01e5fb60 JavaThread "LDAPConnThread 10.13.245.94:4389" daemon [_thread_blocked, id=86]
          0x0342d8c0 JavaThread "Thread-23" [_thread_blocked, id=81]
          0x002add78 JavaThread "EventService" daemon [_thread_blocked, id=78]
          0x004dadf8 JavaThread "LDAPConnThread 10.13.245.94:4389" daemon [_thread_blocked, id=77]
          0x025d5c28 JavaThread "LDAPConnThread 10.13.245.94:4389" daemon [_thread_blocked, id=73]
          0x03518650 JavaThread "amSession[9]" daemon [_thread_blocked, id=70]
          0x067322b0 JavaThread "amSession[8]" daemon [_thread_blocked, id=69]
          0x06731d08 JavaThread "amSession[7]" daemon [_thread_blocked, id=68]
          0x01760b98 JavaThread "amSession[6]" daemon [_thread_blocked, id=67]
          0x068d3ee8 JavaThread "amSession[5]" daemon [_thread_blocked, id=66]
          0x068d3d40 JavaThread "amSession[4]" daemon [_thread_blocked, id=65]
          0x062319a8 JavaThread "amSession[3]" daemon [_thread_blocked, id=64]
          0x06231400 JavaThread "amSession[2]" daemon [_thread_blocked, id=63]
          0x00750e70 JavaThread "amSession[1]" daemon [_thread_blocked, id=62]
          0x00750cc8 JavaThread "amSession[0]" daemon [_thread_blocked, id=61]
          0x0175fdf0 JavaThread "Thread-9" daemon [_thread_blocked, id=60]
          0x005b99d0 JavaThread "Thread-8" [_thread_blocked, id=59]
          0x00b584c0 JavaThread "Thread-7" [_thread_blocked, id=58]
          0x0092d260 JavaThread "Thread-6" [_thread_blocked, id=57]
          0x010614d0 JavaThread "ExecuteThread: '0' for queue: 'weblogic.cluster.MulticastManager'" daemon [_thread_blocked, id=56]
          0x007514c8 JavaThread "weblogic.health.CoreHealthMonitor" daemon [_thread_blocked, id=55]
          0x003d8718 JavaThread "Thread-5" [_thread_blocked, id=54]
          0x014cfc40 JavaThread "ExecuteThread: '0' for queue: 'Multicast'" daemon [_thread_blocked, id=53]
          0x01543540 JavaThread "LDAPConnThread gmacds615a:7001" daemon [_thread_blocked, id=52]
          0x00b5a6d0 JavaThread "VDE Transaction Processor Thread" [_thread_blocked, id=50]
          0x0178f950 JavaThread "ExecuteThread: '2' for queue: 'weblogic.admin.RMI'" daemon [_thread_blocked, id=49]
          0x00ac1c38 JavaThread "ExecuteThread: '1' for queue: 'weblogic.admin.RMI'" daemon [_thread_blocked, id=48]
          0x01450880 JavaThread "ExecuteThread: '0' for queue: 'weblogic.admin.RMI'" daemon [_thread_blocked, id=47]
          0x01944898 JavaThread "ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" daemon [_thread_in_native, id=46]
          0x019446f0 JavaThread "ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" daemon [_thread_blocked, id=45]
          0x00320f90 JavaThread "ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" daemon [_thread_blocked, id=44]
          0x006c7a38 JavaThread "weblogic.security.SpinnerRandomSource" daemon [_thread_blocked, id=43]
          0x011e9168 JavaThread "weblogic.time.TimeEventGenerator" daemon [_thread_blocked, id=42]
          0x00342e90 JavaThread "ExecuteThread: '4' for queue: 'weblogic.kernel.System'" daemon [_thread_blocked, id=41]
          0x00ff8f70 JavaThread "ExecuteThread: '3' for queue: 'weblogic.kernel.System'" daemon [_thread_blocked, id=40]
          0x00ff8378 JavaThread "ExecuteThread: '2' for queue: 'weblogic.kernel.System'" daemon [_thread_blocked, id=39]
          0x00ff77e8 JavaThread "ExecuteThread: '1' for queue: 'weblogic.kernel.System'" daemon [_thread_blocked, id=38]
          0x00d518b0 JavaThread "ExecuteThread: '0' for queue: 'weblogic.kernel.System'" daemon [_thread_blocked, id=37]
          0x00d50ce8 JavaThread "ExecuteThread: '24' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=36]
          0x00297ff8 JavaThread "ExecuteThread: '23' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=35]
          0x00297468 JavaThread "ExecuteThread: '22' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=34]
          0x00296980 JavaThread "ExecuteThread: '21' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=33]
          0x01097670 JavaThread "ExecuteThread: '20' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=32]
          0x01096ae0 JavaThread "ExecuteThread: '19' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=31]
          0x01095ff8 JavaThread "ExecuteThread: '18' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=30]
          0x003e28f8 JavaThread "ExecuteThread: '17' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=29]
          0x003e1d00 JavaThread "ExecuteThread: '16' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=28]
          0x01b28c20 JavaThread "ExecuteThread: '15' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=27]
          0x01b28028 JavaThread "ExecuteThread: '14' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=26]
          0x01b27430 JavaThread "ExecuteThread: '13' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=25]
          0x01c528a8 JavaThread "ExecuteThread: '12' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=24]
          0x017ed310 JavaThread "ExecuteThread: '11' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=23]
          0x017ec748 JavaThread "ExecuteThread: '10' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=22]
          0x01a1b518 JavaThread "ExecuteThread: '9' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=21]
          0x018363c8 JavaThread "ExecuteThread: '8' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=20]
          0x01835e20 JavaThread "ExecuteThread: '7' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=19]
          0x01835c78 JavaThread "ExecuteThread: '6' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=18]
          0x01835ad0 JavaThread "ExecuteThread: '5' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=17]
          0x007d4970 JavaThread "ExecuteThread: '4' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=16]
          0x00140b18 JavaThread "ExecuteThread: '3' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=15]
          0x007d47c8 JavaThread "ExecuteThread: '2' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=14]
          0x007d4620 JavaThread "ExecuteThread: '1' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=13]
          0x01836c18 JavaThread "ExecuteThread: '0' for queue: 'weblogic.kernel.Default'" daemon [_thread_blocked, id=12]
          0x00c9c720 JavaThread "Thread-1" daemon [_thread_blocked, id=11]
          =>0x000f9910 JavaThread "CompilerThread1" daemon [_thread_in_native, id=9]
          0x000f8a08 JavaThread "CompilerThread0" daemon [_thread_blocked, id=8]
          0x000f7d98 JavaThread "AdapterThread" daemon [_thread_blocked, id=7]
          0x000f7160 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=6]
          0x000f1948 JavaThread "Finalizer" daemon [_thread_blocked, id=4]
          0x000f07e0 JavaThread "Reference Handler" daemon [_thread_blocked, id=3]
          0x00039cc0 JavaThread "main" [_thread_blocked, id=1]
          • 2. Re: Hotspot JVM Crash during Compiler Optimization of Portal related JSPs
            gimbal2
            user8321790 wrote:
            JDK Version: j2sdk1.4.2_18
            Yeah, you really expect anyone to seriously answer a question about outdated, unsupported and potentially broken software? You are even running an ancient version of Java 1.4.2, the most current one available on the site is 1.4.2_30.
            • 3. Re: Hotspot JVM Crash during Compiler Optimization of Portal related JSPs
              jschellSomeoneStoleMyAlias
              Portal Server: SUN Portal Server 6.1 (Portal Server installed on top of Weblogic Server)
              JDK Version: j2sdk1.4.2_18
              I think "SUN Portal Server 6.1" is a product. And if so the jdk version is probably what one is supposed to run it on.
              If if is a product and the still have a service contract then one uses that to seek help.

              That said...
              400000 /apps/oblix/netpoint/AccessServerSDK/oblix/lib/libxmlengine.so
              That is JNI code. JNI code can cause crashes. If that code is part of the product then the service contract might cover it. If not then it won't.