3 Replies Latest reply on Jan 12, 2012 1:01 AM by jschellSomeoneStoleMyAlias

    unexpected sigsegv error


      I see a lot of people experiencing similar sigsegv errors but I don't see many "answered" threads. There haven't been any changes to our application or middle tier recently. However, in the last three weeks or so, we've been getting a lot of core dumps. The consensus solution seems to be an upgrade of the JVM, but that's hard to recommend without completely understanding the cause of the error. Any thoughts, questions, or comments would be most appreciated.

      Here's the log -

      # An unexpected error has been detected by HotSpot Virtual Machine:
      # SIGSEGV (0xb) at pc=0xfe8f4cc0, pid=14721, tid=12
      # Java VM: Java HotSpot(TM) Server VM (1.5.0_06-b05 mixed mode)
      # Problematic frame:
      # V [libjvm.so+0xf4cc0]

      --------------- T H R E A D ---------------

      Current thread (0x0015acc0): JavaThread "CompilerThread1" daemon [_thread_in_native, id=12]

      siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x00000004

      O0=0x00003b54 O1=0x00000000 O2=0x00000000 O3=0x00000000
      O4=0x00000000 O5=0x00000ed5 O6=0xb017e828 O7=0x00002000
      G1=0x00003b54 G2=0x00f2a350 G3=0x0089f3b0 G4=0x00000000
      G5=0x003140b8 G6=0x00000000 G7=0xff271600 Y=0x00000000
      PC=0xfe8f4cc0 nPC=0xfe8f4cc4

      Top of Stack: (sp=0xb017e828)
      0xb017e828: b017ea84 00b3b058 003140b8 00000000
      0xb017e838: 00f2a350 00000000 00002000 003140b8
      0xb017e848: b017ea74 0123a9f0 00a51890 00000000
      0xb017e858: 003140b8 00f2a350 b017e888 fea06600
      0xb017e868: feffd8b4 ffffffff feffe7e4 00000001
      0xb017e878: 00000000 feff5d80 feff22b8 fe90f56c
      0xb017e888: 00000000 ffffffff 00000001 b017ea84
      0xb017e898: b017eaac 00a51890 00000001 00000004

      Instructions: (pc=0xfe8f4cc0)
      0xfe8f4cb0: f6 02 20 1c c6 06 20 58 89 2e e0 02 f6 00 c0 04
      0xfe8f4cc0: d2 06 e0 04 d0 02 60 00 80 a2 20 00 32 40 00 11

      Stack: [0xb0100000,0xb0180000), sp=0xb017e828, free space=506k
      Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
      V [libjvm.so+0xf4cc0]
      V [libjvm.so+0x206608]
      V [libjvm.so+0x2047ec]
      V [libjvm.so+0x27fdac]
      V [libjvm.so+0x282748]
      V [libjvm.so+0x278700]
      V [libjvm.so+0x2793bc]
      V [libjvm.so+0x335f28]
      V [libjvm.so+0x2de2a4]
      V [libjvm.so+0x664248]

      Current CompileTask:
      opto:1781 oracle.sql.NUMBER.toBytes(Ljava/lang/String;I)[B (1034 bytes)

      ---------------  P R O C E S S  ---------------

      Java Threads: ( => current thread )
        0x015f1830 JavaThread "SystemThreadGroup-63" [_thread_blocked, id=111]
      0x01210ce0 JavaThread "HTTPThreadGroup-9" [_thread_blocked, id=96]
      0x012476f8 JavaThread "SocketTimeout" daemon [_thread_blocked, id=55]
      0x00947530 JavaThread "Timer-6" daemon [_thread_blocked, id=53]
      0x005dde50 JavaThread "HTTPThreadGroup-8" [_thread_blocked, id=51]
      0x003b8648 JavaThread "HTTPThreadGroup-7" [_thread_blocked, id=50]
      0x00665d80 JavaThread "AJPRequestHandler-HTTPThreadGroup-6" [_thread_in_Java, id=49]
      0x00294310 JavaThread "HTTPThreadGroup-5" [_thread_blocked, id=48]
      0x008d24e0 JavaThread "HTTPThreadGroup-4" [_thread_blocked, id=47]
      0x00037810 JavaThread "DestroyJavaVM" [_thread_blocked, id=1]
      0x00fd76d8 JavaThread "TaskManager" [_thread_blocked, id=46]
      0x01314198 JavaThread "OC4JMonitorThread" daemon [_thread_blocked, id=45]
      0x00d97f10 JavaThread "Thread-18" daemon [_thread_blocked, id=44]
      0x007bfb10 JavaThread "Thread-17" daemon [_thread_in_native, id=43]
      0x00bea468 JavaThread "SystemThreadGroup-6" [_thread_in_native, id=42]
      0x00beedd8 JavaThread "SystemThreadGroup-5" [_thread_in_native, id=41]
      0x01214878 JavaThread "SystemThreadGroup-4" [_thread_blocked, id=40]
      0x009a6d00 JavaThread "Timer-5" [_thread_blocked, id=39]
      0x00989040 JavaThread "Timer-4" [_thread_blocked, id=38]
      0x010308d8 JavaThread "Timer-1" daemon [_thread_blocked, id=35]
      0x01197530 JavaThread "WorkExecutorWorkerThread-1" daemon [_thread_blocked, id=34]
      0x00b4c428 JavaThread "Thread-11" daemon [_thread_blocked, id=33]
      0x0119b6b0 JavaThread "Thread-9" daemon [_thread_blocked, id=31]
      0x00b50920 JavaThread "Timer-0" [_thread_blocked, id=29]
      0x00b9d178 JavaThread "RMIServer [] count:1" [_thread_in_native, id=28]
      0x00b9dcc8 JavaThread "RMIServer [] count:1" [_thread_in_native, id=27]
      0x007b7e20 JavaThread "JMSServer[vs10.hlth.gov.bc.ca:12612]" [_thread_in_native, id=26]
      0x003b3338 JavaThread "WsMgmtWorkScheduler" daemon [_thread_blocked, id=25]
      0x003b2a10 JavaThread "WsMgmtWorkScheduler" daemon [_thread_blocked, id=24]
      0x00112f88 JavaThread "Scheduler ServiceThread" [_thread_blocked, id=23]
      0x00a80470 JavaThread "Timer ServiceThread" [_thread_blocked, id=22]
      0x00947d38 JavaThread "Event ServiceThread" [_thread_blocked, id=21]
      0x00a88760 JavaThread "LogFlusher" daemon [_thread_blocked, id=19]
      0x00a8be48 JavaThread "LogFlusher" daemon [_thread_blocked, id=18]
      0x002eff80 JavaThread "LogFlusher" daemon [_thread_blocked, id=17]
      0x00212fd0 JavaThread "LogFlusher" daemon [_thread_blocked, id=16]
      0x0015bd70 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=13]
      =>0x0015acc0 JavaThread "CompilerThread1" daemon [_thread_in_native, id=12]
      0x00159e50 JavaThread "CompilerThread0" daemon [_thread_blocked, id=11]
      0x00158fd0 JavaThread "AdapterThread" daemon [_thread_blocked, id=10]
      0x00158268 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=9]
      0x0014bb10 JavaThread "Finalizer" daemon [_thread_blocked, id=8]
      0x0014adc8 JavaThread "Reference Handler" daemon [_thread_blocked, id=7]

      Other Threads:
      0x00148ce8 VMThread [id=6]
      0x0015cf58 WatcherThread [id=14]

      VM state:not at safepoint (normal execution)

      VM Mutex/Monitor currently owned by a thread: None

      PSYoungGen total 95232K, used 42890K [0xe3400000, 0xea400000, 0xf8c00000)
      eden space 84224K, 41% used [0xe3400000,0xe56883d8,0xe8640000)
      from space 11008K, 68% used [0xe9940000,0xea09a538,0xea400000)
      to space 14592K, 0% used [0xe8780000,0xe8780000,0xe95c0000)
      PSOldGen total 352256K, used 51492K [0xb8800000, 0xce000000, 0xe3400000)
      object space 352256K, 14% used [0xb8800000,0xbba490e8,0xce000000)
      PSPermGen total 45056K, used 43733K [0xb0800000, 0xb3400000, 0xb8800000)
      object space 45056K, 97% used [0xb0800000,0xb32b5408,0xb3400000)

      Dynamic libraries:
      0x00010000      /usr6/app/oracle/product/10.1.3/app_3/jdk/bin/java
      0xff370000      /usr/lib/libthread.so.1
      0xff360000      /usr/lib/libdl.so.1
      0xff280000      /usr/lib/libc.so.1
      0xff3a0000      /usr/platform/SUNW,Sun-Fire-880/lib/libc_psr.so.1
      0xfe800000      /usr6/app/oracle/product/10.1.3/app_3/jdk/jre/lib/sparc/server/libjvm.so
      0xff230000      /usr/lib/libsocket.so.1
      0xff210000      /usr/lib/libsched.so.1
      0xff1e0000      /usr/lib/libCrun.so.1
      0xff190000      /usr/lib/libm.so.1
      0xff080000      /usr/lib/libnsl.so.1
      0xff160000      /usr/lib/libmp.so.2
      0xff050000      /usr6/app/oracle/product/10.1.3/app_3/jdk/jre/lib/sparc/native_threads/libhpi.so
      0xfe7c0000      /usr6/app/oracle/product/10.1.3/app_3/jdk/jre/lib/sparc/libverify.so
      0xfe780000      /usr6/app/oracle/product/10.1.3/app_3/jdk/jre/lib/sparc/libjava.so
      0xfe750000      /usr6/app/oracle/product/10.1.3/app_3/jdk/jre/lib/sparc/libzip.so
      0xae250000      /usr6/app/oracle/product/10.1.3/app_3/jdk/jre/lib/sparc/libnet.so
      0xac7e0000      /usr6/app/oracle/product/10.1.3/app_3/jdk/jre/lib/sparc/libnio.so
      0xac7c0000      /usr/lib/librt.so.1
      0xac7a0000      /usr/lib/libaio.so.1
      0xac6e0000      /usr/lib/libmd5.so.1
      0xac6c0000      /usr/lib/libsendfile.so.1
      0xaac80000      /usr6/app/oracle/product/10.1.3/app_3/jdk/jre/lib/sparc/libawt.so
      0xaab00000      /usr6/app/oracle/product/10.1.3/app_3/jdk/jre/lib/sparc/libmlib_image.so
      0xac5e0000      /usr6/app/oracle/product/10.1.3/app_3/jdk/jre/lib/sparc/headless/libmawt.so

      VM Arguments:
      jvm_args: -XX:MaxPermSize=128M -Xms512M -Xmx1024M -XX:AppendRatio=3 -Djava.security.policy=/usr6/app/oracle/product/10.1.3/app_3/j2ee/prod/config/java2.policy -Djava.awt.headless=true -Dhttp.webdir.enable=false -Doracle.ons.oraclehome=/usr6/app/oracle/product/10.1.3/app_3 -Doracle.home=/usr6/app/oracle/product/10.1.3/app_3 -Doracle.ons.oracleconfighome=/usr6/app/oracle/product/10.1.3/app_3 -Doracle.ons.clustername=default -Doracle.ons.instancename=APPPUB1013.vs10.hlth.gov.bc.ca -Dopmn.compatible=904 -Doracle.ons.indexid=prod.default_group.1 -Doracle.ons.numprocs=1 -Doracle.ons.uid=1219368890 -Doracle.oc4j.groupname=default_group -Doracle.oc4j.instancename=prod -Doracle.oc4j.islandname=default_group -Doracle.opmn.routingid=g_rt_id -DOPMN=true
      java_command: oc4j.jar -config /usr6/app/oracle/product/10.1.3/app_3/j2ee/prod/config/server.xml -properties -ports default-web-site:ajp:12512,rmi:12412,rmis:12708,jms:12612
      Launcher Type: SUN_STANDARD

      Environment Variables:

      Signal Handlers:
      SIGSEGV: [libjvm.so+0x6f0ca8], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
      SIGBUS: [libjvm.so+0x6f0ca8], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
      SIGFPE: [libjvm.so+0x275d94], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
      SIGPIPE: [libjvm.so+0x275d94], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
      SIGILL: [libjvm.so+0x275d94], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
      SIGUSR1: [libjvm.so+0x666744], sa_mask[0]=0x00000000, sa_flags=0x00000008
      SIGUSR2: [libjvm.so+0x275d94], sa_mask[0]=0x7fbffeff, sa_flags=0x0000000c
      SIGHUP: [libjvm.so+0x665424], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
      SIGINT: [libjvm.so+0x665424], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
      SIGQUIT: [libjvm.so+0x665424], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004
      SIGTERM: [libjvm.so+0x665424], sa_mask[0]=0x7fbffeff, sa_flags=0x00000004

      --------------- S Y S T E M ---------------

      OS: Solaris 9 12/03 s9s_u5wos_08b SPARC
      Copyright 2003 Sun Microsystems, Inc. All Rights Reserved.
      Use is subject to license terms.
      Assembled 21 November 2003

      uname:SunOS 5.9 Generic_122300-39 sun4u (T2 libthread)
      rlimit: STACK 8192k, CORE infinity, NOFILE 65546, AS infinity
      load average:0.36 0.29 0.32

      CPU:total 4 has_v8, has_v9, has_vis1, has_vis2, is_ultra3

      Memory: 8k page, physical 12582912k(7690576k free)

      vm_info: Java HotSpot(TM) Server VM (1.5.0_06-b05) for solaris-sparc, built on Nov 10 2005 11:24:16 by unknown with unknown Workshop:0x550
        • 1. Re: unexpected sigsegv error
          Has anything changed hardware-wise?
          • 2. Re: unexpected sigsegv error
            Gerhardt wrote:

            The consensus solution seems to be an upgrade of the JVM
            That makes no sense at all based on what you say. Unless the VM that had been stable decided on its own to misbehave from one moment to the next - unlikely, pieces of software don't have emotions that can flare up.

            A JVM is like any other application a visitor in your system and dependent on many external modules installed in it (see under "dynamic libraries" for example, a good number of them are loaded from /usr/lib). A possible cause can be that an update of one of them is causing you problems now because a flaw has been introduced. The fact that the JVM crashes is then only a ripple, not the root of the problem. The root is that you have a broken module.

            That is one possible cause, another is as the previous poster suggested; you are having trouble with a piece of hardware in your system. There are probably more reasons that I am not thinking of.
            • 3. Re: unexpected sigsegv error
              Gerhardt wrote:
              I see a lot of people experiencing similar sigsegv errors but I don't see many "answered" threads.
              No idea what you mean by that statement. There are plenty of answers explaining why one might get that. There is NOT one answer.
              There haven't been any changes to our application or middle tier recently
              Doesn't necessarily mean anything. If the server is experiencing different traffic then that can cause a different execution path which causes an error that previously have no impact to cause a crash.

              Could also indicate a hardware failure of some odd type.