1 Reply Latest reply: May 18, 2009 5:31 PM by 843829 RSS

    /tmp filling with hs_err_pidXXXX.logs

    843829
      Hello,

      I am finding my Solaris 10 /tmp directory is rapidly filling up with error logs in the form of hs_err_pidXXXX.log and is being generated every few minutes. I have heard somewhere (I do not recall) that disableing the service `svcadm disable svc:/system/webconsole:console` and rebooting the system will help - but these darn error logs just keep piling up. I've seen listings with differing topics pointing to error code that to me all looks the same, but nobody has a suggested workaround or a fix.

      I am lost - can anybody help?

      Steve K


      Error in total is as follows:

      #
      # An unexpected error has been detected by HotSpot Virtual Machine:
      #
      # SIGSEGV (0xb) at pc=0xfe9b4124, pid=233, tid=1
      #
      # Java VM: Java HotSpot(TM) Server VM (1.5.0_18-b02 mixed mode)
      # Problematic frame:
      # V [libjvm.so+0x1b4124]
      #

      --------------- T H R E A D ---------------

      Current thread (0x000387d8): JavaThread "main" [_thread_in_vm, id=1]

      siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0xc6deda9c

      Registers:
      O0=0xfef6847e O1=0xffffffff O2=0xbaadc043 O3=0x00006c00
      O4=0xfe9225c4 O5=0x00620f64 O6=0xffbfb6e8 O7=0xfe9bd154
      G1=0xfea86948 G2=0x00000040 G3=0x636f6d2e G4=0x00000008
      G5=0x00000034 G6=0x00000000 G7=0xff362a00 Y=0x00000000
      PC=0xfe9b4124 nPC=0xfe9b4128


      Top of Stack: (sp=0xffbfb6e8)
      0xffbfb6e8: fefde000 ff02a278 baadc043 f8c00040
      0xffbfb6f8: 00000048 0000004b baadc044 00001dbc
      0xffbfb708: c6deda9c 73756e2e c6deda9c ff0457a8
      0xffbfb718: 00008490 0000000c ffbfb748 fea86948
      0xffbfb728: 00000050 00000041 00000000 f6885078
      0xffbfb738: 00000000 00000000 00000000 00000000
      0xffbfb748: fefde000 000387d8 ff362a00 ff033030
      0xffbfb758: 00009628 000389a0 00009400 00007400

      Instructions: (pc=0xfe9b4124)
      0xfe9b4114: 9d e3 bf a0 b2 90 00 19 04 40 00 47 b4 10 00 18
      0xfe9b4124: e2 16 a0 00 b0 10 20 00 92 10 20 00 a0 10 00 1a

      Stack: [0xffb80000,0xffc00000), sp=0xffbfb6e8, free space=493k
      Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
      V [libjvm.so+0x1b4124]
      V [libjvm.so+0x286950]
      C [libwebconsole_services.so+0x644] Java_com_sun_management_services_common_CCServiceLibrary_doSyslog+0x34
      j com.sun.management.services.common.CCServiceLibrary.doSyslog([B[BIII)V+37916
      j com.sun.management.services.common.CCServiceLibrary.doSyslog([B[BIII)V+0
      j com.sun.management.services.common.CCServiceLibrary.writeSyslog(Ljava/lang/String;Ljava/lang/String;I)V+40

      <snip>

      Launcher Type: SUN_STANDARD

      Environment Variables:
      PATH=/usr/sbin:/usr/bin
      LD_LIBRARY_PATH=/usr/jdk/instances/jdk1.5.0/jre/lib/sparc/server:/usr/jdk/instances/jdk1.5.0/jre/lib/sparc:/usr/jdk/instances/jdk1.5.
      0/jre/../lib/sparc:/usr/lib/webconsole

      Signal Handlers:
      SIGSEGV: [libjvm.so+0x70d970], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
      SIGBUS: [libjvm.so+0x70d970], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
      SIGFPE: [libjvm.so+0x2731e4], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
      SIGPIPE: [libjvm.so+0x2731e4], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
      SIGILL: [libjvm.so+0x2731e4], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
      SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
      SIGUSR2: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
      SIGHUP: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
      SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
      SIGQUIT: [libjvm.so+0x67ef14], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
      SIGTERM: [libjvm.so+0x67ef14], sa_mask[0]=0xffbffeff, sa_flags=0x00000004


      --------------- S Y S T E M ---------------

      OS: Solaris 10 5/08 s10s_u5wos_10 SPARC
      Copyright 2008 Sun Microsystems, Inc. All Rights Reserved.
      Use is subject to license terms.
      Assembled 24 March 2008

      uname:SunOS 5.10 Generic_139555-08 sun4v (T2 libthread)
      rlimit: STACK 8192k, CORE infinity, NOFILE 12000, AS infinity
      load average:3.92 3.71 3.71

      CPU:total 32 has_v8, has_v9, has_vis1, has_vis2, is_ultra3, is_sun4v, is_niagara1

      Memory: 8k page, physical 33423360k(29884240k free)

      vm_info: Java HotSpot(TM) Server VM (1.5.0_18-b02) for solaris-sparc, built on Feb 25 2009 02:31:38 by unknown with unknown Workshop
      :0x550
        • 1. Re: /tmp filling with hs_err_pidXXXX.logs
          843829
          Well, your system is experiencing segmentation violations (i.e. memory accessing errors) probably due to some errant pointer in some native code, or due to some soft/corrupt memory in your system.

          The errant pointer problem is the most likely but I have seen cases where memory was getting corrupted in some embedded systems and caused similar sort of problems. However, those faults appeared in the pthreads library and not in the VM itself.

          Your memory access problems are in libjvm which is very worrying. This is usually solid as a rock. If it wasn't Java would be unreliable.

          If a later version of the JVM is available I would upgrade to that and see if the problem persists.

          The fact that your system is rapidly filling up with these error logs would make me think about running some kind of extensive memory check on your system. When I have seen soft/corrupted memory on my embedded systems it has been intermittent. The fact that you see the problem so often would tend to make it more likely that you have some kind of memory corruption going on. So, before going any further, I would run that system memory test. If the memory comes up clean it mean that something [awful] is happening in your JVM.

          Regards,

          Gordon J Milne