8 Replies Latest reply: Dec 4, 2009 1:59 PM by user4994457 RSS

    inetd service/program crashes with core dump in Solaris 8 zone/container

    807559
      I have developed a service in C that is launched from inetd when something comes on a specific port.
      When a connection is opened to the port a core dump is created in the same directory where the executable file is located.
      If you run the same service program from the command line everything is working perfect.
      This is running in a Solaris 8 zone/container on a Solaris 10 machine.
      Everything is set correctly in /etc/inetd.conf and in /etc/services.
      I have even stripped down the program to a hello world program that is just printing a string to the screen and it is still crashing with a core dump.

      # ldd test_srv
      /usr/lib/secure/s8_preload.so.1
      libc.so.1 => /usr/lib/libc.so.1
      libdl.so.1 => /usr/lib/libdl.so.1
      /usr/platform/SUNW,Sun-Fire-480R/lib/libc_psr.so.1

      The same service is running on a Linux machine and on a Solaris 10 machine without zones/containers without any problems.
      Can you please help me figure out what am I missing. Is there something specific with zones/containers that should be set / configured?
      Do I have to set some specific env. variables to work in a Solaris 8 zone/container environment or is it something very simple that I'm missing?
        • 1. Re: inetd service/program crashes with core dump in Solaris 8 zone/container
          user4994457
          Can you get a stack trace from the core? That might point to where the crash is coming from.

          --
          Darren
          • 2. Re: inetd service/program crashes with core dump in Solaris 8 zone/container
            807559
            This is the stack trace of the core. It is strange that it is showing the global zone's version - SunOS 5.10 Generic_137137-09.
            There is probably some kind of mismatch between libraries or links. mdb prints out global zone name SunOS 5.10 Generic_137137-09 while the real version is SunOS 5.8 Generic_Virtual sun4u

            # uname -a
            SunOS xxxxxxxx 5.8 Generic_Virtual sun4u sparc SUNW,Sun-Fire-480R

            mdb core
            mdb: warning: core file is from SunOS 5.10 Generic_137137-09; shared text mappings may not match installed libraries
            Loading modules: [ ]

            ::status
            debugging core file of xxxxxxx (32-bit) from XXXXXXX
            executable file: /xxxx/xxxxx/xxxxxx
            initial argv:
            status: SIGSEGV (Segmentation Fault)
            ::stack
            0xff232e2c(ff3dbc6c, 0, 0, 0, 73756c6f, 2e000000)
            0xff3d683c(ffbffc80, ffbffc7c, ff3ee818, 1, ff3eff3c, ff3ee000)
            0xff3db41c(0, 0, 0, 0, 0, 0)
            0(0, 0, 0, 0, 0, 0)
            > ff3d683c::dis
            0xff3d6814:                     call      +0x17b74      <0xff3ee388>
            0xff3d6818:                     mov       0xd7, %o1
            0xff3d681c:                     orcc      %g0, %o0, %g0
            0xff3d6820:                     bne       +0x124        <0xff3d6944>
            0xff3d6824:                     mov       1, %i4
            0xff3d6828:                     ld        [%fp - 0x80], %l3
            0xff3d682c:                     st        %l3, [%i2 + 8]
            0xff3d6830:                     ba        +0x114        <0xff3d6944>
            0xff3d6834:                     mov       1, %i4
            0xff3d6838:                     ld        [%i5 + 0x130], %o0
            0xff3d683c:                     call      +0x17bd0      <0xff3ee40c>
            0xff3d6840:                     ld        [%i1], %o1
            0xff3d6844:                     orcc      %g0, %o0, %g0
            0xff3d6848:                     bne       +0x68         <0xff3d68b0>
            0xff3d684c:                     nop
            0xff3d6850:                     ld        [%i1 + 4], %l4
            0xff3d6854:                     cmp       %l4, 0
            0xff3d6858:                     be        +0xec         <0xff3d6944>
            0xff3d685c:                     mov       1, %i4
            0xff3d6860:                     ld        [%i1 + 8], %l5
            0xff3d6864:                     cmp       %l5, 0
            
            
            0xff232e04:                     bne       +0xc          <0xff232e10>
            0xff232e08:                     andcc     %i4, %l2, %g0
            0xff232e0c:                     ba,a      +0x80         <0xff232e8c>
            0xff232e10:                     bne       +0xc          <0xff232e1c>
            0xff232e14:                     andcc     %i4, 0xff, %g0
            0xff232e18:                     ba,a      +0x74         <0xff232e8c>
            0xff232e1c:                     bne       -0x78         <0xff232da4>
            0xff232e20:                     sll       %i3, 0x10, %i5
            0xff232e24:                     ba,a      +0x68         <0xff232e8c>
            0xff232e28:                     sub       %i0, %i1, %i0
            0xff232e2c:                     ld        [%i1], %i5
            0xff232e30:                     ld        [%i1 + %i0], %i4
            0xff232e34:                     cmp       %i4, %i5
            0xff232e38:                     add       %i1, 4, %i1
            0xff232e3c:                     be        +0xc          <0xff232e48>
            0xff232e40:                     add       %i4, %l6, %l3
            0xff232e44:                     ba,a      +0x50         <0xff232e94>
            0xff232e48:                     xor       %l3, %i4, %l3
            0xff232e4c:                     and       %l3, %l7, %l3
            0xff232e50:                     cmp       %l3, %l7
            0xff232e54:                     be,a      -0x24         <0xff232e30>
            • 3. Re: inetd service/program crashes with core dump in Solaris 8 zone/container
              Robert Cohen
              Are you compiling the program in the solaris 8 zone, or under solaris 10.
              Solaris programs arent backwards compatible to older OS versions, so you should compile under solaris 8.

              Does a simple non networked hello world work?
              • 4. Re: inetd service/program crashes with core dump in Solaris 8 zone/container
                807559
                Yes, this is compiled in a Solaris 8 zone.
                The hello world program and other programs are working fine if run from the command line.
                The crashing of the programs only happens when they are run from inetd.
                • 5. Re: inetd service/program crashes with core dump in Solaris 8 zone/container
                  807559
                  Could you please examine the truss log and advice what the problem is and how to fix it?
                  (some lines deleted)
                  bash-2.03# truss -f -p 18361 #### /usr/sbin/inetd -s -t &
                  18361:  poll(0xFFBFF528, 53, -1)        (sleeping...)
                  18361:  poll(0xFFBFF528, 53, -1)                        = 1
                  
                  18361:  accept(63, 0xFFBFF870, 0xFFBFF914, 1)           = 3
                  
                  18361:  sigprocmask(SIG_BLOCK, 0xFFBFF5F0, 0xFFBFF600)  = 0
                  18361:  lwp_sigtimedwait(0xFFBFF600, 0xFFBFF578, 0x00000010) = 0
                  18361:  lwp_sigtimedwait(0xFFBFF568, 0xFFBFF728, 0x00000010) = 0
                  18361:  fork()                                          = 1921
                  1921:   fork()          (returning as child ...)        = 18361
                  1921:   sigprocmask(0, 0x00000000, 0xFFBFF600)          = 0
                  18361:  sigprocmask(0, 0x00000000, 0xFFBFF600)          = 0
                  1921:   lwp_sigtimedwait(0xFFBFF600, 0xFFBFF578, 0x00000010) = 0
                  
                  18361:  sigprocmask(SIG_SETMASK, 0xFFBFF5F0, 0xFFBFF600) = 0
                  
                  18361:  close(3)                                        = 0
                  18361:  sigprocmask(0, 0x00000000, 0xFFBFF600)          = 0
                  1921:   lwp_sigtimedwait(0xFFBFF668, 0xFFBFF528, 0x00000020) = 0
                  
                  1921:   sigaction(SIGHUP, 0xFFBFF528, 0xFFBFF500)       = 0
                  18361:  lwp_sigtimedwait(0xFFBFF568, 0xFFBFF5F0, 0x00000010) = 0
                  1921:   lwp_sigtimedwait(0xFFBFF508, 0xFFBFF458, 0x00000010) = 0
                  18361:  sigprocmask(SIG_SETMASK, 0xFFBFF5F0, 0xFFBFF600) = 0
                  
                  1921:   sigprocmask(SIG_SETMASK, 0xFFBFF5F0, 0xFFBFF600) = 0
                  1921:   lwp_sigtimedwait(0xFFBFF600, 0xFFBFF578, 0x00000010) = 0
                  1921:   lwp_sigtimedwait(0xFFBFF568, 0xFFBFF728, 0x00000010) = 0
                  1921:   fcntl(3, F_DUP2FD, 0x00000000)                  = 0
                  1921:   close(3)                                        = 0
                  1921:   fcntl(0, F_DUP2FD, 0x00000001)                  = 1
                  1921:   fcntl(0, F_DUP2FD, 0x00000002)                  = 2
                  1921:   open64("/etc/.name_service_door", O_RDONLY)     = 3
                  1921:   fcntl(3, F_SETFD, 0x00000001)                   = 0
                  1921:   door_info(3, 0xFF0C2748)                        = 0
                  1921:   door_call(3, 0xFFBFF278)                        = 0
                  1921:   close(67)                                       Err#9 EBADF
                  1921:   close(66)                                       Err#9 EBADF
                  1921:   close(65)                                       Err#9 EBADF
                  1921:   close(64)                                       Err#9 EBADF
                  1921:   close(63)                                       = 0
                  1921:   close(62)                                       = 0
                  
                  1921:   close(12)                                       = 0
                  1921:   close(11)                                       = 0
                  1921:   close(10)                                       Err#9 EBADF
                  1921:   close(9)                                        Err#9 EBADF
                  1921:   close(8)                                        Err#9 EBADF
                  1921:   close(7)                                        Err#9 EBADF
                  1921:   close(6)                                        Err#9 EBADF
                  1921:   close(5)                                        Err#9 EBADF
                  1921:   close(4)                                        Err#9 EBADF
                  1921:   setrlimit(RLIMIT_NOFILE, 0xFFBFFD20)            = 0
                  1921:   xenix(398872, 0xFFBFF5E4, 0x00000040)           = 38
                  1921:   execve("/tmp/srv/t_srv", 0x0008B5FC, 0xFFBFFDA0)  argc = 0
                  1921:   getuid()                                        = 0 [0]
                  1921:   resolvepath("/usr/lib/ld.so.1", "/usr/lib/ld.so.1", 1023) = 16
                  1921:   open("/var/ld/ld.config", O_RDONLY)             = 3
                  1921:   fstat(3, 0xFFBFF5E8)                            = 0
                  1921:   mmap(0x00000000, 148, PROT_READ, MAP_SHARED, 3, 0) = 0xFF3E0000
                  1921:   close(3)                                        = 0
                  1921:   stat("/usr/lib/libc.so.1", 0xFFBFF648)          = 0
                  1921:   resolvepath("/usr/lib/libc.so.1", "/usr/lib/libc.so.1", 1023) = 18
                  1921:   open("/usr/lib/libc.so.1", O_RDONLY)            = 3
                  1921:   mmap(0x00000000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0xFF340000
                  1921:   mmap(0x00000000, 802816, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON, -1, 0) = 0xFF200000
                  1921:   mmap(0xFF200000, 703520, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF200000
                  1921:   mmap(0xFF2BC000, 24772, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 704512) = 0xFF2BC000
                  1921:   munmap(0xFF2AC000, 65536)                       = 0
                  1921:   memcntl(0xFF200000, 113528, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
                  1921:   close(3)                                        = 0
                  1921:   stat("/usr/lib/libdl.so.1", 0xFFBFF648)         = 0
                  1921:   resolvepath("/usr/lib/libdl.so.1", "/usr/lib/libdl.so.1", 1023) = 19
                  1921:   open("/usr/lib/libdl.so.1", O_RDONLY)           = 3
                  1921:   mmap(0xFF340000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF340000
                  1921:   mmap(0x00000000, 8192, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON, -1, 0) = 0xFF330000
                  1921:   mmap(0xFF330000, 2638, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF330000
                  1921:   close(3)                                        = 0
                  1921:   stat("/usr/platform/SUNW,Sun-Fire-480R/lib/libc_psr.so.1", 0xFFBFF368) = 0
                  1921:   resolvepath("/usr/platform/SUNW,Sun-Fire-480R/lib/libc_psr.so.1", "/usr/platform/sun4u-us3/lib/libc_psr.so.1", 1023) = 41
                  1921:   open("/usr/platform/SUNW,Sun-Fire-480R/lib/libc_psr.so.1", O_RDONLY) = 3
                  1921:   mmap(0xFF340000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFF340000
                  1921:   close(3)                                        = 0
                  1921:   mmap(0x00000000, 8192, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFF320000
                  1921:   dup(0)                                          = 3
                  1921:   llseek(0, 0, SEEK_CUR)                          Err#29 ESPIPE
                  1921:   close(0)                                        = 0
                  1921:   fcntl(3, F_DUP2FD, 0x00000000)                  = 0
                  1921:   close(3)                                        = 0
                  1921:   dup(1)                                          = 3
                  1921:   close(1)                                        = 0
                  1921:   fcntl(3, F_DUP2FD, 0x00000001)                  = 1
                  1921:   close(3)                                        = 0
                  1921:   dup(2)                                          = 3
                  1921:   close(2)                                        = 0
                  1921:   fcntl(3, F_DUP2FD, 0x00000002)                  = 2
                  1921:   close(3)                                        = 0
                  1921:   sys#177(0x00000080, 0xFFBFFB7C, 0xFF3F0518, 0x00000000, 0xFF3C2EF8, 0xFF2C0284) = 0x00000000 [0xFFBFFB7C]
                  1921:   sys#227(0x00000006, 0x00000000, 0x0001ADF0, 0xFF3F0518, 0xFF3C3C18, 0xFF3C2670) = 0x0000000C [0x00000000]
                  1921:   sys#227(0x00000002, 0x0000000C, 0x0000000E, 0xFFBFFCAE, 0x00000002, 0xFF3C2670) = 0x00000002 [0x00000000]
                  1921:       Incurred fault #6, FLTBOUNDS  %pc = 0xFF232E2C
                  1921:         siginfo: SIGSEGV SEGV_MAPERR addr=0x00000000
                  1921:       Received signal #11, SIGSEGV [default]
                  1921:         siginfo: SIGSEGV SEGV_MAPERR addr=0x00000000
                  1921:           *** process killed ***
                  18361:      Received signal #18, SIGCLD, in poll() [caught]
                  18361:        siginfo: SIGCLD CLD_DUMPED pid=1921 status=0x000B
                  18361:  poll(0xFFBFF528, 53, -1)                        Err#4 EINTR
                  18361:  lwp_sigtimedwait(0xFFBFF218, 0xFFBFF140, 0x00000010) = 0
                  18361:  lwp_sigtimedwait(0xFFBFF130, 0xFFBFF218, 0x00000010) = 0
                  18361:  sigprocmask(0, 0x00000000, 0xFFBFEF28)          = 0
                  
                  18361:  poll(0xFFBFF528, 53, -1)        (sleeping...)
                  Thank you in advance
                  • 6. Re: inetd service/program crashes with core dump in Solaris 8 zone/container
                    user4994457
                    truss isn't going to show you much about reasons for a segfault.

                    In this case, the best thing to do is to use a source debugger and see what is going on with your program. Compile it with -g and open it in gdb or dbx or something.

                    # dbx /tmp/srv/t_srv core

                    That should give you more information about where the program died. It appears to be a "simple" segfault. It may be as simple as a code path that is invoked when running via inetd, but since it's in your code, we can't really tell. It could be in a library or system call, but the debugger would help identify.

                    --
                    Darren
                    • 7. Re: inetd service/program crashes with core dump in Solaris 8 zone/container
                      807559
                      The program runs fine when executed from the command line. It crashes only when lunched from inetd.
                      inetd is causing the problem somehow.
                      Could you please advice what is the difference between the way inetd executes a program compared to running it from the command line?
                      • 8. Re: inetd service/program crashes with core dump in Solaris 8 zone/container
                        user4994457
                        Environment will be different, no controlling terminal will be most of the differences.

                        Can you get a source debugger to show where the core is exiting?

                        --
                        Darren