2 Replies Latest reply on Dec 8, 2010 10:12 AM by 821871

    SSH segmentation fault, but only while running as root.

      Hello all,

      I'm trying to solve a weird issue I discovered while debugging some other problem. I have a server ( Fire X4500 running 5.10 Generic_139556-08 ) from which I'm trying to connect using SSH to a series of Acme Packet Session Border Controllers (all running the same software release). This works fine when I do it as an unprivileged user, however when I do the same as user root ssh segfaults (see truss output below).
      This problem only occurs when I ssh to a couple of specific hosts. so and .109 work fine, but .110 and .111 won't. I can SSH from the same box to .110 and .111 as different users.
      The system runs a couple of zones, but the exact same thing happens when I try it from those zones. In a nutshell: from box to hosts a,b,c and d work fine as "user", from box to hosts a and b work fine as "root", but from box to hosts c and d as root will cause ssh to crash. Here's part of the truss output, I can of course post the entire output if that's helpful:

      17699: getpid() = 17699 [17698]
      17699: write(4, "\0\001 t\t14 u98ECE2 j ]".., 376) = 376
      17699: pollsys(0x08045AC0, 1, 0x00000000, 0x00000000) = 1
      17699: read(4, "\0\0018C\f14 &A7BF 8DD G".., 8192) = 400
      17699: time() = 1291737084
      17699: getpid() = 17699 [17698]
      17699: getpid() = 17699 [17698]
      17699: write(4, "\0\0\08C061E\0\0\080 zE3".., 144) = 144
      17699: pollsys(0x08045A40, 1, 0x00000000, 0x00000000) = 1
      17699: read(4, "\0\001F4\f1F\0\001 "\0\0".., 8192) = 504
      17699: getpid() = 17699 [17698]
      17699: stat64("/etc/ssh/ssh_known_hosts2", 0x08047A50) Err#2 ENOENT
      17699: stat64("/.ssh/known_hosts2", 0x08047A50) Err#2 ENOENT
      17699: getpid() = 17699 [17698]
      17699: open64("/.ssh/known_hosts", O_RDONLY) = 5
      17699: fstat64(5, 0x08044B80) = 0
      17699: fstat64(5, 0x08044CC0) = 0
      17699: brk(0x080B6670) = 0
      17699: brk(0x080B8670) = 0
      17699: fstat64(5, 0x08044C00) = 0
      17699: ioctl(5, TCGETA, 0x08044C94) Err#25 ENOTTY
      17699: read(5, " 1 0 . 5 0 . 1 4 5 . 1 1".., 8192) = 822
      17699: getpid() = 17699 [17698]
      17699: llseek(5, 0xFFFFFFFFFFFFFE65, SEEK_CUR) = 411
      17699: close(5) = 0
      17699: getpid() = 17699 [17698]
      17699: getpid() = 17699 [17698]
      17699: Incurred fault #6, FLTBOUNDS %pc = 0xFECA4CF1
      17699: siginfo: SIGSEGV SEGV_MAPERR addr=0xB4D1E218
      17699: Received signal #11, SIGSEGV [default]
      17699: siginfo: SIGSEGV SEGV_MAPERR addr=0xB4D1E218

      I can't seem to figure out what exactly is causing this fault. Just for the record, I know there's no reason to be doing the stuff I'm doing as root, I just found this out while fixing something else, and I thought it was interesting.
      Any help would be appreciated.

        • 1. Re: SSH segmentation fault, but only while running as root.
          Can you get a core file? If so, what's the output from pstack and pmap on that core file?
          • 2. Re: SSH segmentation fault, but only while running as root.
            Yes, here they are:

            root@armarium $ pstack core
            core 'core' of 17593: ssh xx.xx.xx.110
            feca4cf1 realfree (80b1db8) + 43
            feca530f cleanfree (0) + 44
            feca482b mallocunlocked (18, 809ab70, 14, 14, 8047a68, 8076d06) + ad
            feca4754 malloc (14) + 34
            08076d06 xmalloc (14) + 12
            0806e45a ???????? (80af838, 41, 10, 809ab70, 809cd80)
            0806e5d8 kex_derive_keys (80af838, 809ab70, 809cd80) + 20
            0806e8ff kexdh_client (80af838) + 26f
            0806dd28 ???????? (80af838)
            0806dbfd kex_input_kexinit (14, 0, 80af838) + a1
            0806a206 dispatch_run (0, 80af884, 80af838) + 49
            0805d659 ssh_kex2 (80a8828, 809c460) + 1a6
            0805c169 ssh_login (809b494, 8047e98, 809c460, 809d8c0) + b6
            080597f3 main (0, 8047de0, 8047de4) + bd2
            08058b56 ???????? (2, 8047e94, 8047e98, 0, 8047ea6, 8047eb6)

            root@armarium $ pmap core
            core 'core' of 17593: ssh xx.xx.xx.110
            08044000 16K rw--- [ stack ]
            08050000 224K r-x-- /usr/bin/ssh
            08098000 12K rw--- /usr/bin/ssh
            0809B000 120K rw--- [ heap ]
            FE8E0000 8K r-x-- /usr/lib/gss/dh1024-0.so.1
            FE8F2000 4K rw--- /usr/lib/gss/dh1024-0.so.1
            FE900000 28K r-x-- /usr/lib/gss/mech_dh.so.1
            FE917000 4K rw--- /usr/lib/gss/mech_dh.so.1
            FE920000 8K r-x-- /usr/lib/gss/dh640-0.so.1
            FE932000 4K rw--- /usr/lib/gss/dh640-0.so.1
            FE940000 16K r-x-- /usr/lib/gss/mech_spnego.so.1
            FE954000 4K rw--- /usr/lib/gss/mech_spnego.so.1
            FE960000 12K r-x-- /lib/libmp.so.2
            FE973000 4K rw--- /lib/libmp.so.2
            FE980000 24K r-x-- /lib/libgen.so.1
            FE996000 4K rw--- /lib/libgen.so.1
            FE9A0000 24K r-x-- /lib/libuutil.so.1
            FE9B6000 4K rw--- /lib/libuutil.so.1
            FE9C0000 92K r-x-- /lib/libscf.so.1
            FE9E7000 4K rw--- /lib/libscf.so.1
            FE9F0000 216K r-x-- /lib/libresolv.so.2
            FEA36000 8K rw--- /lib/libresolv.so.2
            FEA40000 560K r-x-- /usr/lib/gss/mech_krb5.so.1
            FEADC000 12K rw--- /usr/lib/gss/mech_krb5.so.1
            FEADF000 4K rw--- /usr/lib/gss/mech_krb5.so.1
            FEAF0000 220K r-x-- /usr/lib/security/pkcs11_softtoken_extra.so.1
            FEB37000 8K rw--- /usr/lib/security/pkcs11_softtoken_extra.so.1
            FEB39000 4K rw--- /usr/lib/security/pkcs11_softtoken_extra.so.1
            FEB40000 56K r-x-- /lib/libmd.so.1
            FEB5E000 4K rw--- /lib/libmd.so.1
            FEB60000 96K r-x-- /usr/lib/security/pkcs11_kernel.so.1
            FEB88000 4K rw--- /usr/lib/security/pkcs11_kernel.so.1
            FEB90000 4K r-x-- /lib/libdoor.so.1
            FEBA1000 4K rw--- /lib/libdoor.so.1
            FEBB0000 24K r-x-- /usr/lib/libcryptoutil.so.1
            FEBC0000 4K rwx--
            FEBC6000 8K rw--- /usr/lib/libcryptoutil.so.1
            FEBD0000 76K r-x-- /usr/lib/libpkcs11.so.1
            FEBF0000 4K rwx--
            FEBF3000 24K rw--- /usr/lib/libpkcs11.so.1
            FEC00000 64K rwx--
            FEC20000 40K r-x-- /usr/sfw/lib/libcrypto_extra.so.0.9.7
            FEC30000 24K rwx--
            FEC3A000 4K rw--- /usr/sfw/lib/libcrypto_extra.so.0.9.7
            FEC40000 12K r-x-- /lib/libcmd.so.1
            FEC50000 4K rwx--
            FEC53000 4K rw--- /lib/libcmd.so.1
            FEC60000 1076K r-x-- /lib/libc.so.1
            FED70000 4K rwx--
            FED7D000 32K rw--- /lib/libc.so.1
            FED85000 8K rw--- /lib/libc.so.1
            FED90000 44K r-x-- /usr/lib/libgss.so.1
            FEDA0000 4K rwx--
            FEDAB000 4K rw--- /usr/lib/libgss.so.1
            FEDB0000 936K r-x-- /usr/sfw/lib/libcrypto.so.0.9.7
            FEEA0000 4K rwx--
            FEEAA000 80K rw--- /usr/sfw/lib/libcrypto.so.0.9.7
            FEEBE000 8K rw--- /usr/sfw/lib/libcrypto.so.0.9.7
            FEED0000 60K r-x-- /usr/lib/libz.so.1
            FEEE0000 4K r-x-- /lib/libmd5.so.1
            FEEEE000 4K rwx-- /usr/lib/libz.so.1
            FEEF0000 516K r-x-- /lib/libnsl.so.1
            FEF81000 20K rw--- /lib/libnsl.so.1
            FEF86000 32K rw--- /lib/libnsl.so.1
            FEF90000 44K r-x-- /lib/libsocket.so.1
            FEFA0000 4K rwx--
            FEFAB000 4K rw--- /lib/libsocket.so.1
            FEFB0000 4K rwx--
            FEFC0000 4K rwx--
            FEFC3000 160K r-x-- /lib/ld.so.1
            FEFF0000 4K rwx--
            FEFFB000 8K rwx-- /lib/ld.so.1
            FEFFD000 4K rwx-- /lib/ld.so.1
            total 5180K