4 Replies Latest reply on Jun 5, 2018 8:33 PM by user1574356

    netstat showing a root user / PID that is not shown by ps or ptree - what??

    user1574356

      fresh install of solaris 11.4 beta1 from usb image  (sha256sum digest confirmed)

       

      I was curious to see what ports were open and ran netstat ... was really surprised to see a root user pid without a corresponding process name ...

       

      If the sha256 digest had not matched I might wonder if this was a kernel level hack ...

       

      # uname -a

      SunOS mybox 5.11 11.4.0.3.0 i86pc i386 i86pc

       

      # netstat -anu

      ...

      TCP: IPv4

         Local Address        Remote Address      User     Pid     Command     Swind  Send-Q  Rwind  Recv-Q    State

      -------------------- -------------------- -------- ------ ------------- ------- ------ ------- ------ -----------

      ...

            *.*                  *.*            daemon     1199 rpcbind             0      0  256000      0 IDLE

            *.515                *.*            root       1254 inetd               0      0  256000      0 LISTEN

            *.6787               *.*            root       1346 httpd               0      0  256000      0 LISTEN

            *.*                  *.*            root       1336                     0      0  256000      0 IDLE

            *.443                *.*            root       1336                     0      0  256000      0 LISTEN

            *.*                  *.*            root       1336                     0      0  256000      0 IDLE

      127.0.0.1.25               *.*            root       1421 sendmail            0      0  256000      0 LISTEN

       

      TCP: IPv6

         Local Address                     Remote Address                   User    Pid      Command      Swind  Send-Q  Rwind  Recv-Q   State      If

      --------------------------------- --------------------------------- -------- ------ -------------- ------- ------ ------- ------ ----------- -----

            *.443                             *.*                         root       1336                      0      0  256000      0 LISTEN     

       

       

      # ptree 1336

      <nothing>

       

      # ps -ef | grep 1336

          root  1816  1646   0 13:52:38 pts/1       0:00 grep 1336

       

      $ sha256sum *usb

      22002f0065255e5e8a612573d77279a4ba7eb48f3f7d9004789b3b46a6ea0876  sol-11_4-beta1-text-x86.usb

       

      What in the world is going on here ?

        • 1. Re: netstat showing a root user / PID that is not shown by ps or ptree - what??
          Jens

          No definitive answer but a few bits of additional information:

          I am seeing the same results in a text install on x86.

          netstat -anu |grep 443

              *.443                *.*            root      1646                    0      0  256000      0 LISTEN

           

          Connecting to 443 and running netstat -anu again will reveal a webui apache process.

          echo "" | telnet 0 443; netstat -anu |grep 443

              *.443                *.*            root      1656 httpd              0      0  256000      0 LISTEN

          ps:

          webservd  1656  1649  0 20:04:06 ?          0:00 /usr/apache2/2.4/bin/httpd -f /var/webui/conf/webui.conf -k start

           

          Restarting webui/server and then running netstat -anu again, will show the initial setup, with reference to an 'invisible process' but with a fresh process id.

           

          svcadm restart webui/server

          netstat -anu |grep 443

                *.443                *.*            root       1725                     0      0  256000      0 LISTEN

           

          Dtracing the svcadm restart webui/server with a simple DTT/Bin/execsnoop.d shows some apache rotatelogs process as a child of that shy process but no sampled exec of it. Having a closer look by syscallsbypid.d suggests that it is some apache httpd passing by.

           

          So I'd relax and come up with two questions:

          What types of process creation are not covered in execsnoop (and is there an alternate, already nicely paved way to cover them)?

          What data (in terms of sources, freshnesh guarantees etc) does netstat -anu rely on and how does that relate to a change in ownership/control of a socket?

          • 2. Re: netstat showing a root user / PID that is not shown by ps or ptree - what??
            user1574356

            Thanks Jens.  Very interesting. 

             

            Given that netstat reports a PID, it will be interesting to learn if the info being reported by netstat is just wrong or if 'ps' isn't able to see the PID that is really there (I'm guessing the former).   Hopefully when an Oracle engineer sees this issue he/she will let us know what the root cause is.

            • 3. Re: netstat showing a root user / PID that is not shown by ps or ptree - what??
              Dave Miner-Oracle

              The pid that's reported is the process that created the socket, which may have then forked (letting the child inherit the socket) and then exited, so that when netstat comes along, /proc has no data on the recorded pid.

              • 4. Re: netstat showing a root user / PID that is not shown by ps or ptree - what??
                user1574356

                Hi Dave,

                 

                That makes complete sense, does not contradict the man page, and indeed is what I hoped might be happening (instead of a rootkit of some sort ;-)

                 

                It is, however, a bit unfortunate.  

                 

                I have always viewed the output of netstat -an  as giving me "current" information as to which ports are "open".   I think it would be sweet if netstat -u   could report the current PID that is actually doing the listening rather than obsolete PID that no longer exists (under the conditions you described).

                 

                Just something to think about.

                 

                Thanks for the reply!

                 

                I consider this issue closed - although if you hear they find a way to make netstat show the current PID by all means let us know.