No definitive answer but a few bits of additional information:
I am seeing the same results in a text install on x86.
netstat -anu |grep 443
*.443 *.* root 1646 0 0 256000 0 LISTEN
Connecting to 443 and running netstat -anu again will reveal a webui apache process.
echo "" | telnet 0 443; netstat -anu |grep 443
*.443 *.* root 1656 httpd 0 0 256000 0 LISTEN
webservd 1656 1649 0 20:04:06 ? 0:00 /usr/apache2/2.4/bin/httpd -f /var/webui/conf/webui.conf -k start
Restarting webui/server and then running netstat -anu again, will show the initial setup, with reference to an 'invisible process' but with a fresh process id.
svcadm restart webui/server
netstat -anu |grep 443
*.443 *.* root 1725 0 0 256000 0 LISTEN
Dtracing the svcadm restart webui/server with a simple DTT/Bin/execsnoop.d shows some apache rotatelogs process as a child of that shy process but no sampled exec of it. Having a closer look by syscallsbypid.d suggests that it is some apache httpd passing by.
So I'd relax and come up with two questions:
What types of process creation are not covered in execsnoop (and is there an alternate, already nicely paved way to cover them)?
What data (in terms of sources, freshnesh guarantees etc) does netstat -anu rely on and how does that relate to a change in ownership/control of a socket?
Thanks Jens. Very interesting.
Given that netstat reports a PID, it will be interesting to learn if the info being reported by netstat is just wrong or if 'ps' isn't able to see the PID that is really there (I'm guessing the former). Hopefully when an Oracle engineer sees this issue he/she will let us know what the root cause is.
The pid that's reported is the process that created the socket, which may have then forked (letting the child inherit the socket) and then exited, so that when netstat comes along, /proc has no data on the recorded pid.
That makes complete sense, does not contradict the man page, and indeed is what I hoped might be happening (instead of a rootkit of some sort ;-)
It is, however, a bit unfortunate.
I have always viewed the output of netstat -an as giving me "current" information as to which ports are "open". I think it would be sweet if netstat -u could report the current PID that is actually doing the listening rather than obsolete PID that no longer exists (under the conditions you described).
Just something to think about.
Thanks for the reply!
I consider this issue closed - although if you hear they find a way to make netstat show the current PID by all means let us know.