Working on a small server application using SSLServerSocket to accept() incoming connections, it seems that incomplete SSL negotiation can completely block incoming connections. It seems that each un-negotiated connection occupies one backlog slot - so once you have 50 or so connections that haven't yet completed negotiations no new users are able to connect. These incomplete connections don't seem to time out by default either.
Is there any easy way to change this in my code so that SSL negotiation must be completed within 10 seconds or it fails?
Several wrong statements there. The backlog queue is for connections that have been completed by the TCP stack but not yet accepted by the application. Nothing to do with SSL yet. JSSE doesn't do any negotiation until you do some I/O on the accepted socket, or call startHandshake() on it, both of which you would do in the thread dealing with the connection. I don't see how you can make a DOS vulnerability out of that, at least not an SSL-specific one. If you are experiencing DOS conditions, most probably you are doing I/O in the accept() thread that should be done in the connection-handling thread.
Connection conn = new Connection(client); // starts own thread ...
telnet server 443
this blocks, ok because no SSL negotiate
In another terminal:
openssl s_client --connect server:443 + certs etc
As long as the first tcp connection hangs waiting for SSL negotiation the second legal connection wil
never be answered. There should have been a negotiation timeout in the SSL driver. ( I think ... :-))
But maybe I overlook something.
Suffice for what? I don't understand the question.
Maybe you still don't understand the issue here. I stated it above in the first reply. The original poster above is mistaken in his contention. SSLServerSocket.accept() doesn't do any I/O itself. Specifically it doesn't perform the SSL handshake. So there is no way for one rogue client to block the server's accept loop unless something else in the accept loop performs I/O. For example, creating ObjectInputStreamsor ObjectOutputStreams. Note that in your case the constructor of the Connection object runs in the accept loop, not in the new thread. If the new thread does all the I/O on the accepted socket, only the new thread can be blocked by a rogue client. So all you have to do is ensure that condition.
To avoid the rogue affecting that thread, set a shortish timeout and call startHandshake(): if you get any exception, close the socket and forget about it.