This discussion is archived
5 Replies Latest reply: Dec 12, 2011 12:24 AM by EJP RSS

SSLServerSocket accept() Denial Of Service

856449 Newbie
Currently Being Moderated
Working on a small server application using SSLServerSocket to accept() incoming connections, it seems that incomplete SSL negotiation can completely block incoming connections. It seems that each un-negotiated connection occupies one backlog slot - so once you have 50 or so connections that haven't yet completed negotiations no new users are able to connect. These incomplete connections don't seem to time out by default either.

Is there any easy way to change this in my code so that SSL negotiation must be completed within 10 seconds or it fails?
  • 1. Re: SSLServerSocket accept() Denial Of Service
    EJP Guru
    Currently Being Moderated
    Several wrong statements there. The backlog queue is for connections that have been completed by the TCP stack but not yet accepted by the application. Nothing to do with SSL yet. JSSE doesn't do any negotiation until you do some I/O on the accepted socket, or call startHandshake() on it, both of which you would do in the thread dealing with the connection. I don't see how you can make a DOS vulnerability out of that, at least not an SSL-specific one. If you are experiencing DOS conditions, most probably you are doing I/O in the accept() thread that should be done in the connection-handling thread.
  • 2. Re: SSLServerSocket accept() Denial Of Service
    904496 Newbie
    Currently Being Moderated
    Hi ,
    We experience the same problem:

    listen = getServer();
    while(true) {
    listen.setSoTimeout(accepttimeout * 1000);
    client = (SSLSocket) listen.accept();

    Connection conn = new Connection(client); // starts own thread ...
    ...                         
    }
    I do:
    telnet server 443
    this blocks, ok because no SSL negotiate

    In another terminal:
    openssl s_client --connect server:443  + certs etc
    As long as the first tcp connection hangs waiting for SSL negotiation the second legal connection wil
    never be answered. There should have been a negotiation timeout in the SSL driver. ( I think ... :-))
    But maybe I overlook something.

    regards
    Rob
  • 3. Re: SSLServerSocket accept() Denial Of Service
    EJP Guru
    Currently Being Moderated
    You must be doing I/O in the accepting thread as I said. For example, in the constructor for Connection.
  • 4. Re: SSLServerSocket accept() Denial Of Service
    904496 Newbie
    Currently Being Moderated
    Hi,

    OK, I saw that when the (telnet 443 non ssl) client sends some rubbish
    the connection is immediately dropped and the other legal ssl connection
    resumes.

    Would a read() or write() of zero bytes on the java SSLServer socket suffice?

    Thanks

    Rob
  • 5. Re: SSLServerSocket accept() Denial Of Service
    EJP Guru
    Currently Being Moderated
    Suffice for what? I don't understand the question.

    Maybe you still don't understand the issue here. I stated it above in the first reply. The original poster above is mistaken in his contention. SSLServerSocket.accept() doesn't do any I/O itself. Specifically it doesn't perform the SSL handshake. So there is no way for one rogue client to block the server's accept loop unless something else in the accept loop performs I/O. For example, creating ObjectInputStreamsor ObjectOutputStreams. Note that in your case the constructor of the Connection object runs in the accept loop, not in the new thread. If the new thread does all the I/O on the accepted socket, only the new thread can be blocked by a rogue client. So all you have to do is ensure that condition.

    To avoid the rogue affecting that thread, set a shortish timeout and call startHandshake(): if you get any exception, close the socket and forget about it.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points