2 Replies Latest reply: May 12, 2011 6:27 PM by 800207 RSS

    JSSE Handshake failing; buffer too small?

    861613
      We are having an issue with SSL connections failing from our clients to our java based SSL daemon.

      We require two-way authentication to connect to this daemon so generally we add new certificates whenever a new client comes on board.

      Currently we simply add the new trusted certificates into our keystore with our certificate chain and then in java we point both the keystore and the trust store to the same keystore.

      Just recently we have had clients failing to be able to connect to the daemon so I ran some tests with "-Djavax.net.debug=ssl" set.

      I ran tests with a single client and used the current keystore and then three archived older keystores.

      The tests with keystores where this one cliaent failed to connect all had the same failure point. Here is a sample from the log:

      ===========================
      ...
      ***
      *** CertificateRequest
      Cert Types: RSA, DSS
      Cert Authorities:

      ...dump of the authorities in our keystore...

      qtp3686138-128, WRITE: SSLv3 Handshake, length = 16384
      *** ServerHelloDone
      qtp3686138-128, WRITE: SSLv3 Handshake, length = 320
      qtp3686138-126, called closeOutbound()
      qtp3686138-126, closeOutboundInternal()
      qtp3686138-126, SEND SSLv3 ALERT: warning, description = close_notify
      qtp3686138-126, WRITE: SSLv3 Alert, length = 2
      Using SSLEngineImpl.

      ============================

      Here is a sample at the same point in time with a keystore with a successful connect

      ============================

      ...

      ***
      *** CertificateRequest
      Cert Types: RSA, DSS
      Cert Authorities:

      ...dump of the authorities in our keystore...

      *** ServerHelloDone
      qtp18350021-132, WRITE: SSLv3 Handshake, length = 16233
      Using SSLEngineImpl.
      qtp18350021-131, called closeOutbound()
      qtp18350021-131, closeOutboundInternal()
      qtp18350021-129, READ: SSLv3 Handshake, length = 1077
      *** Certificate chain

      .....

      ============================


      What bothers me about the failures is that there are always the two WRITE log entries with one which a length of 16384 and then a second write with some remaining bytes. 2^14 is 16384 and this smells to me like a buffer limit.

      qtp3686138-128, WRITE: SSLv3 Handshake, length = 16384
      *** ServerHelloDone
      qtp3686138-128, WRITE: SSLv3 Handshake, length = 320

      All of the successes had less than 16384 and so there was only one WRITE entry.

      The keystores have the following:

      Entry Count Test
      ==========================
      keystore -4 76 Good
      keystore -3 77 Good
      keystore -2 80 Fail
      keystore -1 81 Fail
      keystore (current) 79 Fail

      In an attempt to try to fix the problem some older certificates have been removed from the keystore. But this is not really a fix. We can have new clients show up every day who wish to use this daemon.

      Am I on the right track here?

      Thanks!

      ..billg
        • 1. Re: JSSE Handshake failing; buffer too small?
          EJP
          Currently we simply add the new trusted certificates into our keystore with our certificate chain and then in java we point both the keystore and the trust store to the same keystore.
          Wrong wrong wrong. The keystore is a high-security item containing your private key. It must be under a very strict change regime and indeed a very strict access regime. Otherwise your private key can leak and you are open to all kinds of impersonation attacks. The truststore is basically a public file containing known CAs and doesn't need to be any regime particulaarly. Combining them is a major security risk. You need to fix this pronto.
          *** CertificateRequest
          qtp3686138-126, called closeOutbound()
          Unless there is a missing '*** Certificate Chain' message in between those two lines, the client didn't respond with a certificate. Either it doesn't have one or it doesn't have one that is ultimately signed by one of those CAs. And as the server is presumably configured with needClientAuth=true, it closed the connection when the next message from the client after the cert request wasn't a cert chain.
          *** CertificateRequest
          *** Certificate chain
          See?

          And fix your keystore/truststore usage, it is a security disaster waiting to happen.
          • 2. Re: JSSE Handshake failing; buffer too small?
            800207
            Am I on the right track here?
            Possibly, if there is a bug in the client's SSL implementation. The SSL record layer can transmit a maximum of 2^14 bytes per record. Larger messages are fragmented into as many SSL records of <= 2^14 bytes as necessary. However, the record itself carries no indication of whether it is a fragment or not. This means that the receiver must have a way of knowing when it has received a complete message. The SSL implementation itself is a client of its record layer in that the SSL handshake messages are sent in the record layer. An SSL handshake message starts with a 3 byte length field which allows for consuming as many SSL records as necessary. It is possible that a buggy implementation will simply assume or expect to find a a whole handshake message contained in a single SSL record. This is clearly wrong, because the handshake message has a maxiumum length of 2^24 - 1 bytes.

            That really is an enormous handshake message however. Are the certificate chain unusually long, or is some certificate in it unusually large? It might be easier to optimize the size of the server certificate chain than to get the buggy client(s) patched.