2 Replies Latest reply on Aug 6, 2012 1:34 PM by user650622

    TLS client authentication fails when using SunMSCAPI

      I've not been able to use the SunMSCAPI provider for TLS client authentication. When using the default Java 6 provider and a test client certificate, authentication succeeds as expected. However, authentication using the very same test certificate fails with a, *"javax.net.ssl.SSLException: Unexpectedly, privatekey is not an RSA private key"*, when the default provider is replaced by SunMSCAPI.

      The test cert can be accessed by alias and printed from the application. The client Windows platform is a Win2008R2 box, and both the test cert and its signing CA are enabled for all purposes. The unlimited policy jars (we're FIPS 140-2 L1) have been installed as well. As a side note, SunMSCAPI is also being used for server certs, and that has been working quite well.

      Any light you may be able to shed on this will be greatly appreciated.

        • 1. Re: TLS client authentication fails when using SunMSCAPI
          user650622 wrote:
          fails with a, *"javax.net.ssl.SSLException: Unexpectedly, privatekey is not an RSA private key"*, when the default provider is replaced by SunMSCAPI.
          Probably not a great help but this is the error message one gets when trying to use a SunMSCAPI private key as an RSA PrivateKey using the Sun/Oracle JCE provider to perform RSA encryption. The implication is that you have not actually replaced the default provider with the SunMSCAPI provider for all operations. Beyond this I can't help.
          1 person found this helpful
          • 2. Re: TLS client authentication fails when using SunMSCAPI
            Hello, and thanks for the fast response.

            The relevant bits of my code producing this exception are:
            kmf = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());   
            Provider mscapiProvider = new sun.security.mscapi.SunMSCAPI();   
            Security.insertProviderAt (mscapiProvider, 1);   
            clientKeyStore = KeyStore.getInstance("Windows-MY",mscapiProvider);
            System.err.println("Provider: " + clientKeyStore.getProvider().getName());
            System.err.println("KeyStore Sz: " + clientKeyStore.size());
            Enumeration<String> item  = clientKeyStore.aliases();
            while( item.hasMoreElements() )
                String name = item.nextElement();
                System.err.println("CertAlias: " + name);
                System.err.println("privKey: " + clientKeyStore.getKey(name,"".toCharArray()).toString());
                                new java.security.SecureRandom());
            factory = new SSLSocketFactory(sslContext);
            // Lotsa logic removed/simplified here for clarity
            sock = (SSLSocket)factory.createSocket(destHost, destPort);
            // more logic removed/simplified
            BufferedInputStream resp = new BufferedInputStream(sock.getInputStream());
            bytesRead = resp.read(recvBuffer, startingDataOffset, (bufSz - startingDataOffset) );   << Exception thrown here
            The output for this is as expected:

            Provider: SunMSCAPI
            KeyStore Sz: 3
            CertAlias: TestCert
            privKey: RSAPrivateKey [size=1024 bits, type=Exchange, container={860F8919-95BD-445D-9B87-D6280B530FB0}]
            CertAlias: ClientCert
            privKey: RSAPrivateKey [size=1024 bits, type=Exchange, container={0A940E3A-50A4-434A-BD2B-AAAC3AE2C9D8}]
            CertAlias: ClientCert1
            privKey: RSAPrivateKey [size=1024 bits, type=Exchange, container={56463C2B-AEF3-48A3-910C-5F3ECCE8096F}]

            ... the exception is not:

            Caused by:
            java.lang.RuntimeException: Unexpectedly, privatekey is not an RSA private key
                 at com.rsa.jsse.engine.util.c.a(Unknown Source)
                 at com.rsa.sslj.x.cH.<init>(Unknown Source)
                 at com.rsa.sslj.x.t.i(Unknown Source)
                 at com.rsa.sslj.x.t.f(Unknown Source)
                 at com.rsa.sslj.x.t$a.run(Unknown Source)
                 at com.rsa.sslj.x.aK$a$a.run(Unknown Source)
                 at java.security.AccessController.doPrivileged(Native Method)
                 at com.rsa.sslj.x.aK$a.run(Unknown Source)
                 at com.rsa.sslj.x.aV.a(Unknown Source)
                 at com.rsa.sslj.x.aV.b(Unknown Source)
                 at com.rsa.sslj.x.aV.b(Unknown Source)
                 at com.rsa.sslj.x.Z.read(Unknown Source)
                 at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
                 at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
                 at com.xxxx.SSLClientSocket.socketBufferedRead(SSLClientSocket.java:1745)
                 at com.xxxx.SSLClientSocket.socketRead(SSLClientSocket.java:1592)
                 at com.xxxx.SSLClientTestBench.testOneSocket(SSLClientTestBench.java:397)
                 at com.xxxx.SSLClientTestBench.main(SSLClientTestBench.java:97)
            Java Result: 1

            The exception occurs after the Server's Hello Done while the client is responding to the request for it's certificate.
            The client is able to form its certificate chain, the client key exchange and session keys before producing the fatal error:

            ***Session invalidated:
            [Session ID [
              0000: e0 3d 00 00 67 c7 c4 72 67 12 12 c2 e9 f7 33 cd [.=..g..rg.....3.]
            0010: 3c 05 94 3d 60 81 a8 8a f1 1e e0 be ce 99 4a 6c [<..=`.........Jl]
            ], TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA]
            ***SEND Alert Fatal, Internal Error

            Edited by: user650622 on Aug 3, 2012 9:09 AM
            Corrected a transcription/clean-up error in while loop.

            Edited by: user650622 on Aug 4, 2012 9:04 AM
            Added additional detail