This discussion is archived
2 Replies Latest reply: Aug 6, 2012 6:34 AM by user650622 RSS

TLS client authentication fails when using SunMSCAPI

user650622 Newbie
Currently Being Moderated
I've not been able to use the SunMSCAPI provider for TLS client authentication. When using the default Java 6 provider and a test client certificate, authentication succeeds as expected. However, authentication using the very same test certificate fails with a, *" Unexpectedly, privatekey is not an RSA private key"*, when the default provider is replaced by SunMSCAPI.

The test cert can be accessed by alias and printed from the application. The client Windows platform is a Win2008R2 box, and both the test cert and its signing CA are enabled for all purposes. The unlimited policy jars (we're FIPS 140-2 L1) have been installed as well. As a side note, SunMSCAPI is also being used for server certs, and that has been working quite well.

Any light you may be able to shed on this will be greatly appreciated.

  • 1. Re: TLS client authentication fails when using SunMSCAPI
    sabre150 Expert
    Currently Being Moderated
    user650622 wrote:
    fails with a, *" Unexpectedly, privatekey is not an RSA private key"*, when the default provider is replaced by SunMSCAPI.
    Probably not a great help but this is the error message one gets when trying to use a SunMSCAPI private key as an RSA PrivateKey using the Sun/Oracle JCE provider to perform RSA encryption. The implication is that you have not actually replaced the default provider with the SunMSCAPI provider for all operations. Beyond this I can't help.
  • 2. Re: TLS client authentication fails when using SunMSCAPI
    user650622 Newbie
    Currently Being Moderated
    Hello, and thanks for the fast response.

    The relevant bits of my code producing this exception are:
    kmf = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());   
    Provider mscapiProvider = new;   
    Security.insertProviderAt (mscapiProvider, 1);   
    clientKeyStore = KeyStore.getInstance("Windows-MY",mscapiProvider);
    System.err.println("Provider: " + clientKeyStore.getProvider().getName());
    System.err.println("KeyStore Sz: " + clientKeyStore.size());
    Enumeration<String> item  = clientKeyStore.aliases();
    while( item.hasMoreElements() )
        String name = item.nextElement();
        System.err.println("CertAlias: " + name);
        System.err.println("privKey: " + clientKeyStore.getKey(name,"".toCharArray()).toString());
    factory = new SSLSocketFactory(sslContext);
    // Lotsa logic removed/simplified here for clarity
    sock = (SSLSocket)factory.createSocket(destHost, destPort);
    // more logic removed/simplified
    BufferedInputStream resp = new BufferedInputStream(sock.getInputStream());
    bytesRead =, startingDataOffset, (bufSz - startingDataOffset) );   << Exception thrown here
    The output for this is as expected:

    Provider: SunMSCAPI
    KeyStore Sz: 3
    CertAlias: TestCert
    privKey: RSAPrivateKey [size=1024 bits, type=Exchange, container={860F8919-95BD-445D-9B87-D6280B530FB0}]
    CertAlias: ClientCert
    privKey: RSAPrivateKey [size=1024 bits, type=Exchange, container={0A940E3A-50A4-434A-BD2B-AAAC3AE2C9D8}]
    CertAlias: ClientCert1
    privKey: RSAPrivateKey [size=1024 bits, type=Exchange, container={56463C2B-AEF3-48A3-910C-5F3ECCE8096F}]

    ... the exception is not:

    Caused by:
    java.lang.RuntimeException: Unexpectedly, privatekey is not an RSA private key
         at com.rsa.jsse.engine.util.c.a(Unknown Source)
         at com.rsa.sslj.x.cH.<init>(Unknown Source)
         at com.rsa.sslj.x.t.i(Unknown Source)
         at com.rsa.sslj.x.t.f(Unknown Source)
         at com.rsa.sslj.x.t$ Source)
         at com.rsa.sslj.x.aK$a$ Source)
         at Method)
         at com.rsa.sslj.x.aK$ Source)
         at com.rsa.sslj.x.aV.a(Unknown Source)
         at com.rsa.sslj.x.aV.b(Unknown Source)
         at com.rsa.sslj.x.aV.b(Unknown Source)
         at Source)
         at com.xxxx.SSLClientSocket.socketBufferedRead(
         at com.xxxx.SSLClientSocket.socketRead(
         at com.xxxx.SSLClientTestBench.testOneSocket(
         at com.xxxx.SSLClientTestBench.main(
    Java Result: 1

    The exception occurs after the Server's Hello Done while the client is responding to the request for it's certificate.
    The client is able to form its certificate chain, the client key exchange and session keys before producing the fatal error:

    ***Session invalidated:
    [Session ID [
      0000: e0 3d 00 00 67 c7 c4 72 67 12 12 c2 e9 f7 33 cd [.=..g..rg.....3.]
    0010: 3c 05 94 3d 60 81 a8 8a f1 1e e0 be ce 99 4a 6c [<..=`.........Jl]
    ***SEND Alert Fatal, Internal Error

    Edited by: user650622 on Aug 3, 2012 9:09 AM
    Corrected a transcription/clean-up error in while loop.

    Edited by: user650622 on Aug 4, 2012 9:04 AM
    Added additional detail


  • Correct Answers - 10 points
  • Helpful Answers - 5 points