This is the OpenSSO documentation which hasn't been updated for years and just contains the most essential information assuming you already know what you are doing and was written for a very old version of the iplanet web server: http://docs.oracle.com/cd/E19316-01/820-3885/gimrf/index.html
Basically, you need to configure the application server on which you have deployed OpenSSO to run in https mode, ie install a server cert, and also change the security settings of the application server/deployed app (opensso.war see the web.xml file) to require client authentication. This will make the application server send a request to the browser which will prompt the user to select their certificate to send to the server (this assumes the user already has a personal certificate). Enabling the certificate authentication module in OpenSSO simply will accept and parse the client certificate that the application server got from the user so before specifying this auth module as default, ensure that the server on which OpenSSO has been deployed on has the necessary mechanism enabled.
If the application server on which OpenSSO is deployed on is reverse proxied by another server such as apache or iplanet web server, then the client certificate request needs to be initiated at that end and needs to be forwarded to the application server in order to reach OpenSSO.
The document linked above shows how to do that for an old iplanet web server but the same config still applies to the newer iplanet web servers as well.
I recommend you read up on client/mutual certificate authentication first so you understand the concept. Once you do, then implementing it in OpenSSO is quite straight forward.
This link might help get you started: http://docs.oracle.com/javaee/1.4/tutorial/doc/Security5.html
I happened to see this thread. I have configured the Cert Auth Module in OSSO with OCSP validation. Configured a new Auth chain with Cert Auth as primary and LDAP authentication as a fall back. Made the new Auth chain as default for both users and administrators.
I verified by hitting my OSSO URL and it takes me straight to LDAP authentication only (where LDAP auth works fine).
I tried hitting the Cert Auth Module of OSSO directly (something like https://<FQDN>:<port>/osso/UI/Login?module=PKI) and it gives me a "Authentication failed" message without requesting for my browser certificate.
Now my question is that,
How to confirm (from OSSO logs) that the OSSO authentication invoked OCSP validation ? Where is the OCSP validation success / failed messages logged in OSSO ?