SEC_ERROR_CA_CERT_INVALID

Can you try and send the certutil output of origin server (app. server in your case I think) and reverse proxy Web Server instance configuration directory where cert*.db is located? .
$certutil -L -d .

You need to check out trust flags as per
[http://archive.netbsd.se/?ml=mozilla-crypto&a=2007-06&t=4354970|http://archive.netbsd.se/?ml=mozilla-crypto&a=2007-06&t=4354970]

Wondering if your App. Server have cert*.db? Then you could import this Web Server certificate into your App. Server as a trusted CA certificate and App. Server's cert. into Web Server as a trusted CA.

receivables wrote:
1. Generate selfsigned certificate (Web Server) lets say the nickname abc-cert (changing flags to 'CTu,u,u' for trusted CA)
2. Configure web server with http listener points to lets say https://abc.com:443
3. Generate selfsigned certificate (AppServer side) lets say def-cert generated by using keytool. (changing flags to 'CTu,u,u')
4. Configure AppServer listener points to 8181 with SSL nickname as def-cert
5. Imports the certificate into the servers:
1. def-cert into Web Server

Is required. In SSL handshake, when Web Server connects to App. Server as client, App. Server has to send back its certificate (def-cert in this case) to Web Server. As App. Server's server certificate is a self signed certificate and is not signed by a trusted CA, you need to add this App. Server's server certificate into Web Server's NSS database with trusted CA flags. For more info about trust flags : [http://www.mozilla.org/en/projects/security/pki/nss/tools/certutil.html|http://www.mozilla.org/en/projects/security/pki/nss/tools/certutil.html]

2. abc-cert into App Server

This is required only if client auth is enabled on Application Server which most probably is not.

6. Is it required to associate one another, i mean keystore and truststore?

receivables wrote:
I am still getting the same error. Moreover, I could not open the exported certificate after applying the command
/pk12util -o /tmp/exported.crt -n My-CA-Cert -d .

The error is alerted when I tried to open the certifcate.

This file is invalid for use as the following:Security Certificate

To export server certificate from App. Server you can try keytool or migrate-jks-keystore Admin. CLI
[http://blogs.sun.com/meena/entry/migrating_jks_keystore_entries_to |http://blogs.sun.com/meena/entry/migrating_jks_keystore_entries_to ]
[http://docs.sun.com/app/docs/doc/820-4842/migrate-jks-keycert-1?a=view|http://docs.sun.com/app/docs/doc/820-4842/migrate-jks-keycert-1?a=view]
Assuming that Web Server certificate nickname is different from AppServer Certificate. Otherwise it will conflict with the Web Server's server certificate.
You will have to manually run certutil on Web Server NSS database l to modify trust flags of this App. Server certificate imported later.

Or easier way is to connect to App. Server via browser with ssltap
[http://www.mozilla.org/projects/security/pki/nss/tools/ssltap.html|http://www.mozilla.org/projects/security/pki/nss/tools/ssltap.html] in between. It will save the App. Server server certificate as cert.001. You can import that app. server certificate using certutil.

May be you need to tweak trust flags.

Try "CTu,CTu,CTu" [http://www-archive.mozilla.org/projects/security/pki/nss/ref/ssl/gtstd.html|http://www-archive.mozilla.org/projects/security/pki/nss/ref/ssl/gtstd.html] says
"The trust flag settings "CTu,CTu,CTu" indicate that the certificate is a CA certificate that is trusted to issue both client (C) and server (T) SSL certificates as well as email and object-signing certificates. The u flag indicates that the CA certificate can itself be used for signing or authentication operations.

NOTE: The u flag should not be set in a root CA certificate intended for deployment in a real (as opposed to a test) PKI for chaining purposes. Only the authorized CA should be able to sign anything or authenticate itself with a CA certificate. "

Can you attach ssltap [http://www.mozilla.org/projects/security/pki/nss/tools/ssltap.html|http://www.mozilla.org/projects/security/pki/nss/tools/ssltap.html] and see what's happening?

You can also truss to see which function was called and is returning this error

$truss -o truss.out -fall -u "*" <wwebservd-largest-pid>

Are you trying to export server certificate from Web Server? If your Application Server does NOT have client authentication enabled you can skip this step.

Which tool are you using to open the certificate (which is giving this error message)?

I found 2 related links in google search :
[https://knowledge.verisign.com/support/eca-support/index?page=content&id=SO4410&actp=LIST] [https://support.comodo.com/index.php?_m=knowledgebase&_a=viewarticle&kbarticleid=1163]

MS Windows decides what to do with a file based on the "file name extension"
(the letters after the last "dot" in the name). Some common file name
extensions related to certificates are:

.cer  .crt  .der  - a file with a single certificate in PEM or binary DER

.p7b              - a PKCS#7 signedData object, containing a signature 
                      and certificates, also called a "certificate package".

.pfx .p12         - a PKCS#12 file containing a private key and one 
                        or more certificates.

Since you do not intend to copy the private key, but only want to copy the certificate, then pk12util was the wrong tool. In that case, use certutil to create a .crt file, e.g.

  $certutil -L -o /tmp/exported.crt -n My-CA-Cert -d .

receivables wrote:
1. do we need to have server certificate and instance name should exactly the same or not? since the above scenario the instance name and subject or name of the certificate is different.
lets say an example
my certificate url name is 'uat.mycertificate.com' my instance name is 'uatweb.mycompany.com'

Do u get any error at the time of start up of the instance like :
[http://forums.sun.com/thread.jspa?threadID=5272256|http://forums.sun.com/thread.jspa?threadID=5272256]

warning ( 3256): CORE1251: On HTTP listener http-listener-1, server name SSL does not match subject "www.ourdomain.com" of certificate Server-Cert.