Categories
OBIEE 12c will not start after configuring SSL on WebLogic
Hello All,
I recently configured the SSL in our OBIEE 12c environment as per the Oracle documentation - OBIEE 12c: How To Configure SSL Including Examples (Doc ID 2188982.1) using custom certificates. After configuration when i tried to restart the BI services using ./start.sh command, I get the following error.
Starting domain; Using domainHome: /opt/oracle/obiee_domains/fibbi ...
Initializing WebLogic Scripting Tool (WLST) ...
Welcome to WebLogic Server Administration Scripting Shell
Type help() for help on available commands
<Mar 4, 2020 6:17:42 PM MST> <Info> <Security> <BEA-090905> <Disabling the CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogicue.>
<Mar 4, 2020 6:17:42 PM MST> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG128 to HMACDRBG. To disable this change, specify -Dweblog
<Mar 4, 2020 6:17:42 PM MST> <Info> <Security> <BEA-090909> <Using the configured custom SSL Hostname Verifier implementation: weblogic.security.utils.SSLWLSHostnameVerifier$NullHostnameVer
Node manager not running. Starting it...
NMProcess: NODEMGR_HOME is already set to /opt/oracle/obiee_domains/fibbi/nodemanager
NMProcess: CLASSPATH=/opt/java/jdk1.8.0_221/lib/tools.jar:/opt/oracle/middleware/wlserver/server/lib/weblogic.jar:/opt/oracle/middleware/wlserver/../oracle_common/modules/thirdparty/ant-conmodules/features/oracle.wls.common.nodemanager.jar::/opt/java/jdk1.8.0_221/lib/tools.jar:/opt/oracle/middleware/wlserver/modules/features/wlst.wls.classpath.jar:/opt/oracle/middleware/wlserfeatures/oracle.wls.common.grizzly.jar
NMProcess: + /opt/java/jdk1.8.0_221/bin/java -server -Xms32m -Xmx200m -Djdk.tls.ephemeralDHKeySize=2048 -Dcoherence.home=/opt/oracle/middleware/wlserver/../coherence -Dbea.home=/opt/oracle/racle/obiee_domains/fibbi/nodemanager -DLogToStderr=false -DQuitEnabled=true -Dweblogic.RootDirectory=/opt/oracle/obiee_domains/fibbi -Doracle.security.jps.config=/opt/oracle/obiee_domains/mon.components.home=/opt/oracle/middleware/oracle_common -Dopss.version=12.2.1.3 -Dweblogic.RootDirectory=/opt/oracle/obiee_domains/fibbi -Doracle.bi.home.dir=/opt/oracle/middleware/bi -Dorbbi/config/fmwconfig/biconfig -Doracle.bi.environment.dir=/opt/oracle/obiee_domains/fibbi/config/fmwconfig/bienv -Doracle.bi.12c=true -Ddomain.home=/opt/oracle/obiee_domains/fibbi -Dfile.encle.classloader.weblogic.LaunchClassLoader -Djava.security.policy=/opt/oracle/middleware/wlserver/server/lib/weblogic.policy -Dweblogic.nodemanager.JavaHome=/opt/java/jdk1.8.0_221 weblogic.
NMProcess: Mar 04, 2020 6:17:46 PM oracle.security.opss.internal.runtime.ServiceContextManagerImpl getContext
NMProcess: WARNING: Bootstrap services are used by OPSS internally and clients should never need to directly read/write bootstrap credentials. If required, use Wlst or configuration managem
NodeManager started
Reading domain...
/Servers/AdminServer/ListenPort=9500
Accessing admin server using URL t3s://<myserver>:9501
Starting AdminServer ...
nmStart(AdminServer) succeeded
Start Admin Server connect Exception caught Error occurred while performing connect : Cannot connect via t3s or https. If using demo certs, verify that the -Dweblogic.security.TrustKeyStoreinitialize JNDI context, tried 2 time or times totally, the interval of each time is 0ms.
t3s://vmtnofsaarpt01.fib.system.root:9501: Destination <myserverip>, 9501 unreachable.; nested exception is:
java.net.ConnectException: Connection refused (Connection refused); No available router to destination.; nested exception is:
java.rmi.ConnectException: No available router to destination.
Use dumpStack() to view the full stacktrace :
Reading domain...
/Servers/AdminServer/ListenPort=9500
Accessing admin server using URL t3s://<myserver>:9501
Requesting credentials ...
Enter Weblogic login details at prompt
Weblogic Username:
The nodemanager log shows the following:
<Mar 4, 2020 6:51:15 PM MST> <INFO> <Loading domains file: /opt/oracle/obiee_domains/fibbi/nodemanager/nodemanager.domains>
<Mar 4, 2020 6:51:15 PM MST> <INFO> <Loading identity key store: FileName=/opt/oracle/keystore/keystore.jks, Type=jks, PassPhraseUsed=false>
<Mar 4, 2020 6:51:16 PM MST> <SEVERE> <Fatal error in NodeManager server>
weblogic.nodemanager.common.ConfigException: Key store identity alias does not contain a certificate chain: weblogic
at weblogic.nodemanager.server.SSLConfig.loadKeyStoreConfig(SSLConfig.java:239)
at weblogic.nodemanager.server.SSLConfig.access$000(SSLConfig.java:33)
at weblogic.nodemanager.server.SSLConfig$1.run(SSLConfig.java:118)
at java.security.AccessController.doPrivileged(Native Method)
at weblogic.nodemanager.server.SSLConfig.<init>(SSLConfig.java:115)
at weblogic.nodemanager.server.NMServer.<init>(NMServer.java:169)
at weblogic.nodemanager.server.NMServer.getInstance(NMServer.java:134)
at weblogic.nodemanager.server.NMServer.main(NMServer.java:589)
at weblogic.NodeManager.main(NodeManager.java:31)
The certificate imported on the keystore is this one:
$ keytool -list -v -keystore keystore.jks
Enter keystore password:
Keystore type: PKCS12
Keystore provider: SUN
Your keystore contains 1 entry
Alias name: weblogic
Creation date: Mar 4, 2020
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=<myserver>, OU=Unknown, O=<myorg>, L=<mylocation>, ST=<mystate>, C=US
Issuer: CN=<myserver>, OU=Unknown, O=<myorg>, L=<mylocation>, ST=<mystate>, C=US
Serial number: 1e28f3e
Valid from: Wed Oct 09 15:19:27 MDT 2019 until: Thu Oct 08 15:19:27 MDT 2020
Certificate fingerprints:
.....
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3
I am reading something about Certificate Root, Intermediary, etc. Not sure if that's the cause of the problem, but, I don't really understand that concept of root, intermediary certificates.
Please advise, I am stuck on this and can't find any specific documentation that has anything additional to do.
Thanks!
Answers
-
Eduardo,
Which precise OBI? Which precise Operating System? Have you first tested it on ONLY the Managed Server before moving on? that's normally the first test you do beore moving on to the Admin Server and potentially destroy your system.
SSL is pretty tricky so without an exact specification of what you have done it will be nigh impossible to help you.
0 -
OBIEE 12.2.1.x and RHEL 7
Not sure how does that make a difference though, as the Oracle document ID I provided you was specifically to my system.
You make a good point about the Node Manager, but the Oracle instructions obviously don’t state that.
I have a feeling that might be my certificate that it does not “go all the way up to the root”. I am currently checking that with the team which generates the certificates.
0 -
Eduardo Ferrari wrote:OBIEE 12.2.1.x and RHEL 7Not sure how does that make a difference though
Your so called ".x" omits just about any useful information you could have possibly given because that's the only part with which the - so far - 30 releases of the 12c family distinguish themselves and are ALL called "12.2.1...".
And I hate to say it, but: software changes over the span of five years and there where literally thousands of bug fixes. It's how IT works.
Eduardo Ferrari wrote:as the Oracle document ID I provided you was specifically to my system.
Again, sorry, but it isn't and working with such an assumption is dangerous. Believe me, many cases posted in here have people claiming they "have done everything" and "were precise" just to end up with "oh right, this was wrong". Precision helps solving issues. Being vague wastes everybody's time and makes threads unusable for other users.
Eduardo Ferrari wrote:You make a good point about the Node Manager, but the Oracle instructions obviously don’t state that.
Of course it doesn't because it's written for ONE specific case that worked. The other 99'000 flavours of the use case which can differ ever so slightly aren't covered.
Eduardo Ferrari wrote:I have a feeling that might be my certificate that it does not “go all the way up to the root”. I am currently checking that with the team which generates the certificates.
Yes. The certificate are always the absolute crux. I had huge issues when following 2188982.1 step by step or trying to help clients who followed it and maneuvered themselves into a dead-end where they had to set up the machines again. 2188982.1 simply doesn't ever tell you WHY you do WHAT and how you should do it differently in different cases and how it can impact the solution.
Is there one certificate chain? Chain with key? Singular, distinct certificates?
Did you import it all into the original keystore?
etc etc
As I said it's not a straightforward thing and hard to ask the exact question that leads to the solution, especially if you're (me in this case) isn't sitting in front of the system.
0 -
Generally, when you get a certificate, you extract the Root, Intermediate and Personal Certificate, install them and then import them into your identity.
You should see 3 entries in your identity: 1 for Root, 1 for Intermediate and 1 for Personal.
Once all the steps have been covered, you should start the NODEMANAGER Standalone to see if it is pulling all the configurations and is not complaining about any of the changes.
Regarding, the doc, I would suggest you should try and refer a few more docs as this doc is not 100%. Its at very high level.
0 -
SonPat99 wrote:Generally, when you get a certificate, you extract the Root, Intermediate and Personal Certificate, install them and then import them into your identity.
Not necessarily. It depends on what you are being provided with by your security department.
SonPat99 wrote:Once all the steps have been covered, you should start the NODEMANAGER Standalone to see if it is pulling all the configurations and is not complaining about any of the changes.
No. Doing "all steps" is definitely an error. Only ever configure one managed server to use SSL and test it. Do not touch any other component until you have completely proven that it works.
0 -
After I got the correct certificate, then it worked.
1 -
Eduardo Ferrari Please help me
I create new keystore to configure SSL . I do step by step
- keytool -genkeypair
- keytool -certreq
- java utils.CertGen
- java utils.ImportPrivateKey
- cteare Trust.jks from standard java Trust
- keytool -import -v -trustcacert
everything is ok
But finally I check chainhing of certificate it is incomplete
java utils.ValidateCertChain
Certificate chain is incomplete, can't confirm the entire is valid
How to You got the correct certificate ?
Thanks so much
0 -
Are you using a self-signed cert? It seems like a problem on the cert you have. You must have (in most cases) 3 certs if you are not using self-signed.
1