Nodemanager not coming online using START.cmd in OBIEE 12c

SonPat99 · Nov 3, 2017 10:28AM

DEAR GURUS,

We did a fresh installation of OBIEE 12c on Windows Server 2012 R2, OBIEE 12.2.1.3.0.

The issue is like:

We are trying to implement SSL in Weblogic/OBIEE.

I have created the certificates and got it signed by CA. I have applied the certificates and SSL is working fine.

Issue is, when I am using the START.cmd/ STOP.cmd, the Nodemanager is not coming online with below error:

\OBIEE12cDEVInt\wlserver\server\bin>startNodeManager.cmd

CLASSPATH=.;C:\PROGRA~1\Java\JDK18~1.0_1\lib\tools.jar;D:\OBIEE12cDEVInt\wlserve

r\server\lib\weblogic.jar;D:\OBIEE12cDEVInt\oracle_common\modules\thirdparty\ant

-contrib-1.0b3.jar;D:\OBIEE12cDEVInt\wlserver\modules\features\oracle.wls.common

.nodemanager.jar;;D:\OBIEE12cDEVInt\wlserver\modules\features\oracle.wls.common.

grizzly.jar

\OBIEE12cDEVInt\oracle_common\common\nodemanager>"C:\PROGRA~1\Java\JDK18~1.0_1

\bin\java.exe" -server -Xms32m -Xmx200m -Djdk.tls.ephemeralDHKeySize=2048 -Dweb

logic.RootDirectory=D:\OBIEE12cDEVInt\oracle_common\common\nodemanager -Dcoheren

ce.home=D:\OBIEE12cDEVInt\coherence -Dbea.home=D:\OBIEE12cDEVInt -Djava.system.c

lass.loader=com.oracle.classloader.weblogic.LaunchClassLoader "-Djava.security.p

olicy=D:\OBIEE12cDEVInt\wlserver\server\lib\weblogic.policy" "-Dweblogic.nodeman

ager.JavaHome=C:\PROGRA~1\Java\JDK18~1.0_1" weblogic.NodeManager -v

<Nov 3, 2017 6:41:42 AM EDT> <INFO> <Loading domains file: \OBIEE12cDEVInt\ora

cle_common\common\nodemanager\nodemanager.domains>

<Nov 3, 2017 6:41:43 AM EDT> <INFO> <Loading identity key store: FileName=D:\OBI

EE12cDEVInt\oracle_common\common\nodemanager\security\DemoIdentity.jks, Type=jks

, PassPhraseUsed=true>

<Nov 3, 2017 6:41:43 AM EDT> <SEVERE> <Fatal error in NodeManager server>

weblogic.nodemanager.common.ConfigException: Identity key store file not found:

\OBIEE12cDEVInt\oracle_common\common\nodemanager\security\DemoIdentity.jks

at weblogic.nodemanager.server.SSLConfig.loadKeyStoreConfig(SSLConfig.ja

va:225)

at weblogic.nodemanager.server.SSLConfig.access$000(SSLConfig.java:33)

at weblogic.nodemanager.server.SSLConfig$1.run(SSLConfig.java:118)

at java.security.AccessController.doPrivileged(Native Method)

at weblogic.nodemanager.server.SSLConfig.<init>(SSLConfig.java:115)

at weblogic.nodemanager.server.NMServer.<init>(NMServer.java:169)

at weblogic.nodemanager.server.NMServer.getInstance(NMServer.java:134)

at weblogic.nodemanager.server.NMServer.main(NMServer.java:589)

at weblogic.NodeManager.main(NodeManager.java:31)

( was unexpected at this time.

My NodeManager.properties file:

#Fri Nov 03 03:11:48 EDT 2017

#Thu Nov 02 04:55:42 EDT 2017

#Wed Nov 01 07:33:09 EDT 2017

#Node manager properties

#Wed Nov 01 07:15:31 EDT 2017

DomainsFile=D\:\\OBIEE12cDEVInt\\user_projects\\domains\\bi\\nodemanager\\nodemanager.domains

LogLimit=0

PropertiesVersion=12.2.1.3.0

AuthenticationEnabled=true

NodeManagerHome=D\:\\OBIEE12cDEVInt\\user_projects\\domains\\bi\\nodemanager

JavaHome=C\:\\PROGRA~1\\Java\\JDK18~1.0_1

LogLevel=INFO

DomainsFileEnabled=true

ListenAddress=COV2OBIEEDEV.am.tsacorp.com

NativeVersionEnabled=true

ListenPort=9506

LogToStderr=true

weblogic.StartScriptName=startWebLogic.cmd

SecureListener=true

LogCount=1

QuitEnabled=false

LogAppend=true

weblogic.StopScriptEnabled=false

StateCheckInterval=500

CrashRecoveryEnabled=false

weblogic.StartScriptEnabled=true

LogFile=D\:\\OBIEE12cDEVInt\\user_projects\\domains\\bi\\nodemanager\\nodemanager.log

LogFormatter=weblogic.nodemanager.server.LogFormatter

ListenBacklog=50

KeyStores=CustomIdentityAndCustomTrust

CustomIdentityKeystoreType=jks

CustomIdentityKeyStoreFileName=D:\\OBIEE12cDEVInt\\ssl\\myIdentity_DEV.jks

CustomIdentityKeyStorePassPhrase={AES}1B/TPs2f8Xo18UjcXaDpMuaLlR5kq7DalGH7T4uTpDw=

CustomIdentityPrivateKeyPassPhrase={AES}FcUsStSIh+fNF18zwg+AhNAEey5mdGzE4MpSeXQ9yRQ=

CustomIdentityAlias=COV2OBIEEDEV

CustomTrustKeystoreType=jks

CustomTrustKeyStoreFileName=D:\\OBIEE12cDEVInt\\ssl\\myTrust_DEV.jks

Question: I am unable to understand why the nodemanager is trying to search the certificate in demoidentity when I am using CustomIdentity and Custum Trust?

Am I missing any file updates/path changes here, please?

Note: Nodemanager is working fine when I am starting it using STARTNODEMANAGER.CMD

I am following the below Oracle link for this:

https://docs.oracle.com/middleware/1221/biee/BIESC/ssl.htm#BIESC6423

Any pointer/documents are appreciated.

Thanks in Advance.

Joel · Nov 3, 2017 10:54AM

Hi @user1137177

Does the \OBIEE12cDEVInt\oracle_common\common\nodemanager\security\DemoIdentity.jks keystore file exist in the location?

mac2 · Nov 3, 2017 11:44AM

It sounds to me like your mistake is probably somewhere in Steps 2-5 of DOC ID 2188982.1. You should review that doc and compare it to what you have configured in your environment. Specifically, I'd check that you are using "Custom Identity and Custom Trust", not "Demo Identity and Demo Trust" in weblogic console in these two places:

Environment -> Servers -> bi_server1 -> Keystores.
Environment -> Servers -> AdminServer -> Keystores.

You should also make sure that your SSL tab is correct in both of those areas as well.

SonPat99 · Nov 3, 2017 1:06PM

Hello Joel,

Yes, the file was missing. I had copied the file to this location.

Question: If I am using CustomIdentityandCustomTrust, why do I still need DemoIdentity?

Hello Mac2,

Yes, I had followed all the steps upto step 3. Step 4 is not applicable and step 5 will be performed post monitoring this for a few days as we have to do this on Prod as well.

Yes, I have selected CustomIdentityandCustomTrust from the drop-downs in the Keystores page.

SonPat99 · Nov 4, 2017 3:11AM

Hello,

I am able to start the services now, however, there is another error:

I am able to start all the services now using START.CMD successfully.

However, when I try to login to CONSOLE, I am facing some issues:

"An unexpected error condition has been detected. Details have been recorded in the admin server log. "

Error in the ADMIN.LOG is as below:

####<Nov 3, 2017 12:26:33,559 PM EDT> <Error> <Console> <COV2OBIEEDEV> <AdminServer> <[ACTIVE] ExecuteThread: '63' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <fe13a5c8-765b-43cb-9f0a-c10a198d5e15-000000a1> <1509726393559> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-240003> <Administration Console encountered the following error: Exception during RequestDispatcher.include().

Throwable: java.lang.InternalError: Unexpected CryptoAPI failure generating seed

Stack Trace:

java.lang.InternalError: Unexpected CryptoAPI failure generating seed

at sun.security.provider.NativeSeedGenerator.getSeedBytes(NativeSeedGenerator.java:62)

at sun.security.provider.SeedGenerator.generateSeed(SeedGenerator.java:144)

at sun.security.provider.SecureRandom.engineGenerateSeed(SecureRandom.java:139)

at java.security.SecureRandom.generateSeed(SecureRandom.java:533)

at com.bea.security.utils.random.AbstractRandomData.ensureInittedAndSeeded(AbstractRandomData.java:92)

at com.bea.security.utils.random.AbstractRandomData.getRandomBytes(AbstractRandomData.java:106)

at com.bea.security.utils.random.AbstractRandomData.getRandomBytes(AbstractRandomData.java:101)

at com.bea.console.utils.CSRFUtils.getSecret(CSRFUtils.java:56)

at jsp_servlet._jsp._changemgmt.__changemanager._jspService(ChangeManager.jsp:20)

at weblogic.servlet.jsp.JspBase.service(JspBase.java:35)

at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:286)

at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:260)

at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:137)

at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:350)

at weblogic.servlet.internal.ServletStubImpl.onAddToMapException(ServletStubImpl.java:489)

at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:376)

at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:247)

at weblogic.servlet.internal.RequestDispatcherImpl.invokeServlet(RequestDispatcherImpl.java:630)

at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:502)

at org.apache.beehive.netui.pageflow.scoping.internal.ScopedRequestDispatcher.include(ScopedRequestDispatcher.java:119)

at com.bea.netuix.servlets.controls.content.JspContent.beginRender(JspContent.java:558)

at com.bea.netuix.servlets.controls.content.NetuiContent.beginRender(NetuiContent.java:365)

at com.bea.netuix.nf.ControlLifecycle$7.visit(ControlLifecycle.java:485)

at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:518)

at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:529)

at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:220)

at com.bea.netuix.nf.Lifecycle.processLifecycles(Lifecycle.java:399)

at com.bea.netuix.nf.Lifecycle.processLifecycles(Lifecycle.java:361)

at com.bea.netuix.nf.Lifecycle.processLifecycles(Lifecycle.java:352)

at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:326)

at com.bea.netuix.nf.UIControl.render(UIControl.java:582)

at com.bea.netuix.servlets.controls.PresentationContext.render(PresentationContext.java:488)

at com.bea.netuix.servlets.util.RenderToolkit.renderChild(RenderToolkit.java:152)

at com.bea.netuix.servlets.jsp.taglib.skeleton.Child.doTag(Child.java:63)

at jsp_servlet._framework._skeletons._wlsconsole.__nolayout._jspx___tag2(__nolayout.java:199)

at jsp_servlet._framework._skeletons._wlsconsole.__nolayout._jspx___tag1(__nolayout.java:155)

at jsp_servlet._framework._skeletons._wlsconsole.__nolayout._jspService(__nolayout.java:94)

at weblogic.servlet.jsp.JspBase.service(JspBase.java:35)

at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:286)

at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:260)

at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:137)

at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:350)

at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:247)

at weblogic.servlet.internal.RequestDispatcherImpl.invokeServlet(RequestDispatcherImpl.java:633)

at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:502)

at com.bea.netuix.servlets.controls.application.laf.JspTools.renderJsp(JspTools.java:148)

at com.bea.netuix.servlets.controls.application.laf.JspControlRenderer.beginRender(JspControlRenderer.java:72)

at com.bea.netuix.servlets.controls.application.laf.PresentationControlRenderer.beginRender(PresentationControlRenderer.java:65)

at com.bea.netuix.nf.ControlLifecycle$7.visit(ControlLifecycle.java:481)

at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:518)

at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:529)

at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:220)

at com.bea.netuix.nf.Lifecycle.processLifecycles(Lifecycle.java:399)

at com.bea.netuix.nf.Lifecycle.processLifecycles(Lifecycle.java:361)

at com.bea.netuix.nf.Lifecycle.processLifecycles(Lifecycle.java:352)

at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:326)

at com.bea.netuix.nf.UIControl.render(UIControl.java:582)

at com.bea.netuix.servlets.controls.PresentationContext.render(PresentationContext.java:488)

at com.bea.netuix.servlets.util.RenderToolkit.renderChild(RenderToolkit.java:152)

at com.bea.netuix.servlets.jsp.taglib.skeleton.Child.doTag(Child.java:63)

at jsp_servlet._framework._skeletons._wlsconsole.__twocollayout._jspx___tag8(__twocollayout.java:452)

at jsp_servlet._framework._skeletons._wlsconsole.__twocollayout._jspx___tag7(__twocollayout.java:418)

at jsp_servlet._framework._skeletons._wlsconsole.__twocollayout._jspx___tag4(__twocollayout.java:308)

at jsp_servlet._framework._skeletons._wlsconsole.__twocollayout._jspx___tag3(__twocollayout.java:263)

at jsp_servlet._framework._skeletons._wlsconsole.__twocollayout._jspx___tag2(__twocollayout.java:209)

at jsp_servlet._framework._skeletons._wlsconsole.__twocollayout._jspx___tag1(__twocollayout.java:163)

at jsp_servlet._framework._skeletons._wlsconsole.__twocollayout._jspService(__twocollayout.java:103)