Oracle Analytics Cloud and Server

Home› Oracle Analytics› Oracle Analytics Forums› Oracle Analytics Cloud and Server› Oracle Analytics Cloud and Server

Categories

Welcome to the Oracle Analytics Community: Please complete your User Profile and upload your Profile Picture

OBI 12C Weblogic and Em not opening. Possibly due to stuck threads

Received Response
131
Views
9
Comments
saketsrv
saketsrv Rank 5 - Community Champion
edited August 2024 in Oracle Analytics Cloud and Server

I have recently configured obi 12c application ver. 12.21.2.0 with two domains.One of the them is working fine.but the 2nd one..i am trying to open weblogic and Em console and its keeps loading forever( though it had opened few days before while setting up the instance).

Also unable to stop the domain

Below is Admin log snippet taken from below path:

/home/oraclebi/Oracle/Middleware/Oracle_Home/user_projects/domains/clubitc/servers/AdminServer/logs

####<Apr 5, 2017, 5:25:10,866 PM IST> <Error> <WebLogicServer> <ubiuat> <AdminServer> <[ACTIVE] ExecuteThread: '34' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <6d2e862a-4230-4925-bde2-cb5c8f501cd2-00002e16> <1491393310866> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000337> <[STUCK] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "621" seconds working on the request "javax.management.remote.rmi.RMIConnectionImpl", which is more than the configured time (StuckThreadMaxTime) of "600" seconds in "server-failure-trigger". Stack trace:

  java.lang.Object.wait(Native Method)

  weblogic.rjvm.ResponseImpl.waitForDataInternal(ResponseImpl.java:115)

  weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:99)

  weblogic.rjvm.ResponseImpl.getTxContext(ResponseImpl.java:162)

  weblogic.rjvm.BasicOutboundRequest.sendReceive(BasicOutboundRequest.java:129)

  weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:299)

  weblogic.server.RemoteLifeCycleOperationsImpl_12212_WLStub.getState(Unknown Source)

  weblogic.server.ServerLifeCycleRuntime$3.run(ServerLifeCycleRuntime.java:891)

  weblogic.server.ServerLifeCycleRuntime$3.run(ServerLifeCycleRuntime.java:889)

  weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:368)

  weblogic.security.service.SecurityManager.runAs(SecurityManager.java:163)

  weblogic.server.ServerLifeCycleRuntime.executePrivilegedAction(ServerLifeCycleRuntime.java:788)

  weblogic.server.ServerLifeCycleRuntime.getStateRemote(ServerLifeCycleRuntime.java:888)

  weblogic.server.ServerLifeCycleRuntime.getState(ServerLifeCycleRuntime.java:716)

  sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

  sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

  sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

  java.lang.reflect.Method.invoke(Method.java:498)

  weblogic.management.jmx.modelmbean.WLSModelMBean.getAttribute(WLSModelMBean.java:619)

  com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)

  com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)

  weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$17.run(WLSMBeanServerInterceptorBase.java:466)

  java.security.AccessController.doPrivileged(Native Method)

  weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.getAttribute(WLSMBeanServerInterceptorBase.java:464)

  weblogic.management.mbeanservers.internal.JMXContextInterceptor.getAttribute(JMXContextInterceptor.java:157)

  weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$17.run(WLSMBeanServerInterceptorBase.java:466)

  java.security.AccessController.doPrivileged(Native Method)

  weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.getAttribute(WLSMBeanServerInterceptorBase.java:464)

  weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$17.run(WLSMBeanServerInterceptorBase.java:466)

  java.security.AccessController.doPrivileged(Native Method)

  weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.getAttribute(WLSMBeanServerInterceptorBase.java:464)

  weblogic.management.mbeanservers.internal.SecurityInterceptor.getAttribute(SecurityInterceptor.java:294)

  weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$17.run(WLSMBeanServerInterceptorBase.java:466)

  java.security.AccessController.doPrivileged(Native Method)

  weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.getAttribute(WLSMBeanServerInterceptorBase.java:464)

  weblogic.management.mbeanservers.internal.MBeanCICInterceptor.getAttribute(MBeanCICInterceptor.java:139)

  weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$17.run(WLSMBeanServerInterceptorBase.java:466)

  java.security.AccessController.doPrivileged(Native Method)

  weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.getAttribute(WLSMBeanServerInterceptorBase.java:464)

  weblogic.management.mbeanservers.internal.PartitionJMXInterceptor.getAttribute(PartitionJMXInterceptor.java:303)

  weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.getAttribute(FederatedMBeanServerInterceptor.java:314)

  weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$17.run(WLSMBeanServerInterceptorBase.java:466)

  java.security.AccessController.doPrivileged(Native Method)

  weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.getAttribute(WLSMBeanServerInterceptorBase.java:464)

  weblogic.management.mbeanservers.internal.CallerPartitionContextInterceptor.getAttribute(CallerPartitionContextInterceptor.java:177)

  weblogic.management.jmx.mbeanserver.WLSMBeanServer.getAttribute(WLSMBeanServer.java:283)

  weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder$5$1.run(JMXConnectorSubjectForwarder.java:308)

  java.security.AccessController.doPrivileged(Native Method)

  weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder$5.run(JMXConnectorSubjectForwarder.java:306)

  weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:368)

  weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder.getAttribute(JMXConnectorSubjectForwarder.java:301)

  javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1445)

  javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)

  javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)

  java.security.AccessController.doPrivileged(Native Method)

  javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1408)

  javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:639)

  javax.management.remote.rmi.RMIConnectionImpl_WLSkel.invoke(Unknown Source)

  weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:645)

  weblogic.rmi.internal.BasicServerRef$2.run(BasicServerRef.java:534)

  weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:368)

  weblogic.security.service.SecurityManager.runAs(SecurityManager.java:163)

  weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:531)

  weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:138)

  weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:348)

  weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:333)

  weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:54)

  weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)

  weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:640)

  weblogic.work.ExecuteThread.execute(ExecuteThread.java:406)

  weblogic.work.ExecuteThread.run(ExecuteThread.java:346)

Let me know if anyone has faced the same problem and the resolution.

Answers

  • Prashant Umap
    Prashant Umap Rank 3 - Community Apprentice

    Hello Saket,

    Can you check the JVM value for 2nd Node where you are having an issue. If its less try to increase and check.

    Regards

    Prashant Umap

  • Prashant Umap
    Prashant Umap Rank 3 - Community Apprentice

    Hello Saket,

    Can you check opmn services on the server ?

    Ex. ./$MW_HOME/instance/instance1/bin/opmnctl status

    Regards,

    Prashant Umap

  • Gianni Ceresa
    Gianni Ceresa Mod

    The OP wrote:

    saketsrv wrote:I have recently configured obi 12c application ver. 12.21.2.0

    There is no opmnctl anymore in 12c, everything is done via the bitools/bin/status.sh command.

  • Prashant Umap
    Prashant Umap Rank 3 - Community Apprentice

    Thank you Gianni for the correction.

  • saketsrv
    saketsrv Rank 5 - Community Champion

    HI Prashant,

    Please look at the status below.It stops after displaying AdimiSever status.

    [obiee@ubiuat:/home/oraclebi/Oracle/Middleware/Oracle_Home/user_projects/domains/clubitc/bitools/bin]$./status.sh

    Domain status; Using domainHome: /home/oraclebi/Oracle/Middleware/Oracle_Home/user_projects/domains/clubitc ...

    Initializing WebLogic Scripting Tool (WLST) ...

    Welcome to WebLogic Server Administration Scripting Shell

    Type help() for help on available commands

    <Apr 6, 2017 10:41:30 AM IST> <Info> <Security> <BEA-090905> <Disabling the CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true.>

    <Apr 6, 2017 10:41:30 AM IST> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG128 to HMACDRBG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true.>

    <Apr 6, 2017 10:41:32 AM IST> <Info> <Security> <BEA-090909> <Using the configured custom SSL Hostname Verifier implementation: weblogic.security.utils.SSLWLSHostnameVerifier$NullHostnameVerifier.>

    /Servers/AdminServer/ListenPort=9504

    Accessing admin server using URL t3://ubiuat:9504

    Status of Domain: /home/oraclebi/Oracle/Middleware/Oracle_Home/user_projects/domains/clubitc

    NodeManager (ubiuat:9519): RUNNING

    Name            Type            Machine                   Status

    ----            ----            -------                   ------

    AdminServer     Server          ubiuat                    RUNNING

  • Prashant Umap
    Prashant Umap Rank 3 - Community Apprentice

    It looks like - Weblogic Admin serve is up and running. Can you check and login.

    Also check the EM status.

    Prashant Umap

  • saketsrv
    saketsrv Rank 5 - Community Champion

    Hi Prashant,

    Weblogic Admin and em was up the whole time, i never said it was down but after login,page showed up blank and kept on loading (in both).

    Anyways i had to restart the application by killing all the running java processes and everything worked fine thereon.(initially i was refraining myself to do this and rather wanted the root cause and the fix but i was running out of time).

    Also,Any inputs on what caused this and how to find the resolution  would be helpful if anyone encounters the same issue in future.

    Thanks !!

  • Prashant Umap
    Prashant Umap Rank 3 - Community Apprentice

    Thanks Saket for the clarification.

    Always you can check the weblogic logs, where the process got stucked and caused the issue.

    Let me know if you have more questions on this topic.

    Regards

    Prashant Umap

  • saketsrv
    saketsrv Rank 5 - Community Champion

    Hello Everyone,

    Just to update the result.it turns out the Java process was choking the server memory utilization and causing the stuck threads since i have have two managed servers running on the same machine .Tried to tune the JVM as well but that didn't go well either.

    Finally decided to increase the Memory and it worked.

    Thanks again for all your cooperation.!!