9 Replies Latest reply on Apr 19, 2017 9:46 AM by saketsrv

    OBI 12C Weblogic and Em not opening. Possibly due to stuck threads

    saketsrv

      I have recently configured obi 12c application ver. 12.21.2.0 with two domains.One of the them is working fine.but the 2nd one..i am trying to open weblogic and Em console and its keeps loading forever( though it had opened few days before while setting up the instance).

       

      Also unable to stop the domain

      Below is Admin log snippet taken from below path:

      /home/oraclebi/Oracle/Middleware/Oracle_Home/user_projects/domains/clubitc/servers/AdminServer/logs

       

       

      ####<Apr 5, 2017, 5:25:10,866 PM IST> <Error> <WebLogicServer> <ubiuat> <AdminServer> <[ACTIVE] ExecuteThread: '34' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <6d2e862a-4230-4925-bde2-cb5c8f501cd2-00002e16> <1491393310866> <[severity-value: 8] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-000337> <[STUCK] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "621" seconds working on the request "javax.management.remote.rmi.RMIConnectionImpl", which is more than the configured time (StuckThreadMaxTime) of "600" seconds in "server-failure-trigger". Stack trace:

        java.lang.Object.wait(Native Method)

        weblogic.rjvm.ResponseImpl.waitForDataInternal(ResponseImpl.java:115)

        weblogic.rjvm.ResponseImpl.waitForData(ResponseImpl.java:99)

        weblogic.rjvm.ResponseImpl.getTxContext(ResponseImpl.java:162)

        weblogic.rjvm.BasicOutboundRequest.sendReceive(BasicOutboundRequest.java:129)

        weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:299)

        weblogic.server.RemoteLifeCycleOperationsImpl_12212_WLStub.getState(Unknown Source)

        weblogic.server.ServerLifeCycleRuntime$3.run(ServerLifeCycleRuntime.java:891)

        weblogic.server.ServerLifeCycleRuntime$3.run(ServerLifeCycleRuntime.java:889)

        weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:368)

        weblogic.security.service.SecurityManager.runAs(SecurityManager.java:163)

        weblogic.server.ServerLifeCycleRuntime.executePrivilegedAction(ServerLifeCycleRuntime.java:788)

        weblogic.server.ServerLifeCycleRuntime.getStateRemote(ServerLifeCycleRuntime.java:888)

        weblogic.server.ServerLifeCycleRuntime.getState(ServerLifeCycleRuntime.java:716)

        sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        java.lang.reflect.Method.invoke(Method.java:498)

        weblogic.management.jmx.modelmbean.WLSModelMBean.getAttribute(WLSModelMBean.java:619)

        com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)

        com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)

        weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$17.run(WLSMBeanServerInterceptorBase.java:466)

        java.security.AccessController.doPrivileged(Native Method)

        weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.getAttribute(WLSMBeanServerInterceptorBase.java:464)

        weblogic.management.mbeanservers.internal.JMXContextInterceptor.getAttribute(JMXContextInterceptor.java:157)

        weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$17.run(WLSMBeanServerInterceptorBase.java:466)

        java.security.AccessController.doPrivileged(Native Method)

        weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.getAttribute(WLSMBeanServerInterceptorBase.java:464)

        weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$17.run(WLSMBeanServerInterceptorBase.java:466)

        java.security.AccessController.doPrivileged(Native Method)

        weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.getAttribute(WLSMBeanServerInterceptorBase.java:464)

        weblogic.management.mbeanservers.internal.SecurityInterceptor.getAttribute(SecurityInterceptor.java:294)

        weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$17.run(WLSMBeanServerInterceptorBase.java:466)

        java.security.AccessController.doPrivileged(Native Method)

        weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.getAttribute(WLSMBeanServerInterceptorBase.java:464)

        weblogic.management.mbeanservers.internal.MBeanCICInterceptor.getAttribute(MBeanCICInterceptor.java:139)

        weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$17.run(WLSMBeanServerInterceptorBase.java:466)

        java.security.AccessController.doPrivileged(Native Method)

        weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.getAttribute(WLSMBeanServerInterceptorBase.java:464)

        weblogic.management.mbeanservers.internal.PartitionJMXInterceptor.getAttribute(PartitionJMXInterceptor.java:303)

        weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.getAttribute(FederatedMBeanServerInterceptor.java:314)

        weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$17.run(WLSMBeanServerInterceptorBase.java:466)

        java.security.AccessController.doPrivileged(Native Method)

        weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.getAttribute(WLSMBeanServerInterceptorBase.java:464)

        weblogic.management.mbeanservers.internal.CallerPartitionContextInterceptor.getAttribute(CallerPartitionContextInterceptor.java:177)

        weblogic.management.jmx.mbeanserver.WLSMBeanServer.getAttribute(WLSMBeanServer.java:283)

        weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder$5$1.run(JMXConnectorSubjectForwarder.java:308)

        java.security.AccessController.doPrivileged(Native Method)

        weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder$5.run(JMXConnectorSubjectForwarder.java:306)

        weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:368)

        weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder.getAttribute(JMXConnectorSubjectForwarder.java:301)

        javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1445)

        javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)

        javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)

        java.security.AccessController.doPrivileged(Native Method)

        javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1408)

        javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:639)

        javax.management.remote.rmi.RMIConnectionImpl_WLSkel.invoke(Unknown Source)

        weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:645)

        weblogic.rmi.internal.BasicServerRef$2.run(BasicServerRef.java:534)

        weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:368)

        weblogic.security.service.SecurityManager.runAs(SecurityManager.java:163)

        weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:531)

        weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:138)

        weblogic.invocation.ComponentInvocationContextManager._runAs(ComponentInvocationContextManager.java:348)

        weblogic.invocation.ComponentInvocationContextManager.runAs(ComponentInvocationContextManager.java:333)

        weblogic.work.LivePartitionUtility.doRunWorkUnderContext(LivePartitionUtility.java:54)

        weblogic.work.PartitionUtility.runWorkUnderContext(PartitionUtility.java:41)

        weblogic.work.SelfTuningWorkManagerImpl.runWorkUnderContext(SelfTuningWorkManagerImpl.java:640)

        weblogic.work.ExecuteThread.execute(ExecuteThread.java:406)

        weblogic.work.ExecuteThread.run(ExecuteThread.java:346)

       

      Let me know if anyone has faced the same problem and the resolution.