4 Replies Latest reply: May 15, 2012 1:50 PM by Don Kleppinger RSS

    Rootcause and solution for stuck thread in Live Environment

    863345
      We are using wls 10.3.2 and oracle thin jdbc release 2 version drivers. We have following stuck thread in Live environment. Kindly throw some light on rootcause of stuck thread and ways to resolve stuck thread.

      Here is the stack trace obtained fron managed server log file:

      ]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
      Thread-23 "[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
      jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
      jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:31)
      java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
      java.net.SocketInputStream.read(SocketInputStream.java:107)
      oracle.net.ns.Packet.receive(Packet.java:240)
      oracle.net.ns.DataPacket.receive(DataPacket.java:103)
      oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:222)
      oracle.net.ns.NetInputStream.read(NetInputStream.java:167)
      oracle.net.ns.NetInputStream.read(NetInputStream.java:100)
      oracle.net.ns.NetInputStream.read(NetInputStream.java:85)
      oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:119)
      oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:74)
      oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1117)
      oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1099)
      oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:279)
      oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:175)
      oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:379)
      oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:67)
      oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:849)
      oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1118)
      oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1232)
      oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3553)
      oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3612)
      ^-- Holding lock: oracle.jdbc.driver.T4CConnection@5a897d60[thin lock]
      oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1491)
      weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:116)
      oracle.jbo.server.ViewObjectImpl.getQueryHitCount(ViewObjectImpl.java:4101)
      oracle.jbo.server.ViewObjectImpl.getQueryHitCount(ViewObjectImpl.java:4095)
      oracle.jbo.server.QueryCollection.getEstimatedRowCount(QueryCollection.java:3632)
      ^-- Holding lock: oracle.jbo.JboSyncLock@52d380ba[thin lock]
      oracle.jbo.server.ViewRowSetImpl.getEstimatedRowCount(ViewRowSetImpl.java:2463)
      oracle.jbo.server.ViewObjectImpl.getEstimatedRowCount(ViewObjectImpl.java:9099)
      in.sis.ps.view.backing.fa.brs.PS00220.fetchDetails(PS00220.java:902)
      sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:???)
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:27)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      java.lang.reflect.Method.invoke(Method.java:575)
      com.sun.el.parser.AstValue.invoke(AstValue.java:153)
      com.sun.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:281)
      org.apache.myfaces.trinidad.component.MethodExpressionMethodBinding.invoke(MethodExpressionMethodBinding.java:46)
      com.sun.faces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:85)
      org.apache.myfaces.trinidad.component.UIXCommand.broadcast(UIXCommand.java:171)
      oracle.adf.view.rich.component.fragment.ContextSwitchingComponent$1.run(ContextSwitchingComponent.java:90)
      oracle.adf.view.rich.component.fragment.ContextSwitchingComponent._processPhase(ContextSwitchingComponent.java:300)
      oracle.adf.view.rich.component.fragment.ContextSwitchingComponent.broadcast(ContextSwitchingComponent.java:82)
      oracle.adf.view.rich.component.fragment.UIXInclude.broadcast(UIXInclude.java:96)
      oracle.adf.view.rich.component.fragment.ContextSwitchingComponent$1.run(ContextSwitchingComponent.java:90)
      oracle.adf.view.rich.component.fragment.ContextSwitchingComponent._processPhase(ContextSwitchingComponent.java:300)
      oracle.adf.view.rich.component.fragment.ContextSwitchingComponent.broadcast(ContextSwitchingComponent.java:82)
      oracle.adf.view.rich.component.fragment.UIXInclude.broadcast(UIXInclude.java:96)
      javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:420)
      javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:751)
      oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._invokeApplication(LifecycleImpl.java:697)
      oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._executePhase(LifecycleImpl.java:244)
      oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:157)
      javax.faces.webapp.FacesServlet.service(FacesServlet.java:246)
      weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
      weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
      weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:198)
      weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
      oracle.adf.model.servlet.ADFBindingFilter.doFilter(ADFBindingFilter.java:134)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
      oracle.adfinternal.view.faces.webapp.rich.RegistrationFilter.doFilter(RegistrationFilter.java:69)
      org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:411)
      oracle.adfinternal.view.faces.activedata.AdsFilter.doFilter(AdsFilter.java:44)
      org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:411)
      org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:210)
      org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:126)
      org.apache.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:91)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
      oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:94)
      oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
      oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:405)
      oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:82)
      oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:69)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
      oracle.dms.wls.DMSServletFilter.doFilter(DMSServletFilter.java:111)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
      weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:25)
      weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
      weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3564)
      weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
      weblogic.security.service.SecurityManager.runAs(SecurityManager.java:117)
      weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2138)
      weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2060)
      weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1399)
      weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
      weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
        • 1. Re: Rootcause and solution for stuck thread in Live Environment
          René van Wijk
          Holding lock: oracle.jdbc.driver.T4CConnection@5a897d60[thin lock]

          Can you check in your database what is happening? When this occurs - maybe some procedures are running
          or is it just a very complex query that is executed?
          • 2. Re: Rootcause and solution for stuck thread in Live Environment
            863345
            We have another stuck thread in Live today in different application module.

            We are using wls 10.3.2 and oracle database 11gR2. Here is the stack trace of stuck thread occured. Kindly help out in finding the rootcause and sol to avoid re-occurance.

            ]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
            Thread-38 "[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
            jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
            jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:31)
            java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
            java.net.SocketInputStream.read(SocketInputStream.java:107)
            oracle.net.ns.Packet.receive(Packet.java:240)
            oracle.net.ns.DataPacket.receive(DataPacket.java:103)
            oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:222)
            oracle.net.ns.NetInputStream.read(NetInputStream.java:167)
            oracle.net.ns.NetInputStream.read(NetInputStream.java:100)
            oracle.net.ns.NetInputStream.read(NetInputStream.java:85)
            oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:119)
            oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:74)
            oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1117)
            oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1099)
            oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:279)
            oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:175)
            oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:379)
            oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:67)
            oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:948)
            oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1118)
            oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1232)
            oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3553)
            oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3612)
            ^-- Holding lock: oracle.jdbc.driver.T4CConnection@53a2c05e[thin lock]
            oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1491)
            weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:116)
            oracle.jbo.server.ViewObjectImpl.getQueryHitCount(ViewObjectImpl.java:4101)
            oracle.jbo.server.ViewObjectImpl.getQueryHitCount(ViewObjectImpl.java:4095)
            oracle.jbo.server.QueryCollection.getEstimatedRowCount(QueryCollection.java:3632)
            ^-- Holding lock: oracle.jbo.JboSyncLock@6aea788b[thin lock]
            oracle.jbo.server.ViewRowSetImpl.getEstimatedRowCount(ViewRowSetImpl.java:2463)
            oracle.jbo.server.ViewObjectImpl.getEstimatedRowCount(ViewObjectImpl.java:9099)
            in.sis.ps.view.backing.fa.brs.PS00220.fetchDetails(PS00220.java:902)
            sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:???)
            sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:27)
            sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            java.lang.reflect.Method.invoke(Method.java:575)
            com.sun.el.parser.AstValue.invoke(AstValue.java:153)
            com.sun.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:281)
            org.apache.myfaces.trinidad.component.MethodExpressionMethodBinding.invoke(MethodExpressionMethodBinding.java:46)
            com.sun.faces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:85)
            org.apache.myfaces.trinidad.component.UIXCommand.broadcast(UIXCommand.java:171)
            oracle.adf.view.rich.component.fragment.ContextSwitchingComponent$1.run(ContextSwitchingComponent.java:90)
            oracle.adf.view.rich.component.fragment.ContextSwitchingComponent._processPhase(ContextSwitchingComponent.java:300)
            oracle.adf.view.rich.component.fragment.ContextSwitchingComponent.broadcast(ContextSwitchingComponent.java:82)
            oracle.adf.view.rich.component.fragment.UIXInclude.broadcast(UIXInclude.java:96)
            oracle.adf.view.rich.component.fragment.ContextSwitchingComponent$1.run(ContextSwitchingComponent.java:90)
            oracle.adf.view.rich.component.fragment.ContextSwitchingComponent._processPhase(ContextSwitchingComponent.java:300)
            oracle.adf.view.rich.component.fragment.ContextSwitchingComponent.broadcast(ContextSwitchingComponent.java:82)
            oracle.adf.view.rich.component.fragment.UIXInclude.broadcast(UIXInclude.java:96)
            javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:420)
            javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:751)
            oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._invokeApplication(LifecycleImpl.java:697)
            oracle.adfinternal.view.faces.lifecycle.LifecycleImpl._executePhase(LifecycleImpl.java:244)
            oracle.adfinternal.view.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:157)
            javax.faces.webapp.FacesServlet.service(FacesServlet.java:246)
            weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
            weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
            weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:198)
            weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
            weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
            oracle.adf.model.servlet.ADFBindingFilter.doFilter(ADFBindingFilter.java:134)
            weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
            oracle.adfinternal.view.faces.webapp.rich.RegistrationFilter.doFilter(RegistrationFilter.java:69)
            org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:411)
            oracle.adfinternal.view.faces.activedata.AdsFilter.doFilter(AdsFilter.java:44)
            org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl$FilterListChain.doFilter(TrinidadFilterImpl.java:411)
            org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl._doFilterImpl(TrinidadFilterImpl.java:210)
            org.apache.myfaces.trinidadinternal.webapp.TrinidadFilterImpl.doFilter(TrinidadFilterImpl.java:126)
            org.apache.myfaces.trinidad.webapp.TrinidadFilter.doFilter(TrinidadFilter.java:91)
            weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
            oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:94)
            oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:299)
            oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:405)
            oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:82)
            oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:69)
            weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
            oracle.dms.wls.DMSServletFilter.doFilter(DMSServletFilter.java:111)
            weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
            weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:25)
            weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
            weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3564)
            weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
            weblogic.security.service.SecurityManager.runAs(SecurityManager.java:117)
            weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2138)
            weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2060)
            weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1399)
            weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
            weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
            • 3. Re: Rootcause and solution for stuck thread in Live Environment
              René van Wijk
              Same issue: -- Holding lock: oracle.jdbc.driver.T4CConnection@53a2c05e[thin lock] see above
              • 4. Re: Rootcause and solution for stuck thread in Live Environment
                Don Kleppinger
                I'm seeing the same problem with one query.

                getEstimatedRowCount (as shown in the stack trace) is usually called after the query has successfully executed. This values is used to scale the scrollbar on ADF Tables. This wraps the oriiginal query in a select count(*) from {original query}. The count query doesn't seem to ever timeout like the original query does and produces the stuck thread.