5 Replies Latest reply: Feb 4, 2011 1:54 PM by PHCharbonneau RSS

    #STUCK THREAD#

    746539
      Hello guys!

      We have a weblogic server that is getting stuck thread as its load increases, but the threads in stuck state don't say much. Looking at the logs, I see that threads are stuck while doing many different operations and they get stuck while performing some DB operations, I/O and some others. I don't believe that we have an issue regarding our connection pool, since we don't get connections waiting or failed. Do you guys have any clue about what is going on in my environment?

      Thanks in advance for your help.

      <Apr 17, 2010 1:52:27 PM BRT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '203' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "628" seconds working on the request "Http Request: /portal/site/PortalWAP", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
      Thread-312 "[STUCK] ExecuteThread: '203' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
      jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
      jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:25)
      java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
      java.net.SocketInputStream.read(SocketInputStream.java:107)
      oracle.net.ns.Packet.receive(Unknown Source)
      oracle.net.ns.DataPacket.receive(Unknown Source)
      oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
      oracle.net.ns.NetInputStream.read(Unknown Source)
      oracle.net.ns.NetInputStream.read(Unknown Source)
      oracle.net.ns.NetInputStream.read(Unknown Source)
      oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1093)
      oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1070)
      oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:449)
      oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:65)
      oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:785)
      oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1004)
      oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:837)
      oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1095)
      oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3295)
      oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3354)
      ^-- Holding lock: oracle.jdbc.driver.T4CConnection@52c05b92[thin lock]
      ^-- Holding lock: oracle.jdbc.driver.T4CPreparedStatement@72f1fb18[thin lock]
      weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement.java:88)
      com.vignette.dbms.SpyPreparedStatement.executeQuery(SpyPreparedStatement.java:54)
      com.vignette.dbms.SpyPreparedStatement.executeQuery(SpyPreparedStatement.java:132)
      com.vignette.as.server.persist.AsObjectSqlQueryBase.executeMultiValuedSqlQuery(AsObjectSqlQueryBase.java:1691)
      com.vignette.as.server.persist.AsObjectSqlQueryBase.executeSqlQuery(AsObjectSqlQueryBase.java:1822)
      com.vignette.as.server.persist.ManagedObjectSqlQuery.executeSqlQuery(ManagedObjectSqlQuery.java:1359)
      com.vignette.as.server.logic.query.QueryManagerOpsLogic.executeQuery(QueryManagerOpsLogic.java:211)
      com.vignette.as.server.logic.query.QueryResultPagingList.subList(QueryResultPagingList.java:89)
      com.vignette.as.server.logic.query.QueryResultPagingList.asList(QueryResultPagingList.java:130)
      com.vignette.as.server.logic.common.RemotablePagingListDispatcherOpsLogic.invokeAsList(RemotablePagingListDispatcherOpsLogic.java:128)
      com.vignette.as.server.logic.common.RemotablePagingListDispatcherOpsLogicGen.invokeAsList(RemotablePagingListDispatcherOpsLogicGen.java:267)
      com.vignette.as.client.javabean.RemotablePagingList.asList(RemotablePagingList.java:182)
      com.accenture.portaltim.portlets.googlesearch.utils.VCMContentManager.getContents(VCMContentManager.java:58)

      <Apr 17, 2010 1:52:27 PM BRT> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '199' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "627" seconds working on the request "Http Request: /portal/site/PortalWAP", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
      Thread-308 "[STUCK] ExecuteThread: '199' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, priority=1, DAEMON> {
      weblogic.common.resourcepool.ResourcePoolImpl.suspectHang(ResourcePoolImpl.java:2053)
      weblogic.common.resourcepool.ResourcePoolImpl.checkHang(ResourcePoolImpl.java:2021)
      weblogic.common.resourcepool.ResourcePoolImpl.checkResource(ResourcePoolImpl.java:1477)
      weblogic.common.resourcepool.ResourcePoolImpl.checkAndReturnResource(ResourcePoolImpl.java:1395)
      weblogic.common.resourcepool.ResourcePoolImpl.checkAndReturnResource(ResourcePoolImpl.java:1388)
      weblogic.common.resourcepool.ResourcePoolImpl.reserveResourceInternal(ResourcePoolImpl.java:325)
      weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:299)
      weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:292)
      weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:322)
      weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:316)
      weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:85)
      weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:106)
      weblogic.jdbc.pool.Driver.connect(Driver.java:76)
      weblogic.jdbc.jts.Driver.getNonTxConnection(Driver.java:622)
      weblogic.jdbc.jts.Driver.connect(Driver.java:101)
      weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:337)
      com.vignette.as.config.ConfigUtil.getConnection(ConfigUtil.java:1316)
      com.vignette.as.config.ConfigUtil.getASConnection(ConfigUtil.java:1453)
      com.vignette.as.server.persist.ContentInstancePersistor.populateCategoriesAndChannels(ContentInstancePersistor.java:4051)
      com.vignette.as.server.persist.ContentInstancePersistor.exceptionResistantPopulateContentInstances(ContentInstancePersistor.java:640)
      com.vignette.as.server.persist.ContentInstancePersistor.findByIds(ContentInstancePersistor.java:558)
      com.vignette.as.server.persist.FindByManagedObjectRefsPagingList.findByIds(FindByManagedObjectRefsPagingList.java:66)
      com.vignette.as.server.persist.FindByIdsPagingList.subList(FindByIdsPagingList.java:76)
      com.vignette.as.client.javabean.PagingList.subList(PagingList.java:126)
      com.vignette.as.server.logic.common.RemotablePagingListDispatcherOpsLogic.invokeSubList(RemotablePagingListDispatcherOpsLogic.java:86)
      com.vignette.as.server.logic.common.RemotablePagingListDispatcherOpsLogicGen.invokeSubList(RemotablePagingListDispatcherOpsLogicGen.java:424)
      com.vignette.as.client.javabean.RemotablePagingList.subList(RemotablePagingList.java:138)
      com.vignette.as.server.persist.ManagedObjectPersistor.findByIds(ManagedObjectPersistor.java:1243)
      com.vignette.as.server.logic.common.ManagedObjectOpsLogic.findByContentManagementId(ManagedObjectOpsLogic.java:433)
      com.vignette.as.server.logic.common.ManagedObjectOpsLogicGen.findByContentManagementId(ManagedObjectOpsLogicGen.java:855)
      com.vignette.as.client.javabean.ManagedObject.findByContentManagementId(ManagedObject.java:1580)
      com.vignette.as.client.common.ref.ManagedObjectVCMRef.retrieveManagedObject(ManagedObjectVCMRef.java:150)
      com.accenture.pwap.vcm.util.ManagedObjectUtil.findManagedObjectByVCMRefId(ManagedObjectUtil.java:32)
      com.accenture.pwap.dps.util.presentation.PresentationHelper.loadActiveSubChannels(PresentationHelper.java:616)
      6650,5 7%
        • 1. Re: #STUCK THREAD#
          René van Wijk
          You probably already did the following, but could you check to following parameters in the monitoring tab of your connection pool

          - Active Connections High Count tells you the maximum number of connections reached in the pool at any time since the server started. By comparing this number with your pool's Initial Capacity and Maximum Capacity settings, you can determine if the pool is sized properly for the application load the server has experienced.

          - Waiting for Connection High Count value tells you the maximum number of threads that were waiting (at any particular point in time) to get a database connection from the pool because there were no connections available.

          - Wait Seconds High Count tells you the longest time a thread had to wait to get a connection.


          Are there any methods which perform some SQL statement which is taking a long time?

          It could also be a problem with TCP/IP , that a TCP connection which is closed will go into TIME_WAIT state before the operating system discards it. Maybe all your TCP/IP ports are used up in some point in time, when the server gets extremely busy. On most unix system you can use netstat to check the number of TCP connections in TIME_WAIT state, for example, netstat –a | grep TIME_WAIT | wc –l


          These are just some points you can check upon - maybe it helps
          • 2. Re: #STUCK THREAD#
            751846
            Hi,

            Could you please let me know how to get Active Connections High Count & Waiting for Connection High Count in Weblogic 8 & 10. I am not able to find these values in Connetion pool monitoring tab.

            Thanks,
            Sanjeev
            • 3. Re: #STUCK THREAD#
              746539
              In the monitoring tab. Try to find it by clicking on customize this table.

              I hope this helps.
              • 4. Re: #STUCK THREAD#
                774544
                Most of the cases, stuck threads caused by a thread waiting for the database to perform its work. Make sure database is properly tuned and has good capacity to handle the requests. Did you do thread dumps ? Please take thread dumps and find for your own classes mentioned inthe thread dumps. Looks like from the stack trace, the problem might come from

                1.
                weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:337)
                com.vignette.as.config.ConfigUtil.getConnection(ConfigUtil.java:1316)
                com.vignette.as.config.ConfigUtil.getASConnection(ConfigUtil.java:1453)

                2. Why is this holding the lock, please consult the db admin to find out why.
                oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3354)
                ^-- Holding lock: oracle.jdbc.driver.T4CConnection@52c05b92[thin lock]
                ^-- Holding lock: oracle.jdbc.driver.T4CPreparedStatement@72f1fb18[thin lock]
                weblogic.jdbc.wrapper.PreparedStatement.executeQuery(PreparedStatement
                • 5. Re: #STUCK THREAD#
                  PHCharbonneau
                  Hi user10455324,

                  Your thread snapshot is potentially showing a long running query from your Oracle database. This is why you see the actual JDBC driver socket haning in a Socket.read() operation. This means that the Weblogic thread is simply waiting from a response from the database, for that particular SQL query.

                  Please the following to investigate further the root cause:

                  1) Capture a few thread dump and identify proper pattern(s) of your application e.g. which application code/SQL is causing you the problem
                  2) Generate an Oracle AWR report for the same time interval of the Thread dump and correlate any long running query with your application code/third part tool

                  Long running SQL can be anything from bad SQL design to lack of proper indexes or non optimal Oracle execution plan.


                  Regards,
                  P-H
                  http://javaeesupportpatterns.blogspot.com/