4 Replies Latest reply: Jan 10, 2012 3:24 PM by the_assface RSS

    Stuck Thread

    the_assface
      Getting some occasional stuck threads on my portal servers. Apparently not affecting application as it seems to be up and running. However, the only way to clear the warnings in the console is via an app server restart. Not ideal since this is production. Any clues on from this thread dump on what the issue may be? Or even better, what a potential fix may be?

      Thanks

      (I had to cut out some of the dump due to 30,000 character limit)


      ===== FULL THREAD DUMP ===============



      "[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" id=42 idx=0xac tid=776 prio=1 alive, in native, daemon

      at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)
               
                      at jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:46)[optimized]

      at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)

      at java/net/SocketInputStream.read(SocketInputStream.java:129)

      at java/io/BufferedInputStream.fill(BufferedInputStream.java:218)[inlined]

      at java/io/BufferedInputStream.read(BufferedInputStream.java:237)[optimized]

      ^-- Holding lock: java/io/BufferedInputStream@0x000000001B6B2020[biased lock]

      at weblogic/net/http/MessageHeader.isHTTP(MessageHeader.java:220)

      at weblogic/net/http/MessageHeader.parseHeader(MessageHeader.java:143)[optimized]

      at weblogic/net/http/HttpClient.parseHTTP(HttpClient.java:454)[optimized]

      ^-- Holding lock: weblogic/net/http/HttpClient@0x000000001B6B20A0[biased lock]

      at weblogic/net/http/HttpURLConnection.getInputStream(HttpURLConnection.java:364)[optimized]

      ^-- Holding lock: weblogic/net/http/SOAPHttpURLConnection@0x000000001B6B21F8[biased lock]

      at weblogic/net/http/SOAPHttpURLConnection.getInputStream(SOAPHttpURLConnection.java:36)

      at weblogic/net/http/HttpURLConnection.getResponseCode(HttpURLConnection.java:952)

      at com/fc/logic/product/Navs.getNavs(Navs.java:224)

      at com/fc/logic/product/Navs.getNavs(Navs.java:138)

      at com/fc/logic/investments/FundRetriever.retrieveStock(FundRetriever.java:269)

      at com/fc/logic/investments/FundRetriever.getStockQuote(FundRetriever.java:118)

      at com/fc/logic/investments/NavsFundMgr.getStockQuote(NavsFundMgr.java:39)

      at portlets/OtherInvestmentSummary/OtherInvestmentSummaryController.getFundValue(OtherInvestmentSummaryController.java:1509)

      at portlets/OtherInvestmentSummary/OtherInvestmentSummaryController.loadOtherInvestmentSummaries(OtherInvestmentSummaryController.java:180)

      at portlets/OtherInvestmentSummary/OtherInvestmentSummaryController.begin(OtherInvestmentSummaryController.java:429)

      at portlets/OtherInvestmentSummary/OtherInvestmentSummaryController.AfterAdd(OtherInvestmentSummaryController.java:1191)

      at sun/reflect/GeneratedMethodAccessor865.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
               
                      at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]

      at java/lang/reflect/Method.invoke(Method.java:597)[optimized]

      at org/apache/beehive/netui/pageflow/FlowController.invokeActionMethod(FlowController.java:870)

      at org/apache/beehive/netui/pageflow/FlowController.getActionMethodForward(FlowController.java:809)

      at org/apache/beehive/netui/pageflow/FlowController.internalExecute(FlowController.java:478)

      at org/apache/beehive/netui/pageflow/PageFlowController.internalExecute(PageFlowController.java:306)

      at org/apache/beehive/netui/pageflow/FlowController.execute(FlowController.java:336)

      ^--
      "[STUCK] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'" id=43 idx=0xb0 tid=3556 prio=1 alive, in native, daemon

      at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)
               
                      at jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:46)[optimized]

      at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)

      at java/net/SocketInputStream.read(SocketInputStream.java:129)

      at java/io/BufferedInputStream.fill(BufferedInputStream.java:218)[inlined]

      at java/io/BufferedInputStream.read(BufferedInputStream.java:237)[optimized]

      ^-- Holding lock: java/io/BufferedInputStream@0x000000003725B838[biased lock]

      at weblogic/net/http/MessageHeader.isHTTP(MessageHeader.java:220)

      at weblogic/net/http/MessageHeader.parseHeader(MessageHeader.java:143)[optimized]

      at weblogic/net/http/HttpClient.parseHTTP(HttpClient.java:454)[optimized]




      --


      "[STUCK] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'" id=1075 idx=0x370 tid=5100 prio=1 alive, in native, blocked, daemon

      -- Blocked trying to get lock: portlets/OtherInvestmentSummary/OtherInvestmentSummaryController@0x000000001B87D3E0[thin lock]

      at jrockit/vm/Threads.sleep(I)V(Native Method)

      at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)[optimized]

      at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)[optimized]

      at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)[optimized]

      at jrockit/vm/Locks.monitorEnter(Locks.java:2456)[optimized]

      at org/apache/beehive/netui/pageflow/FlowController.execute(FlowController.java:320)

      at org/apache/beehive/netui/pageflow/internal/FlowControllerAction.execute(FlowControllerAction.java:52)

      at org/apache/struts/action/RequestProcessor.processActionPerform(RequestProcessor.java:484)

      at org/apache/beehive/netui/pageflow/PageFlowRequestProcessor.access$201(PageFlowRequestProcessor.java:97)

      at org/apache/beehive/netui/pageflow/PageFlowRequestProcessor$ActionRunner.execute(PageFlowRequestProcessor.java:2044)

      at org/apache/beehive/netui/pageflow/interceptor/action/internal/ActionInterceptors$WrapActionInterceptorChain.continueChain(ActionInterceptors.java:64)

      at org/apache/beehive/netui/pageflow/interceptor/action/ActionInterceptor.wrapAction(ActionInterceptor.java:184)

      at org/apache/beehive/netui/pageflow/interceptor/action/internal/ActionInterceptors$WrapActionInterceptorChain.invoke(ActionInterceptors.java:50)

      at org/apache/beehive/netui/pageflow/interceptor/action/internal/ActionInterceptors$WrapActionInterceptorChain.continueChain(ActionInterceptors.java:58)

      at org/apache/beehive/netui/pageflow/interceptor/action/internal/ActionInterceptors.wrapAction(ActionInterceptors.java:87)[inlined]

      at org/apache/beehive/netui/pageflow/PageFlowRequestProcessor.processActionPerform(PageFlowRequestProcessor.java:2116)[optimized]

      at org/apache/struts/action/RequestProcessor.process(RequestProcessor.java:274)[optimized]

      at org/apache/beehive/netui/pageflow/PageFlowRequestProcessor.processInternal(PageFlowRequestProcessor.java:556)[optimized]

      at org/apache/beehive/netui/pageflow/PageFlowRequestProcessor.process(PageFlowRequestProcessor.java:853)[optimized]

      at org/apache/beehive/netui/pageflow/AutoRegisterActionServlet.process(AutoRegisterActionServlet.java:631)[optimized]

      at org/apache/beehive/netui/pageflow/PageFlowActionServlet.process(PageFlowActionServlet.java:158)

      at org/apache/struts/action/ActionServlet.doGet(ActionServlet.java:507)

      at org/apache/beehive/netui/pageflow/PageFlowUtils.strutsLookup(PageFlowUtils.java:1199)

      at org/apache/beehive/netui/pageflow/PageFlowUtils.strutsLookup(PageFlowUtils.java:1129)

      at com/bea/portlet/adapter/scopedcontent/framework/internal/PageFlowUtilsBeehiveDelegate.strutsLookupInternal(PageFlowUtilsBeehiveDelegate.java:43)

      at com/bea/portlet/adapter/scopedcontent/framework/PageFlowUtils.strutsLookup(PageFlowUtils.java:108)

      at com/bea/portlet/adapter/scopedcontent/ScopedContentCommonSupport.executeAction(ScopedContentCommonSupport.java:707)

      at com/bea/portlet/adapter/scopedcontent/ScopedContentCommonSupport.processActionInternal(ScopedContentCommonSupport.java:141)

      at com/bea/portlet/adapter/scopedcontent/PageFlowStubImpl.processAction(PageFlowStubImpl.java:108)

      at com/bea/portlet/adapter/NetuiActionHandler.raiseScopedAction(NetuiActionHandler.java:111)

      at com/bea/netuix/servlets/controls/content/NetuiContent.raiseScopedAction(NetuiContent.java:181)

      at com/bea/netuix/servlets/controls/content/NetuiContent.raiseScopedAction(NetuiContent.java:167)

      at com/bea/netuix/servlets/controls/content/NetuiContent.handlePostbackData(NetuiContent.java:225)

      at com/bea/netuix/nf/ControlLifecycle$2.visit(ControlLifecycle.java:180)[optimized]

      at com/bea/netuix/nf/ControlTreeWalker.walkRecursive(ControlTreeWalker.java:324)[optimized]

      at com/bea/netuix/nf/ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)[optimized]

      at com/bea/netuix/nf/ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)[optimized]

      at com/bea/netuix/nf/ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)[optimized]

      at com/bea/netuix/nf/ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)[optimized]

      at com/bea/netuix/nf/ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)[optimized]

      at com/bea/netuix/nf/ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)[optimized]

      at com/bea/netuix/nf/ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)[optimized]

      at com/bea/netuix/nf/ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)[optimized]

      at com/bea/netuix/nf/ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)[optimized]

      at com/bea/netuix/nf/ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)[optimized]

      at com/bea/netuix/nf/ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)[optimized]

      at com/bea/netuix/nf/ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)[optimized]

      at com/bea/netuix/nf/ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)[optimized]

      at com/bea/netuix/nf/ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)[inlined]

      at com/bea/netuix/nf/ControlTreeWalker.walk(ControlTreeWalker.java:130)[optimized]

      at com/bea/netuix/nf/Lifecycle.processLifecycles(Lifecycle.java:399)[optimized]

      at com/bea/netuix/nf/Lifecycle.processLifecycles(Lifecycle.java:361)

      at com/bea/netuix/nf/Lifecycle.processLifecycles(Lifecycle.java:352)

      at com/bea/netuix/nf/Lifecycle.runInbound(Lifecycle.java:184)

      at com/bea/netuix/nf/Lifecycle.run(Lifecycle.java:159)

      at com/bea/netuix/servlets/manager/UIServlet.runLifecycle(UIServlet.java:465)

      at com/bea/netuix/servlets/manager/UIServlet.processControlTree(UIServlet.java:373)

      at com/bea/netuix/servlets/manager/PortalServlet.service(PortalServlet.java:981)

      at javax/servlet/http/HttpServlet.service(HttpServlet.java:820)[optimized]

      at weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)[optimized]

      at weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)[inlined]

      at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:292)[optimized]

      at weblogic/servlet/internal/TailFilter.doFilter(TailFilter.java:26)[optimized]

      at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]

      at com/bea/content/manager/servlets/ContentServletFilter.doFilter(ContentServletFilter.java:178)[optimized]

      at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]

      at com/bea/p13n/servlets/PortalServletFilter.doFilter(PortalServletFilter.java:336)[optimized]

      at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]

      at com/bea/portal/tools/servlet/http/HttpContextFilter.doFilter(HttpContextFilter.java:60)[optimized]

      at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[inlined]

      at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3592)[optimized]

      at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)[optimized]

      at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:121)[inlined]

      at weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServletContext.java:2202)[inlined]

      at weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext.java:2108)[optimized]

      at weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:1432)[optimized]

      at weblogic/work/ExecuteThread.execute(ExecuteThread.java:201)[inlined]

      at weblogic/work/ExecuteThread.run(ExecuteThread.java:173)[optimized]

      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

      -- end of trace

      "[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" id=1227 idx=0x388 tid=4524 prio=1 alive, in native, blocked, daemon

      -- Blocked trying to get lock: portlets/OtherInvestmentSummary/OtherInvestmentSummaryController@0x000000001B87D3E0[thin lock]



      "Thread-3035" id=3223 idx=0x400 tid=6028 prio=5 alive, in native, daemon

      at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)
               
                      at jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:46)[optimized]

      at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)

      at java/net/SocketInputStream.read(SocketInputStream.java:129)

      at java/io/BufferedInputStream.fill(BufferedInputStream.java:218)

      at java/io/BufferedInputStream.read1(BufferedInputStream.java:258)

      at java/io/BufferedInputStream.read(BufferedInputStream.java:317)

      ^-- Holding lock: java/io/BufferedInputStream@0x000000003FA7C368[biased lock]

      at com/sun/jndi/ldap/Connection.run(Connection.java:805)[optimized]

      at java/lang/Thread.run(Thread.java:619)[optimized]

      at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

      -- end of trace

      Blocked lock chains

      ===================

      Chain 2:

      "[STUCK] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'" id=1075 idx=0x370 tid=5100 waiting for portlets/OtherInvestmentSummary/OtherInvestmentSummaryController@0x000000001B87D3E0 held by:

      "[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" id=42 idx=0xac tid=776 in chain 1

      Chain 3:

      "[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" id=1227 idx=0x388 tid=4524 waiting for portlets/OtherInvestmentSummary/OtherInvestmentSummaryController@0x000000001B87D3E0 held by:

      "[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" id=42 idx=0xac tid=776 in chain 1

      Open lock chains

      ================

      Chain 1:

      "[STUCK] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" id=18 idx=0x60 tid=3292 waiting for portlets/OtherInvestmentSummary/OtherInvestmentSummaryController@0x000000001B87D3E0 held by:

      "[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" id=42 idx=0xac tid=776 (active)

      ===== END OF THREAD DUMP ===============
        • 1. Re: Stuck Thread
          René van Wijk
          Holding lock: java/io/BufferedInputStream@0x000000001B6B2020[biased lock]

          Looks like there is contention for the file system when HTTP connections are made.

          How are the file descriptors of your operating system configured?

          Information concerning file descriptor configuration can be found here - http://middlewaremagic.com/weblogic/?p=7737
          especially the section 'Maximum number of open file descriptors'
          • 2. Re: Stuck Thread
            the_assface
            We are on a Windows platform, do you have any feedback on how those recommendations are checked or altered?

            Thanks
            • 3. Re: Stuck Thread
              René van Wijk
              From the documentation Professional Oracle WebLogic Server book:

              "Most of the TCP/IP parameter settings are located in the registry under the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services folder.
              To change the maximum length of the TCP listen queue, whose default value is 15, you create a DWORD entry called ListenBackLog in the
              Inetinfo\Parameters subfolder in your Windows registry. Windows sets the default for the time wait interval to four minutes; the TcpTimedWaitDelay
              parameter in the Tcpip\Parameters subfolder allows you to change its value."

              A paper that discusses Windows Server 2008 performance tuning is located here - http://msdn.microsoft.com/en-us/windows/hardware/gg463394.aspx
              • 4. Re: Stuck Thread
                the_assface
                Thanks Rene...although I have some stuck thread issues occasionally this is the first one I have seen with this particular error. So I am going to treat it as a one off for now and if it pops up again then I will review these suggestions. Unfortunately I have not seen this error in my 2 test environments (they aren't hit nearly as hard as my prod environments which is normal I am sure).

                My complaint on the WL stuck threads is that they never seem to clear the console unless you restart the managed server. I don't know if I should assume that whatever caused them is still an issue or if WL just does a poor job on updating the cleanup of the error. I have had no user complaints though....


                Thanks for the assistance and direction!