This discussion is archived
4 Replies Latest reply: Jan 10, 2012 1:24 PM by the_assface RSS

Stuck Thread

the_assface Explorer
Currently Being Moderated
Getting some occasional stuck threads on my portal servers. Apparently not affecting application as it seems to be up and running. However, the only way to clear the warnings in the console is via an app server restart. Not ideal since this is production. Any clues on from this thread dump on what the issue may be? Or even better, what a potential fix may be?

Thanks

(I had to cut out some of the dump due to 30,000 character limit)


===== FULL THREAD DUMP ===============



"[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" id=42 idx=0xac tid=776 prio=1 alive, in native, daemon

at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)
         
                at jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:46)[optimized]

at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)

at java/net/SocketInputStream.read(SocketInputStream.java:129)

at java/io/BufferedInputStream.fill(BufferedInputStream.java:218)[inlined]

at java/io/BufferedInputStream.read(BufferedInputStream.java:237)[optimized]

^-- Holding lock: java/io/BufferedInputStream@0x000000001B6B2020[biased lock]

at weblogic/net/http/MessageHeader.isHTTP(MessageHeader.java:220)

at weblogic/net/http/MessageHeader.parseHeader(MessageHeader.java:143)[optimized]

at weblogic/net/http/HttpClient.parseHTTP(HttpClient.java:454)[optimized]

^-- Holding lock: weblogic/net/http/HttpClient@0x000000001B6B20A0[biased lock]

at weblogic/net/http/HttpURLConnection.getInputStream(HttpURLConnection.java:364)[optimized]

^-- Holding lock: weblogic/net/http/SOAPHttpURLConnection@0x000000001B6B21F8[biased lock]

at weblogic/net/http/SOAPHttpURLConnection.getInputStream(SOAPHttpURLConnection.java:36)

at weblogic/net/http/HttpURLConnection.getResponseCode(HttpURLConnection.java:952)

at com/fc/logic/product/Navs.getNavs(Navs.java:224)

at com/fc/logic/product/Navs.getNavs(Navs.java:138)

at com/fc/logic/investments/FundRetriever.retrieveStock(FundRetriever.java:269)

at com/fc/logic/investments/FundRetriever.getStockQuote(FundRetriever.java:118)

at com/fc/logic/investments/NavsFundMgr.getStockQuote(NavsFundMgr.java:39)

at portlets/OtherInvestmentSummary/OtherInvestmentSummaryController.getFundValue(OtherInvestmentSummaryController.java:1509)

at portlets/OtherInvestmentSummary/OtherInvestmentSummaryController.loadOtherInvestmentSummaries(OtherInvestmentSummaryController.java:180)

at portlets/OtherInvestmentSummary/OtherInvestmentSummaryController.begin(OtherInvestmentSummaryController.java:429)

at portlets/OtherInvestmentSummary/OtherInvestmentSummaryController.AfterAdd(OtherInvestmentSummaryController.java:1191)

at sun/reflect/GeneratedMethodAccessor865.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
         
                at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]

at java/lang/reflect/Method.invoke(Method.java:597)[optimized]

at org/apache/beehive/netui/pageflow/FlowController.invokeActionMethod(FlowController.java:870)

at org/apache/beehive/netui/pageflow/FlowController.getActionMethodForward(FlowController.java:809)

at org/apache/beehive/netui/pageflow/FlowController.internalExecute(FlowController.java:478)

at org/apache/beehive/netui/pageflow/PageFlowController.internalExecute(PageFlowController.java:306)

at org/apache/beehive/netui/pageflow/FlowController.execute(FlowController.java:336)

^--
"[STUCK] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'" id=43 idx=0xb0 tid=3556 prio=1 alive, in native, daemon

at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)
         
                at jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:46)[optimized]

at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)

at java/net/SocketInputStream.read(SocketInputStream.java:129)

at java/io/BufferedInputStream.fill(BufferedInputStream.java:218)[inlined]

at java/io/BufferedInputStream.read(BufferedInputStream.java:237)[optimized]

^-- Holding lock: java/io/BufferedInputStream@0x000000003725B838[biased lock]

at weblogic/net/http/MessageHeader.isHTTP(MessageHeader.java:220)

at weblogic/net/http/MessageHeader.parseHeader(MessageHeader.java:143)[optimized]

at weblogic/net/http/HttpClient.parseHTTP(HttpClient.java:454)[optimized]




--


"[STUCK] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'" id=1075 idx=0x370 tid=5100 prio=1 alive, in native, blocked, daemon

-- Blocked trying to get lock: portlets/OtherInvestmentSummary/OtherInvestmentSummaryController@0x000000001B87D3E0[thin lock]

at jrockit/vm/Threads.sleep(I)V(Native Method)

at jrockit/vm/Locks.waitForThinRelease(Locks.java:1209)[optimized]

at jrockit/vm/Locks.monitorEnterSecondStageHard(Locks.java:1342)[optimized]

at jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1259)[optimized]

at jrockit/vm/Locks.monitorEnter(Locks.java:2456)[optimized]

at org/apache/beehive/netui/pageflow/FlowController.execute(FlowController.java:320)

at org/apache/beehive/netui/pageflow/internal/FlowControllerAction.execute(FlowControllerAction.java:52)

at org/apache/struts/action/RequestProcessor.processActionPerform(RequestProcessor.java:484)

at org/apache/beehive/netui/pageflow/PageFlowRequestProcessor.access$201(PageFlowRequestProcessor.java:97)

at org/apache/beehive/netui/pageflow/PageFlowRequestProcessor$ActionRunner.execute(PageFlowRequestProcessor.java:2044)

at org/apache/beehive/netui/pageflow/interceptor/action/internal/ActionInterceptors$WrapActionInterceptorChain.continueChain(ActionInterceptors.java:64)

at org/apache/beehive/netui/pageflow/interceptor/action/ActionInterceptor.wrapAction(ActionInterceptor.java:184)

at org/apache/beehive/netui/pageflow/interceptor/action/internal/ActionInterceptors$WrapActionInterceptorChain.invoke(ActionInterceptors.java:50)

at org/apache/beehive/netui/pageflow/interceptor/action/internal/ActionInterceptors$WrapActionInterceptorChain.continueChain(ActionInterceptors.java:58)

at org/apache/beehive/netui/pageflow/interceptor/action/internal/ActionInterceptors.wrapAction(ActionInterceptors.java:87)[inlined]

at org/apache/beehive/netui/pageflow/PageFlowRequestProcessor.processActionPerform(PageFlowRequestProcessor.java:2116)[optimized]

at org/apache/struts/action/RequestProcessor.process(RequestProcessor.java:274)[optimized]

at org/apache/beehive/netui/pageflow/PageFlowRequestProcessor.processInternal(PageFlowRequestProcessor.java:556)[optimized]

at org/apache/beehive/netui/pageflow/PageFlowRequestProcessor.process(PageFlowRequestProcessor.java:853)[optimized]

at org/apache/beehive/netui/pageflow/AutoRegisterActionServlet.process(AutoRegisterActionServlet.java:631)[optimized]

at org/apache/beehive/netui/pageflow/PageFlowActionServlet.process(PageFlowActionServlet.java:158)

at org/apache/struts/action/ActionServlet.doGet(ActionServlet.java:507)

at org/apache/beehive/netui/pageflow/PageFlowUtils.strutsLookup(PageFlowUtils.java:1199)

at org/apache/beehive/netui/pageflow/PageFlowUtils.strutsLookup(PageFlowUtils.java:1129)

at com/bea/portlet/adapter/scopedcontent/framework/internal/PageFlowUtilsBeehiveDelegate.strutsLookupInternal(PageFlowUtilsBeehiveDelegate.java:43)

at com/bea/portlet/adapter/scopedcontent/framework/PageFlowUtils.strutsLookup(PageFlowUtils.java:108)

at com/bea/portlet/adapter/scopedcontent/ScopedContentCommonSupport.executeAction(ScopedContentCommonSupport.java:707)

at com/bea/portlet/adapter/scopedcontent/ScopedContentCommonSupport.processActionInternal(ScopedContentCommonSupport.java:141)

at com/bea/portlet/adapter/scopedcontent/PageFlowStubImpl.processAction(PageFlowStubImpl.java:108)

at com/bea/portlet/adapter/NetuiActionHandler.raiseScopedAction(NetuiActionHandler.java:111)

at com/bea/netuix/servlets/controls/content/NetuiContent.raiseScopedAction(NetuiContent.java:181)

at com/bea/netuix/servlets/controls/content/NetuiContent.raiseScopedAction(NetuiContent.java:167)

at com/bea/netuix/servlets/controls/content/NetuiContent.handlePostbackData(NetuiContent.java:225)

at com/bea/netuix/nf/ControlLifecycle$2.visit(ControlLifecycle.java:180)[optimized]

at com/bea/netuix/nf/ControlTreeWalker.walkRecursive(ControlTreeWalker.java:324)[optimized]

at com/bea/netuix/nf/ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)[optimized]

at com/bea/netuix/nf/ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)[optimized]

at com/bea/netuix/nf/ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)[optimized]

at com/bea/netuix/nf/ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)[optimized]

at com/bea/netuix/nf/ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)[optimized]

at com/bea/netuix/nf/ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)[optimized]

at com/bea/netuix/nf/ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)[optimized]

at com/bea/netuix/nf/ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)[optimized]

at com/bea/netuix/nf/ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)[optimized]

at com/bea/netuix/nf/ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)[optimized]

at com/bea/netuix/nf/ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)[optimized]

at com/bea/netuix/nf/ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)[optimized]

at com/bea/netuix/nf/ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)[optimized]

at com/bea/netuix/nf/ControlTreeWalker.walkRecursive(ControlTreeWalker.java:334)[inlined]

at com/bea/netuix/nf/ControlTreeWalker.walk(ControlTreeWalker.java:130)[optimized]

at com/bea/netuix/nf/Lifecycle.processLifecycles(Lifecycle.java:399)[optimized]

at com/bea/netuix/nf/Lifecycle.processLifecycles(Lifecycle.java:361)

at com/bea/netuix/nf/Lifecycle.processLifecycles(Lifecycle.java:352)

at com/bea/netuix/nf/Lifecycle.runInbound(Lifecycle.java:184)

at com/bea/netuix/nf/Lifecycle.run(Lifecycle.java:159)

at com/bea/netuix/servlets/manager/UIServlet.runLifecycle(UIServlet.java:465)

at com/bea/netuix/servlets/manager/UIServlet.processControlTree(UIServlet.java:373)

at com/bea/netuix/servlets/manager/PortalServlet.service(PortalServlet.java:981)

at javax/servlet/http/HttpServlet.service(HttpServlet.java:820)[optimized]

at weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)[optimized]

at weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)[inlined]

at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:292)[optimized]

at weblogic/servlet/internal/TailFilter.doFilter(TailFilter.java:26)[optimized]

at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]

at com/bea/content/manager/servlets/ContentServletFilter.doFilter(ContentServletFilter.java:178)[optimized]

at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]

at com/bea/p13n/servlets/PortalServletFilter.doFilter(PortalServletFilter.java:336)[optimized]

at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]

at com/bea/portal/tools/servlet/http/HttpContextFilter.doFilter(HttpContextFilter.java:60)[optimized]

at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[inlined]

at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3592)[optimized]

at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)[optimized]

at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:121)[inlined]

at weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServletContext.java:2202)[inlined]

at weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext.java:2108)[optimized]

at weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:1432)[optimized]

at weblogic/work/ExecuteThread.execute(ExecuteThread.java:201)[inlined]

at weblogic/work/ExecuteThread.run(ExecuteThread.java:173)[optimized]

at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

-- end of trace

"[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" id=1227 idx=0x388 tid=4524 prio=1 alive, in native, blocked, daemon

-- Blocked trying to get lock: portlets/OtherInvestmentSummary/OtherInvestmentSummaryController@0x000000001B87D3E0[thin lock]



"Thread-3035" id=3223 idx=0x400 tid=6028 prio=5 alive, in native, daemon

at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)
         
                at jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:46)[optimized]

at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)

at java/net/SocketInputStream.read(SocketInputStream.java:129)

at java/io/BufferedInputStream.fill(BufferedInputStream.java:218)

at java/io/BufferedInputStream.read1(BufferedInputStream.java:258)

at java/io/BufferedInputStream.read(BufferedInputStream.java:317)

^-- Holding lock: java/io/BufferedInputStream@0x000000003FA7C368[biased lock]

at com/sun/jndi/ldap/Connection.run(Connection.java:805)[optimized]

at java/lang/Thread.run(Thread.java:619)[optimized]

at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)

-- end of trace

Blocked lock chains

===================

Chain 2:

"[STUCK] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'" id=1075 idx=0x370 tid=5100 waiting for portlets/OtherInvestmentSummary/OtherInvestmentSummaryController@0x000000001B87D3E0 held by:

"[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" id=42 idx=0xac tid=776 in chain 1

Chain 3:

"[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" id=1227 idx=0x388 tid=4524 waiting for portlets/OtherInvestmentSummary/OtherInvestmentSummaryController@0x000000001B87D3E0 held by:

"[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" id=42 idx=0xac tid=776 in chain 1

Open lock chains

================

Chain 1:

"[STUCK] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" id=18 idx=0x60 tid=3292 waiting for portlets/OtherInvestmentSummary/OtherInvestmentSummaryController@0x000000001B87D3E0 held by:

"[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" id=42 idx=0xac tid=776 (active)

===== END OF THREAD DUMP ===============
  • 1. Re: Stuck Thread
    RenévanWijk Oracle ACE
    Currently Being Moderated
    Holding lock: java/io/BufferedInputStream@0x000000001B6B2020[biased lock]

    Looks like there is contention for the file system when HTTP connections are made.

    How are the file descriptors of your operating system configured?

    Information concerning file descriptor configuration can be found here - http://middlewaremagic.com/weblogic/?p=7737
    especially the section 'Maximum number of open file descriptors'
  • 2. Re: Stuck Thread
    the_assface Explorer
    Currently Being Moderated
    We are on a Windows platform, do you have any feedback on how those recommendations are checked or altered?

    Thanks
  • 3. Re: Stuck Thread
    RenévanWijk Oracle ACE
    Currently Being Moderated
    From the documentation Professional Oracle WebLogic Server book:

    "Most of the TCP/IP parameter settings are located in the registry under the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services folder.
    To change the maximum length of the TCP listen queue, whose default value is 15, you create a DWORD entry called ListenBackLog in the
    Inetinfo\Parameters subfolder in your Windows registry. Windows sets the default for the time wait interval to four minutes; the TcpTimedWaitDelay
    parameter in the Tcpip\Parameters subfolder allows you to change its value."

    A paper that discusses Windows Server 2008 performance tuning is located here - http://msdn.microsoft.com/en-us/windows/hardware/gg463394.aspx
  • 4. Re: Stuck Thread
    the_assface Explorer
    Currently Being Moderated
    Thanks Rene...although I have some stuck thread issues occasionally this is the first one I have seen with this particular error. So I am going to treat it as a one off for now and if it pops up again then I will review these suggestions. Unfortunately I have not seen this error in my 2 test environments (they aren't hit nearly as hard as my prod environments which is normal I am sure).

    My complaint on the WL stuck threads is that they never seem to clear the console unless you restart the managed server. I don't know if I should assume that whatever caused them is still an issue or if WL just does a poor job on updating the cleanup of the error. I have had no user complaints though....


    Thanks for the assistance and direction!

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points