3 Replies Latest reply: Mar 1, 2011 7:57 AM by PHCharbonneau RSS

    Help on stuck thread

    user9503787
      Hi, I am seeing stuck thread in log file:
      ####<Feb 21, 2011 10:47:42 AM PST> <Error> <WebLogicServer> <prod.cd.com> <server2> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <ExecuteThread: '22' for queue: 'weblogic.kernel.Default' has been busy for "1,553" seconds working on the request "Http Request: /appmanager/NewPortal/CustDesktop", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>
      ####<Feb 21, 2011 10:47:42 AM PST> <Error> <WebLogicServer> <prod.cd.com> <server2> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <ExecuteThread: '24' for queue: 'weblogic.kernel.Default' has been busy for "1,417" seconds working on the request "Http Request: /appmanager/NewPortal/CustDesktop", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>
      ####<Feb 21, 2011 10:57:42 AM PST> <Error> <WebLogicServer> <prod.cd.comt> <server2> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <ExecuteThread: '23' for queue: 'weblogic.kernel.Default' has been busy for "790" seconds working on the request "Http Request: /appmanager/NewPortal/CustDesktop", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>
      ####<Feb 21, 2011 11:07:42 AM PST> <Error> <WebLogicServer> <prod.cd.com> <server2> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <ExecuteThread: '23' for queue: 'weblogic.kernel.Default' has been busy for "1,390" seconds working on the request "Http Request: /appmanager/NewPortal/CustDesktop", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>
      Full thread dump Java HotSpot(TM) Server VM (1.4.2_11-b06 mixed mode):
      ...
      "ExecuteThread: '24' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x0090e6b0 nid=0x3b waiting for monitor entry [0x600fd000..0x600ffc28]
      at com.bea.wlw.netui.script.el.util.ParseUtils.parse(ParseUtils.java:100)
      - waiting to lock <0x8d3a3f48> (a com.bea.wlw.netui.script.el.util.ParseUtils$Cache)
      at com.bea.wlw.netui.script.el.ExpressionEvaluatorImpl.containsExpression(ExpressionEvaluatorImpl.java:239)
      at com.bea.wlw.netui.pageflow.ProcessPopulate.populate(ProcessPopulate.java:295)
      at com.bea.wlw.netui.pageflow.PageFlowRequestProcessor.processPopulate(PageFlowRequestProcessor.java:402)
      at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:254)
      at com.bea.wlw.netui.pageflow.PageFlowRequestProcessor.process(PageFlowRequestProcessor.java:691)
      at com.bea.wlw.netui.pageflow.AutoRegisterActionServlet.process(AutoRegisterActionServlet.java:527)
      at com.bea.wlw.netui.pageflow.PageFlowActionServlet.process(PageFlowActionServlet.java:152)
      at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:507)
      at com.bea.wlw.netui.pageflow.PageFlowUtils.strutsLookup(PageFlowUtils.java:1786)
      at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.executeAction(ScopedContentCommonSupport.java:561)
      at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.processActionInternal(ScopedContentCommonSupport.java:121)
      at com.bea.portlet.adapter.scopedcontent.PageFlowStubImpl.processAction(PageFlowStubImpl.java:98)
      at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:154)
      at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:113)
      at com.bea.netuix.servlets.controls.content.NetuiContent.handlePostbackData(NetuiContent.java:228)
      at com.bea.netuix.nf.ControlLifecycle$3.visit(ControlLifecycle.java:171)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:356)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:126)
      at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:105)
      at com.bea.netuix.nf.Lifecycle.runInbound(Lifecycle.java:173)
      at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:137)
      at com.bea.netuix.servlets.manager.UIServlet.runLifecycle(UIServlet.java:324)
      at com.bea.netuix.servlets.manager.UIServlet.processControlTree(UIServlet.java:220)
      at com.bea.netuix.servlets.manager.PortalServlet.doPost(PortalServlet.java:820)
      at com.bea.netuix.servlets.manager.UIServlet.service(UIServlet.java:150)
      at com.cd.servlets.portal.MainPortalServlet.service(MainPortalServlet.java:96)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
      at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
      at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
      at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
      at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
      at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:293)
      at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
      at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7053)
      at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
      at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
      at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3902)
      at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2773)
      at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
      at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)

      "ExecuteThread: '23' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x0090dab8 nid=0x3a runnable [0x601fb000..0x601ffc28]
      at java.util.HashMap.getEntry(HashMap.java:361)
      at java.util.LinkedHashMap.get(LinkedHashMap.java:266)
      at com.bea.wlw.netui.script.el.util.ParseUtils.parse(ParseUtils.java:84)
      at com.bea.wlw.netui.script.el.util.ParseUtils.evaluate(ParseUtils.java:123)
      at com.bea.wlw.netui.script.el.ExpressionEvaluatorImpl.evaluateStrict(ExpressionEvaluatorImpl.java:87)
      at com.bea.wlw.netui.tags.AbstractBaseTag.evaluateExpressionInternal(AbstractBaseTag.java:573)
      at com.bea.wlw.netui.tags.AbstractBaseTag.evaluateExpression(AbstractBaseTag.java:362)
      at com.bea.wlw.netui.tags.databinding.script.GetData.doEndTag(GetData.java:249)
      at jsp_servlet._portlets.___portlet1._jspService(_portlet1.jsp:27)
      at weblogic.servlet.jsp.JspBase.service(JspBase.java:33)
      at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
      at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
      at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
      at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
      at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:293)
      at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
      at com.bea.wlw.netui.pageflow.PageFlowJspFilter.doFilter(PageFlowJspFilter.java:250)
      - locked <0x78ccf6c8> (a portlets.Portlet1Controller)
      at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
      at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:652)
      at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:431)
      at com.bea.wlw.netui.pageflow.scoping.internal.ScopedRequestDispatcher.include(ScopedRequestDispatcher.java:120)
      at com.bea.netuix.servlets.controls.content.JspContent.beginRender(JspContent.java:534)
      at com.bea.netuix.servlets.controls.content.NetuiContent.beginRender(NetuiContent.java:425)
      at com.bea.netuix.nf.ControlLifecycle$1.visit(ControlLifecycle.java:495)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:543)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
      at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:247)
      at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:105)
      at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:356)
      at com.bea.netuix.nf.UIControl.render(UIControl.java:536)
      at com.bea.netuix.servlets.controls.PresentationContext.render(PresentationContext.java:405)
      at com.bea.netuix.servlets.util.RenderToolkit.renderChild(RenderToolkit.java:123)
      at com.bea.netuix.servlets.jsp.taglib.RenderChild.doStartTag(RenderChild.java:58)
      at jsp_servlet._framework._skeletons._cd.__flowlayout._jspService(flowlayout.jsp:30)
      at weblogic.servlet.jsp.JspBase.service(JspBase.java:33)
      at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
      at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
      at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:348)
      at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:646)
      at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:431)
      at com.bea.netuix.servlets.controls.JspRenderer.renderAlt(JspRenderer.java:194)
      at com.bea.netuix.servlets.controls.JspRenderer.beginRender(JspRenderer.java:96)
      at com.bea.netuix.nf.ControlLifecycle$1.visit(ControlLifecycle.java:491)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:543)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
      at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:247)
      at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:105)
      at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:356)
      at com.bea.netuix.nf.UIControl.render(UIControl.java:536)
      at com.bea.netuix.servlets.controls.PresentationContext.render(PresentationContext.java:405)
      at com.bea.netuix.servlets.util.RenderToolkit.renderChild(RenderToolkit.java:123)
      at com.bea.netuix.servlets.jsp.taglib.RenderChild.doStartTag(RenderChild.java:58)
      at jsp_servlet._framework._skeletons._cd.__flowlayout._jspService(flowlayout.jsp:30)
      at weblogic.servlet.jsp.JspBase.service(JspBase.java:33)
      at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
      at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
      at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:348)
      at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:646)
      at weblogic.servlet.internal.RequestDispatcherImpl.include(RequestDispatcherImpl.java:431)
      at com.bea.netuix.servlets.controls.JspRenderer.renderAlt(JspRenderer.java:194)
      at com.bea.netuix.servlets.controls.JspRenderer.beginRender(JspRenderer.java:96)
      at com.bea.netuix.nf.ControlLifecycle$1.visit(ControlLifecycle.java:491)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:543)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursiveRender(ControlTreeWalker.java:554)
      at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:247)
      at com.bea.netuix.nf.Lifecycle.runOutbound(Lifecycle.java:204)
      at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:153)
      at com.bea.netuix.servlets.manager.UIServlet.runLifecycle(UIServlet.java:324)
      at com.bea.netuix.servlets.manager.UIServlet.processControlTree(UIServlet.java:220)
      at com.bea.netuix.servlets.manager.PortalServlet.doPost(PortalServlet.java:820)
      at com.bea.netuix.servlets.manager.PortalServlet.doGet(PortalServlet.java:671)
      at com.bea.netuix.servlets.manager.UIServlet.service(UIServlet.java:147)
      at com.cd.servlets.portal.MainPortalServlet.service(MainPortalServlet.java:96)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
      at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
      at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
      at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
      at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
      at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:293)
      at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
      at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7053)
      at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
      at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
      at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3902)
      at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2773)
      at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
      at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)

      "ExecuteThread: '22' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x0090cec0 nid=0x39 runnable [0x602fd000..0x602ffc28]
      at java.util.LinkedHashMap.transfer(LinkedHashMap.java:224)
      at java.util.HashMap.resize(HashMap.java:452)
      at java.util.LinkedHashMap.addEntry(LinkedHashMap.java:399)
      at java.util.HashMap.put(HashMap.java:392)
      at com.bea.wlw.netui.script.el.util.ParseUtils.parse(ParseUtils.java:100)
      - locked <0x8d3a3f48> (a com.bea.wlw.netui.script.el.util.ParseUtils$Cache)
      at com.bea.wlw.netui.script.el.ExpressionEvaluatorImpl.containsExpression(ExpressionEvaluatorImpl.java:239)
      at com.bea.wlw.netui.pageflow.ProcessPopulate.populate(ProcessPopulate.java:295)
      at com.bea.wlw.netui.pageflow.PageFlowRequestProcessor.processPopulate(PageFlowRequestProcessor.java:402)
      at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:254)
      at com.bea.wlw.netui.pageflow.PageFlowRequestProcessor.process(PageFlowRequestProcessor.java:691)
      at com.bea.wlw.netui.pageflow.AutoRegisterActionServlet.process(AutoRegisterActionServlet.java:527)
      at com.bea.wlw.netui.pageflow.PageFlowActionServlet.process(PageFlowActionServlet.java:152)
      at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:507)
      at com.bea.wlw.netui.pageflow.PageFlowUtils.strutsLookup(PageFlowUtils.java:1786)
      at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.executeAction(ScopedContentCommonSupport.java:561)
      at com.bea.portlet.adapter.scopedcontent.ScopedContentCommonSupport.processActionInternal(ScopedContentCommonSupport.java:121)
      at com.bea.portlet.adapter.scopedcontent.PageFlowStubImpl.processAction(PageFlowStubImpl.java:98)
      at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:154)
      at com.bea.netuix.servlets.controls.content.NetuiContent.raiseScopedAction(NetuiContent.java:113)
      at com.bea.netuix.servlets.controls.content.NetuiContent.handlePostbackData(NetuiContent.java:228)
      at com.bea.netuix.nf.ControlLifecycle$3.visit(ControlLifecycle.java:171)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:356)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walkRecursive(ControlTreeWalker.java:366)
      at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:126)
      at com.bea.netuix.nf.ControlTreeWalker.walk(ControlTreeWalker.java:105)
      at com.bea.netuix.nf.Lifecycle.runInbound(Lifecycle.java:173)
      at com.bea.netuix.nf.Lifecycle.run(Lifecycle.java:137)
      at com.bea.netuix.servlets.manager.UIServlet.runLifecycle(UIServlet.java:324)
      at com.bea.netuix.servlets.manager.UIServlet.processControlTree(UIServlet.java:220)
      at com.bea.netuix.servlets.manager.PortalServlet.doPost(PortalServlet.java:820)
      at com.bea.netuix.servlets.manager.UIServlet.service(UIServlet.java:150)
      at com.cd.servlets.portal.MainPortalServlet.service(MainPortalServlet.java:96)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
      at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)
      at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
      at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
      at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
      at com.bea.p13n.servlets.PortalServletFilter.doFilter(PortalServletFilter.java:293)
      at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
      at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7053)
      at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
      at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
      at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3902)
      at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2773)
      at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
      at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)

      ...
      "VM Thread" prio=5 tid=0x001b5518 nid=0x19 runnable

      "VM Periodic Task Thread" prio=10 tid=0x001c0ba0 nid=0x21 runnable
      "Suspend Checker Thread" prio=10 tid=0x001bbb30 nid=0x1c runnable
        • 1. Re: Help on stuck thread
          PHCharbonneau
          Hi user9503787,

          Can you please confirm if the Weblogic instance / java process you took the Thread Dump from was showing high CPU utilization?

          Your Thread Dump data is quite interesting and a perfect example of a non Thread safe HashMap combined with a read/write race condition:

          Thread #23 (hanging for 600 seconds+) is showing a potential infinite looping condition in at java.util.HashMap.getEntry(HashMap.java:361)
          Thread #22 (hanging for 600 seconds+) is showing a potential infinite looping condition while performing a resize() following a put() operation

          Weblogic Portal has some known issues which regards to usage of non Thread safe HashHap data structures; which under load can cause infinite looping and sharp surge of CPU (due to corruption of the internal HashMap index preventing exiting the internal loop); ultimately hanging the whole JVM.

          We had the same issue for one of our Weblogic Portal 10.0 production environment and Oracle provided a patch (via Collections.synchronizedMap() ).

          Please open a SR and Oracle, share these 2 Threads and discuss with the engineer about a patch / know issues regarding non Thread safe HashMap and infinite looping.

          Regards,
          P-H
          http://javaeesupportpatterns.blogspot.com/
          • 2. Re: Help on stuck thread
            user9503787
            Thanks for the info. It is rare and I don't know how to replicate. When it happend for this thread dump, there were only 3 requests. The rest of the threads were idle. But if any thread got request, it would stuck. So it wasn't caused by load.
            • 3. Re: Help on stuck thread
              PHCharbonneau
              Hi user9503787,

              Your situation is not surprising. We had the same situation on 2 of our production environments happening only once per week; only 2 or 3 threads showing the infinite looping HashMap condition with major CPU surge.

              Problem is hard to reproduce since replicating the internal HashMap structure corruption is unpredictable. You may need to run an endurance test (run for several hours/days) as opposed to load test.

              At this point, your next step right now is to share your Thread Dump data with Oracle support and get a possible patch.

              Regards,
              P-H