This discussion is archived
2 Replies Latest reply: Jan 18, 2013 1:25 AM by 985684 RSS

Stuck thread

721500 Newbie
Currently Being Moderated
Encounter thread stuck which cause the 100% CPU usage and eventually server crash.

Weblogic server version: 10.3.0

Thread Dump for one of the stuck thread:

"[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" id=14 idx=0x40 tid=11353 prio=1 alive, daemon

at java/util/WeakHashMap.transfer(WeakHashMap.java:468)
at java/util/WeakHashMap.resize(WeakHashMap.java:445)
at java/util/WeakHashMap.put(WeakHashMap.java:418)
at com/bea/core/repackaged/aspectj/lang/reflect/AjTypeSystem.getAjType(AjTypeSystem.java:50)
at com/bea/core/repackaged/springframework/aop/aspectj/annotation/AbstractAspectJAdvisorFactory.isAspect(AbstractAspectJAdvisorFactory.java:113)
at com/bea/core/repackaged/springframework/aop/aspectj/annotation/BeanFactoryAspectJAdvisorsBuilder.buildAspectJAdvisors(BeanFactoryAspectJAdvisorsBuilder.java:95)
at com/bea/core/repackaged/springframework/jee/intercept/InterceptionMetadata.addAspectJAdvisors(InterceptionMetadata.java:835)
at com/bea/core/repackaged/springframework/jee/intercept/InterceptionMetadata.createProxyIfNecessary(InterceptionMetadata.java:477)
at com/bea/core/repackaged/springframework/jee/intercept/InterceptionMetadata.createProxyIfNecessary(InterceptionMetadata.java:410)
at com/bea/core/repackaged/springframework/jee/spi/EjbComponentCreatorBrokerImpl.invokeCreateProxyIfNecessary(EjbComponentCreatorBrokerImpl.java:97)
at com/bea/core/repackaged/springframework/jee/spi/EjbComponentCreatorBrokerImpl.getBean(EjbComponentCreatorBrokerImpl.java:53)
at weblogic/ejb/container/injection/EjbComponentCreatorImpl.getBean(EjbComponentCreatorImpl.java:67)
at weblogic/ejb/container/manager/BaseEJBManager.createNewBeanInstance(BaseEJBManager.java:216)
at weblogic/ejb/container/manager/BaseEJBManager.allocateBean(BaseEJBManager.java:233)
at weblogic/ejb/container/manager/StatelessManager.createBean(StatelessManager.java:303)
at weblogic/ejb/container/pool/StatelessSessionPool.createBean(StatelessSessionPool.java:190)
at weblogic/ejb/container/pool/StatelessSessionPool.getBean(StatelessSessionPool.java:121)
at weblogic/ejb/container/manager/StatelessManager.preInvoke(StatelessManager.java:148)
at weblogic/ejb/container/internal/BaseLocalObject.preInvoke(BaseLocalObject.java:239)
at weblogic/ejb/container/internal/BaseLocalObject.preInvoke(BaseLocalObject.java:186)
at weblogic/ejb/container/internal/StatelessLocalObject.preInvoke(StatelessLocalObject.java:52)
....

"[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'" id=55 idx=0xd4 tid=11443 prio=1 alive, daemon

at java/util/WeakHashMap.containsKey(WeakHashMap.java:369)[optimized]
at com/bea/core/repackaged/aspectj/lang/reflect/AjTypeSystem.getAjType(AjTypeSystem.java:37)[inlined]
at com/bea/core/repackaged/springframework/aop/aspectj/annotation/AbstractAspectJAdvisorFactory.isAspect(AbstractAspectJAdvisorFactory.java:113)[optimized]
at com/bea/core/repackaged/springframework/aop/aspectj/annotation/BeanFactoryAspectJAdvisorsBuilder.buildAspectJAdvisors(BeanFactoryAspectJAdvisorsBuilder.java:95)[inlined]
at com/bea/core/repackaged/springframework/jee/intercept/InterceptionMetadata.addAspectJAdvisors(InterceptionMetadata.java:835)[optimized]
at com/bea/core/repackaged/springframework/jee/intercept/InterceptionMetadata.createProxyIfNecessary(InterceptionMetadata.java:477)[inlined]
at com/bea/core/repackaged/springframework/jee/intercept/InterceptionMetadata.createProxyIfNecessary(InterceptionMetadata.java:410)[optimized]
at com/bea/core/repackaged/springframework/jee/spi/EjbComponentCreatorBrokerImpl.invokeCreateProxyIfNecessary(EjbComponentCreatorBrokerImpl.java:97)[optimized]
at com/bea/core/repackaged/springframework/jee/spi/EjbComponentCreatorBrokerImpl.getBean(EjbComponentCreatorBrokerImpl.java:53)
at weblogic/ejb/container/injection/EjbComponentCreatorImpl.getBean(EjbComponentCreatorImpl.java:67)
at weblogic/ejb/container/manager/BaseEJBManager.createNewBeanInstance(BaseEJBManager.java:216)
at weblogic/ejb/container/manager/BaseEJBManager.allocateBean(BaseEJBManager.java:233)
at weblogic/ejb/container/manager/StatelessManager.createBean(StatelessManager.java:303)
at weblogic/ejb/container/pool/StatelessSessionPool.createBean(StatelessSessionPool.java:190)
at weblogic/ejb/container/pool/StatelessSessionPool.getBean(StatelessSessionPool.java:121)
at weblogic/ejb/container/manager/StatelessManager.preInvoke(StatelessManager.java:148)
at weblogic/ejb/container/internal/BaseLocalObject.preInvoke(BaseLocalObject.java:239)
at weblogic/ejb/container/internal/BaseLocalObject.preInvoke(BaseLocalObject.java:186)
at weblogic/ejb/container/internal/StatelessLocalObject.preInvoke(StatelessLocalObject.java:52)
...

All the stuck thread has similar thread dump. All are stuck at:

at java/util/WeakHashMap.containsKey(WeakHashMap.java:369)[optimized]
at com/bea/core/repackaged/aspectj/lang/reflect/AjTypeSystem.getAjType(AjTypeSystem.java:37)[inlined]

After do some research on aspectj, it seems that there is a bug on aspectj - Bug 269867 - Non synchronized access to WeakHashMap causes infinite loop (https://bugs.eclipse.org/bugs/show_bug.cgi?id=269867)

This bug has been fixed in version 1.6.4.

Since Weblogic repackaged aspectj jar, i was unable to upgrade the aspectj. Is there any way upgrade the aspectj in weblogic.

I had try install Weblogic 11g, but it seems that the aspectj used is same version (without the bug fix).

Pls advice. Thanks.
  • 1. Re: Stuck thread
    741198 Newbie
    Currently Being Moderated
    I have the same problem as yours,but I am in websphere.This is the heapdump below.

    at java/util/WeakHashMap.transfer(WeakHashMap.java:497(Compiled Code))
    at java/util/WeakHashMap.resize(WeakHashMap.java:462)
    at java/util/WeakHashMap.put(WeakHashMap.java:435(Compiled Code))
    at org/aspectj/weaver/World$TypeMap.put(World.java:884(Compiled Code))
    at org/aspectj/weaver/World.resolve(World.java:273(Compiled Code))
    at org/aspectj/weaver/World.resolve(World.java:185(Compiled Code))
    at org/aspectj/weaver/World.resolve(World.java:313(Compiled Code))
    at org/aspectj/weaver/internal/tools/PointcutExpressionImpl.couldMatchJoinPointsInType(PointcutExpressionImpl.java:76(Compiled Code))
    at org/springframework/aop/aspectj/AspectJExpressionPointcut.matches(AspectJExpressionPointcut.java:226(Compiled Code))
    at org/springframework/aop/support/AopUtils.canApply(AopUtils.java:189(Compiled Code))
    at org/springframework/aop/support/AopUtils.canApply(AopUtils.java:244(Compiled Code))
    at org/springframework/aop/support/AopUtils.findAdvisorsThatCanApply(AopUtils.java:266(Compiled Code))
    at org/springframework/aop/framework/autoproxy/AbstractAdvisorAutoProxyCreator.findAdvisorsThatCanApply(AbstractAdvisorAutoProxyCreator.java:113)
    at org/springframework/aop/framework/autoproxy/AbstractAdvisorAutoProxyCreator.findEligibleAdvisors(AbstractAdvisorAutoProxyCreator.java:85)
    at org/springframework/aop/framework/autoproxy/AbstractAdvisorAutoProxyCreator.getAdvicesAndAdvisorsForBean(AbstractAdvisorAutoProxyCreator.java:66) at org/springframework/aop/framework/autoproxy/AbstractAutoProxyCreator.postProcessAfterInitialization(AbstractAutoProxyCreator.java:296)
    at org/springframework/beans/factory/support/AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsAfterInitialization(AbstractAutowireCapableBeanFactory.java:331(Compiled Code))
    at com/opensymphony/xwork2/spring/SpringObjectFactory.buildBean(SpringObjectFactory.java:154)
    at com/opensymphony/xwork2/spring/SpringObjectFactory.buildBean(SpringObjectFactory.java:129) at com/opensymphony/xwork2/ObjectFactory.buildBean(ObjectFactory.java:143)
    at com/opensymphony/xwork2/ObjectFactory.buildInterceptor(ObjectFactory.java:184)
    at com/opensymphony/xwork2/config/providers/InterceptorBuilder.constructInterceptorReference(InterceptorBuilder.java:57)
    at com/opensymphony/xwork2/config/providers/XmlConfigurationProvider.lookupInterceptorReference(XmlConfigurationProvider.java:905)
    at com/opensymphony/xwork2/config/providers/XmlConfigurationProvider.loadInterceptorStack(XmlConfigurationProvider.java:743)
    at com/opensymphony/xwork2/config/providers/XmlConfigurationProvider.loadInterceptorStacks(XmlConfigurationProvider.java:756)
    at com/opensymphony/xwork2/config/providers/XmlConfigurationProvider.loadInterceptors(XmlConfigurationProvider.java:777)
    at com/opensymphony/xwork2/config/providers/XmlConfigurationProvider.addPackage(XmlConfigurationProvider.java:410)
    at com/opensymphony/xwork2/config/providers/XmlConfigurationProvider.loadPackages(XmlConfigurationProvider.java:239)
    at org/apache/struts2/config/StrutsXmlConfigurationProvider.loadPackages(StrutsXmlConfigurationProvider.java:111)
    at com/opensymphony/xwork2/config/impl/DefaultConfiguration.reload(DefaultConfiguration.java:152)
    at com/opensymphony/xwork2/config/ConfigurationManager.getConfiguration(ConfigurationManager.java:52)
    at org/apache/struts2/dispatcher/Dispatcher.init_PreloadConfiguration(Dispatcher.java:395)
    at org/apache/struts2/dispatcher/Dispatcher.init(Dispatcher.java:452)
    at org/apache/struts2/dispatcher/FilterDispatcher.init(FilterDispatcher.java:205)
    at plantix/web/core/dispatcher/FilterDispatcher.init(FilterDispatcher.java:39)
    at com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.init(FilterInstanceWrapper.java:142)
    at com/ibm/ws/webcontainer/filter/WebAppFilterManager._loadFilter(WebAppFilterManager.java:472)
    at com/ibm/ws/webcontainer/filter/WebAppFilterManager.loadFilter(WebAppFilterManager.java:386)
    at com/ibm/ws/webcontainer/filter/WebAppFilterManager.getFilterInstanceWrapper(WebAppFilterManager.java:245)
    at com/ibm/ws/webcontainer/filter/WebAppFilterManager.getFilterChain(WebAppFilterManager.java:303)
    at com/ibm/ws/webcontainer/filter/WebAppFilterManager.doFilter(WebAppFilterManager.java:797)
    at com/ibm/ws/webcontainer/filter/WebAppFilterManager.doFilter(WebAppFilterManager.java:680)
    at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:588)
    at com/ibm/ws/wswebcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:481)
    at com/ibm/wsspi/webcontainer/servlet/GenericServletWrapper.handleRequest(GenericServletWrapper.java:122)
    at com/ibm/ws/jsp/webcontainerext/AbstractJSPExtensionServletWrapper.handleRequest(AbstractJSPExtensionServletWrapper.java:232)
    at com/ibm/ws/jsp/webcontainerext/AbstractJSPExtensionProcessor.handleRequest(AbstractJSPExtensionProcessor.java:286)
    at com/ibm/ws/webcontainer/webapp/WebApp.handleRequest(WebApp.java:3507)
    at com/ibm/ws/webcontainer/webapp/WebGroup.handleRequest(WebGroup.java:269)
    at com/ibm/ws/webcontainer/WebContainer.handleRequest(WebContainer.java:815)
    at com/ibm/ws/wswebcontainer/WebContainer.handleRequest(WebContainer.java:1466) at com/ibm/ws/webcontainer/channel/WCChannelLink.ready(WCChannelLink.java:122)
    at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleDiscrimination(HttpInboundLink.java:458)
    at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleNewInformation(HttpInboundLink.java:387)
    at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.ready(HttpInboundLink.java:267)
    at com/ibm/ws/tcp/channel/impl/NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
    at com/ibm/ws/tcp/channel/impl/NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
    at com/ibm/ws/tcp/channel/impl/AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
    at com/ibm/io/async/AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
    at com/ibm/io/async/AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
    at com/ibm/io/async/AsyncFuture.completed(AsyncFuture.java:136)
    at com/ibm/io/async/ResultHandler.complete(ResultHandler.java:196)
    at com/ibm/io/async/ResultHandler.runEventProcessingLoop(ResultHandler.java:751)
    at com/ibm/io/async/ResultHandler$2.run(ResultHandler.java:881)
    at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1497)

    Thank you for your word,I solute this problem.
  • 2. Re: Stuck thread
    985684 Newbie
    Currently Being Moderated
    Hi,
    i too have the same problem.

    have your problem solved?

    Thanks in advance for your reply

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points