11 Replies Latest reply on Nov 28, 2011 10:39 AM by 846061

    Weblogic server + stuck threads

    846061
      Hi All,

      We have java application deployed on weblogic server 10.1.3.4 which is integrated with BPEL (SOA Oracle 11g). The application creates a task through BPEL. We have configured StuckThread Max Time to 600 secs. Now after certain time (5-7 days) the servers goes into warning state due to stuck threads and finally it gets crashed
      We noticed the OOM error in jrockit dump, could any one help me here as how to avoid this? your help is very much appreciated, thanks
        • 1. Re: Weblogic server + stuck threads
          846061
          Error message noticed in jrockit dump is

          Error Message: Illegal memory access. [54]
          • 2. Re: Weblogic server + stuck threads
            846061
            Did any of have encountered this error before? Is it possible for you to help me?
            • 3. Re: Weblogic server + stuck threads
              846061
              Thread dumps were taken , can some one please help analyzing the threaddump

              Blocked lock chains
              ===================
              Chain 3:
              "ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" id=28 idx=0x6c tid=29143 waiting for java/lang/String@0x71dd7d30 held by:
              "ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" id=26 idx=0x64 tid=29141 in chain 2

              Chain 4:
              "ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'" id=29 idx=0x70 tid=29144 waiting for java/lang/String@0x71dd7d30 held by:
              "ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" id=26 idx=0x64 tid=29141 in chain 2

              Open lock chains
              ================
              Chain 1:
              "Finalizer" id=7 idx=0x2c tid=29097 waiting for HTTPClient/StreamDemultiplexor@0xad8bbe68 held by:
              "[ACTIVE] ExecuteThread: '26' for queue: 'weblogic.kernel.Default (self-tuning)'" id=274 idx=0xd0 tid=9944 (active)

              Chain 2:
              "ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" id=27 idx=0x68 tid=29142 waiting for java/lang/String@0x71dd7d30 held by:
              "ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" id=26 idx=0x64 tid=29141 (active)
              • 4. Re: Weblogic server + stuck threads
                PHCharbonneau
                Hi Aravindh,

                This problem could be a symptom of a Java Heap memory leak e.g. Java Heap growing over time and reaching break point after 5-7 days.
                Once your JRockit Java Heap is full, OutOfMemoryError will be observed along with severe performance degradation, Stuck Threads etc.

                Can you please provide the following data below:

                -     Full Thread Stack Trace of a few Weblogic sample Threads from your generated Thread Dump
                -     Sample of your Weblogic logs showing the java.lang.OutOfMemoryError along with Stack Trace if available
                -     The version of JRockit that you are currently using along with your Java Heap size arguments e.g. Xms & Xmx
                -     
                If you are using JRockit R28+ then please add the following parameters below. This will trigger JRockit to generate a Heap Dump following an OutOfMemoryError in HPROF binary format. Such Heap Dump can then be analyzed to determine the potential source(s) of memory leak.
                -XX:+HeapDumpOnOutOfMemoryError

                Find below also a tutorial on how to load and analyze HPROF Heap Dump files using the Memory Analyzer tool (MAT).

                Hope this helps.

                http://javaeesupportpatterns.blogspot.com/2011/11/hprof-memory-leak-analysis-tutorial.html
                Regards,
                P-H
                1 person found this helpful
                • 5. Re: Weblogic server + stuck threads
                  846061
                  Hi PHCharbonneau,

                  Thank you so much for your response
                  Please find the information you asked for, looking forward to your response, thanks


                  1)Oracle JRockit(R) build R28.1.1-14-139783-1.6.0_22-20101206-0241-linux-ia32, compiled mode
                  GC mode: Garbage collection optimized for throughput, strategy: genparpar

                  2)JAVA Memory arguments: -Xms1536m -Xmx1536m


                  Attaching the complete thread dump below
                  • 6. Re: Weblogic server + stuck threads
                    846061
                    Hi PHCharbonneau,

                    If you are looking for the stuck thread trace PFA below thanks


                    "[STUCK] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" id=21 idx=0x68 tid=17842 prio=1 alive, waiting, native_blocked, daemon

                    -- Waiting for notification on: weblogic/transaction/internal/ServerTransactionImpl@0xb7910e20[fat lock]

                    at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                    at java/lang/Object.wait(J)V(Native Method)

                    at weblogic/transaction/internal/ServerTransactionImpl.globalRetryRollback(ServerTransactionImpl.java:3133)[optimized]

                    ^-- Lock released while waiting: weblogic/transaction/internal/ServerTransactionImpl@0xb7910e20[fat lock]

                    at weblogic/transaction/internal/ServerTransactionImpl.globalRollback(ServerTransactionImpl.java:2872)[optimized]

                    at weblogic/transaction/internal/ServerTransactionImpl.internalCommit(ServerTransactionImpl.java:343)[inlined]

                    at weblogic/transaction/internal/ServerTransactionImpl.commit(ServerTransactionImpl.java:239)[optimized]

                    at weblogic/ejb/container/internal/BaseLocalObject.postInvoke1(BaseLocalObject.java:622)[optimized]

                    at weblogic/ejb/container/internal/BaseLocalObject.__WL_postInvokeTxRetry(BaseLocalObject.java:455)[inlined]

                    at weblogic/ejb/container/internal/SessionLocalMethodInvoker.invoke(SessionLocalMethodInvoker.java:52)[inlined]

                    at com/collaxa/cube/engine/ejb/impl/bpel/BPELEngineBean_51369e_ICubeEngineLocalBeanImpl.syncCreateAndInvoke(Loracle/soa/management/ComponentDN;Ljava/lang/String;Ljava/lang/String;Loracle/fabric/common/NormalizedMessage;)Lcom/collaxa/cube/engine/dispatch/message/IMessage;(Unknown Source)[optimized]

                    at com/collaxa/cube/engine/delivery/DeliveryHandler.callCreateAndInvoke(DeliveryHandler.java:790)[optimized]

                    at com/collaxa/cube/engine/delivery/DeliveryHandler.initialRequestAnyType(DeliveryHandler.java:528)[optimized]

                    at com/collaxa/cube/engine/delivery/DeliveryHandler.initialRequest(DeliveryHandler.java:487)[inlined]

                    at com/collaxa/cube/engine/delivery/DeliveryHandler.request(DeliveryHandler.java:162)[inlined]

                    at com/collaxa/cube/engine/ejb/impl/CubeDeliveryBean.request(CubeDeliveryBean.java:493)[optimized]

                    at sun/reflect/GeneratedMethodAccessor1703.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)[optimized]

                    at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]

                    at java/lang/reflect/Method.invoke(Method.java:597)[optimized]

                    at com/bea/core/repackaged/springframework/aop/support/AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)[inlined]

                    at com/bea/core/repackaged/springframework/aop/framework/ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)[inlined]

                    at com/bea/core/repackaged/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)[optimized]

                    at com/bea/core/repackaged/springframework/jee/intercept/MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:104)[optimized]

                    at oracle/security/jps/ee/ejb/JpsAbsInterceptor$1.run(JpsAbsInterceptor.java:94)[inlined]

                    at jrockit/vm/AccessController.doPrivileged(AccessController.java:254)[inlined]

                    at oracle/security/jps/util/JpsSubject.doAsPrivileged(JpsSubject.java:313)[inlined]

                    at oracle/security/jps/ee/util/JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:413)[inlined]

                    at oracle/security/jps/ee/ejb/JpsAbsInterceptor.runJaasMode(JpsAbsInterceptor.java:81)[inlined]

                    at oracle/security/jps/ee/ejb/JpsAbsInterceptor.intercept(JpsAbsInterceptor.java:112)[optimized]

                    at oracle/security/jps/ee/ejb/JpsInterceptor.intercept(JpsInterceptor.java:105)[inlined]

                    at sun/reflect/GeneratedMethodAccessor1603.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)[optimized]

                    at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]

                    at java/lang/reflect/Method.invoke(Method.java:597)[optimized]

                    at com/bea/core/repackaged/springframework/aop/support/AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)[inlined]

                    at com/bea/core/repackaged/springframework/jee/intercept/JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:69)[optimized]

                    at com/bea/core/repackaged/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)[optimized]

                    at com/bea/core/repackaged/springframework/aop/support/DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)[inlined]

                    at com/bea/core/repackaged/springframework/aop/support/DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)[optimized]

                    at com/bea/core/repackaged/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)[optimized]

                    at com/bea/core/repackaged/springframework/jee/spi/MethodInvocationVisitorImpl.visit(MethodInvocationVisitorImpl.java:37)[optimized]

                    at weblogic/ejb/container/injection/EnvironmentInterceptorCallbackImpl.callback(EnvironmentInterceptorCallbackImpl.java:54)[optimized]

                    at com/bea/core/repackaged/springframework/jee/spi/EnvironmentInterceptor.invoke(EnvironmentInterceptor.java:50)[optimized]

                    at com/bea/core/repackaged/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)[optimized]

                    at com/bea/core/repackaged/springframework/aop/interceptor/ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)[optimized]

                    at com/bea/core/repackaged/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)[optimized]

                    at com/bea/core/repackaged/springframework/aop/support/DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)[inlined]

                    at com/bea/core/repackaged/springframework/aop/support/DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)[optimized]

                    at com/bea/core/repackaged/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)[optimized]

                    at com/bea/core/repackaged/springframework/aop/framework/JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)[optimized]

                    at $Proxy288.request(Loracle/soa/management/ComponentDN;Ljava/lang/String;Ljava/lang/String;Loracle/fabric/common/NormalizedMessage;Z)Loracle/fabric/common/NormalizedMessage;(Unknown Source)[optimized]

                    at com/collaxa/cube/engine/ejb/impl/bpel/BPELDeliveryBean_5k948i_ICubeDeliveryLocalBeanImpl.__WL_invoke(Ljava/lang/Object;[Ljava/lang/Object;I)Ljava/lang/Object;(Unknown Source)[inlined]

                    at weblogic/ejb/container/internal/SessionLocalMethodInvoker.invoke(SessionLocalMethodInvoker.java:39)[inlined]

                    at com/collaxa/cube/engine/ejb/impl/bpel/BPELDeliveryBean_5k948i_ICubeDeliveryLocalBeanImpl.request(Loracle/soa/management/ComponentDN;Ljava/lang/String;Ljava/lang/String;Loracle/fabric/common/NormalizedMessage;Z)Loracle/fabric/common/NormalizedMessage;(Unknown Source)[optimized]

                    at oracle/fabric/CubeServiceEngine.request(CubeServiceEngine.java:380)[optimized]

                    at oracle/integration/platform/blocks/mesh/SynchronousMessageHandler.doRequest(SynchronousMessageHandler.java:139)[optimized]

                    at oracle/integration/platform/blocks/mesh/MessageRouter.request(MessageRouter.java:179)[optimized]

                    at oracle/integration/platform/blocks/mesh/MeshImpl.request(MeshImpl.java:155)[optimized]

                    at sun/reflect/GeneratedMethodAccessor1699.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)[optimized]

                    at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]

                    at java/lang/reflect/Method.invoke(Method.java:597)[optimized]

                    at org/springframework/aop/support/AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)[inlined]

                    at org/springframework/aop/framework/ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)[inlined]

                    at org/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)[optimized]

                    at oracle/integration/platform/metrics/PhaseEventAspect.invoke(PhaseEventAspect.java:59)[optimized]

                    at org/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)[optimized]

                    at org/springframework/aop/framework/JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)[optimized]

                    at $Proxy305.request(Loracle/fabric/common/NormalizedMessage;Loracle/fabric/common/Operation;Loracle/fabric/common/InvocationContext;)Loracle/fabric/common/NormalizedMessage;(Unknown Source)[optimized]

                    at oracle/integration/platform/blocks/soap/WebServiceEntryBindingComponent.doMessageProcessing(WebServiceEntryBindingComponent.java:1281)[optimized]

                    at oracle/integration/platform/blocks/soap/WebServiceEntryBindingComponent.processIncomingMessage(WebServiceEntryBindingComponent.java:859)[optimized]

                    at oracle/integration/platform/blocks/soap/FabricProvider.processMessage(FabricProvider.java:113)[optimized]

                    at oracle/j2ee/ws/server/provider/ProviderProcessor.doEndpointProcessing(ProviderProcessor.java:1187)[optimized]

                    at oracle/j2ee/ws/server/WebServiceProcessor.invokeEndpointImplementation(WebServiceProcessor.java:1081)[optimized]

                    at oracle/j2ee/ws/server/provider/ProviderProcessor.doRequestProcessing(ProviderProcessor.java:581)[optimized]

                    at oracle/j2ee/ws/server/WebServiceProcessor.processRequest(WebServiceProcessor.java:232)[optimized]

                    at oracle/j2ee/ws/server/WebServiceProcessor.doService(WebServiceProcessor.java:192)[inlined]

                    at oracle/j2ee/ws/server/WebServiceServlet.doPost(WebServiceServlet.java:459)[inlined]

                    at oracle/integration/platform/blocks/soap/FabricProviderServlet.doPost(FabricProviderServlet.java:507)[optimized]

                    at javax/servlet/http/HttpServlet.service(HttpServlet.java:727)[optimized]

                    at javax/servlet/http/HttpServlet.service(HttpServlet.java:820)[optimized]

                    at weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)

                    at weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)[inlined]

                    at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:300)[optimized]

                    at weblogic/servlet/internal/TailFilter.doFilter(TailFilter.java:26)[optimized]

                    at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]

                    at oracle/security/jps/ee/http/JpsAbsFilter$1.run(JpsAbsFilter.java:111)[optimized]

                    at jrockit/vm/AccessController.doPrivileged(AccessController.java:254)[inlined]

                    at oracle/security/jps/util/JpsSubject.doAsPrivileged(JpsSubject.java:313)[inlined]

                    at oracle/security/jps/ee/util/JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:413)[inlined]

                    at oracle/security/jps/ee/http/JpsAbsFilter.runJaasMode(JpsAbsFilter.java:94)[inlined]

                    at oracle/security/jps/ee/http/JpsAbsFilter.doFilter(JpsAbsFilter.java:161)[optimized]

                    at oracle/security/jps/ee/http/JpsFilter.doFilter(JpsFilter.java:71)[optimized]

                    at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]

                    at oracle/dms/servlet/DMSServletFilter.doFilter(DMSServletFilter.java:136)[optimized]

                    at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[inlined]

                    at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3715)[inlined]

                    at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)[optimized]

                    at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)[optimized]

                    at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:120)[inlined]

                    at weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServletContext.java:2277)[inlined]

                    at weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext.java:2183)[optimized]

                    at weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:1454)[optimized]

                    at weblogic/work/ExecuteThread.execute(ExecuteThread.java:207)[optimized]

                    at weblogic/work/ExecuteThread.run(ExecuteThread.java:176)

                    at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
                    • 7. Re: Weblogic server + stuck threads
                      846061
                      Also,


                      "[STUCK] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'" id=65 idx=0x104 tid=17900 prio=1 alive, waiting, native_blocked, daemon

                      -- Waiting for notification on: weblogic/transaction/internal/ServerTransactionImpl@0xb78fb1a8[fat lock]

                      at jrockit/vm/Threads.waitForNotifySignal(JLjava/lang/Object;)Z(Native Method)

                      at java/lang/Object.wait(J)V(Native Method)

                      at weblogic/transaction/internal/ServerTransactionImpl.globalRetryRollback(ServerTransactionImpl.java:3133)[optimized]

                      ^-- Lock released while waiting: weblogic/transaction/internal/ServerTransactionImpl@0xb78fb1a8[fat lock]

                      at weblogic/transaction/internal/ServerTransactionImpl.globalRollback(ServerTransactionImpl.java:2872)[optimized]

                      at weblogic/transaction/internal/ServerTransactionImpl.internalCommit(ServerTransactionImpl.java:343)[inlined]

                      at weblogic/transaction/internal/ServerTransactionImpl.commit(ServerTransactionImpl.java:239)[optimized]

                      at weblogic/ejb/container/internal/BaseLocalObject.postInvoke1(BaseLocalObject.java:622)[optimized]

                      at weblogic/ejb/container/internal/BaseLocalObject.__WL_postInvokeTxRetry(BaseLocalObject.java:455)[inlined]

                      at weblogic/ejb/container/internal/SessionLocalMethodInvoker.invoke(SessionLocalMethodInvoker.java:52)[inlined]

                      at com/collaxa/cube/engine/ejb/impl/bpel/BPELEngineBean_51369e_ICubeEngineLocalBeanImpl.syncCreateAndInvoke(Loracle/soa/management/ComponentDN;Ljava/lang/String;Ljava/lang/String;Loracle/fabric/common/NormalizedMessage;)Lcom/collaxa/cube/engine/dispatch/message/IMessage;(Unknown Source)[optimized]

                      at com/collaxa/cube/engine/delivery/DeliveryHandler.callCreateAndInvoke(DeliveryHandler.java:790)[optimized]

                      at com/collaxa/cube/engine/delivery/DeliveryHandler.initialRequestAnyType(DeliveryHandler.java:528)[optimized]

                      at com/collaxa/cube/engine/delivery/DeliveryHandler.initialRequest(DeliveryHandler.java:487)[inlined]

                      at com/collaxa/cube/engine/delivery/DeliveryHandler.request(DeliveryHandler.java:162)[inlined]

                      at com/collaxa/cube/engine/ejb/impl/CubeDeliveryBean.request(CubeDeliveryBean.java:493)[optimized]

                      at sun/reflect/GeneratedMethodAccessor1703.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)[optimized]

                      at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]

                      at java/lang/reflect/Method.invoke(Method.java:597)[optimized]

                      at com/bea/core/repackaged/springframework/aop/support/AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)[inlined]

                      at com/bea/core/repackaged/springframework/aop/framework/ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)[inlined]

                      at com/bea/core/repackaged/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)[optimized]

                      at com/bea/core/repackaged/springframework/jee/intercept/MethodInvocationInvocationContext.proceed(MethodInvocationInvocationContext.java:104)[optimized]

                      at oracle/security/jps/ee/ejb/JpsAbsInterceptor$1.run(JpsAbsInterceptor.java:94)[inlined]

                      at jrockit/vm/AccessController.doPrivileged(AccessController.java:254)[inlined]

                      at oracle/security/jps/util/JpsSubject.doAsPrivileged(JpsSubject.java:313)[inlined]

                      at oracle/security/jps/ee/util/JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:413)[inlined]

                      at oracle/security/jps/ee/ejb/JpsAbsInterceptor.runJaasMode(JpsAbsInterceptor.java:81)[inlined]

                      at oracle/security/jps/ee/ejb/JpsAbsInterceptor.intercept(JpsAbsInterceptor.java:112)[optimized]

                      at oracle/security/jps/ee/ejb/JpsInterceptor.intercept(JpsInterceptor.java:105)[inlined]

                      at sun/reflect/GeneratedMethodAccessor1603.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)[optimized]

                      at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]

                      at java/lang/reflect/Method.invoke(Method.java:597)[optimized]

                      at com/bea/core/repackaged/springframework/aop/support/AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)[inlined]

                      at com/bea/core/repackaged/springframework/jee/intercept/JeeInterceptorInterceptor.invoke(JeeInterceptorInterceptor.java:69)[optimized]

                      at com/bea/core/repackaged/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)[optimized]

                      at com/bea/core/repackaged/springframework/aop/support/DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)[inlined]

                      at com/bea/core/repackaged/springframework/aop/support/DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)[optimized]

                      at com/bea/core/repackaged/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)[optimized]

                      at com/bea/core/repackaged/springframework/jee/spi/MethodInvocationVisitorImpl.visit(MethodInvocationVisitorImpl.java:37)[optimized]

                      at weblogic/ejb/container/injection/EnvironmentInterceptorCallbackImpl.callback(EnvironmentInterceptorCallbackImpl.java:54)[optimized]

                      at com/bea/core/repackaged/springframework/jee/spi/EnvironmentInterceptor.invoke(EnvironmentInterceptor.java:50)[optimized]

                      at com/bea/core/repackaged/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)[optimized]

                      at com/bea/core/repackaged/springframework/aop/interceptor/ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)[optimized]

                      at com/bea/core/repackaged/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)[optimized]

                      at com/bea/core/repackaged/springframework/aop/support/DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)[inlined]

                      at com/bea/core/repackaged/springframework/aop/support/DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)[optimized]

                      at com/bea/core/repackaged/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)[optimized]

                      at com/bea/core/repackaged/springframework/aop/framework/JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)[optimized]

                      at $Proxy288.request(Loracle/soa/management/ComponentDN;Ljava/lang/String;Ljava/lang/String;Loracle/fabric/common/NormalizedMessage;Z)Loracle/fabric/common/NormalizedMessage;(Unknown Source)[optimized]

                      at com/collaxa/cube/engine/ejb/impl/bpel/BPELDeliveryBean_5k948i_ICubeDeliveryLocalBeanImpl.__WL_invoke(Ljava/lang/Object;[Ljava/lang/Object;I)Ljava/lang/Object;(Unknown Source)[inlined]

                      at weblogic/ejb/container/internal/SessionLocalMethodInvoker.invoke(SessionLocalMethodInvoker.java:39)[inlined]

                      at com/collaxa/cube/engine/ejb/impl/bpel/BPELDeliveryBean_5k948i_ICubeDeliveryLocalBeanImpl.request(Loracle/soa/management/ComponentDN;Ljava/lang/String;Ljava/lang/String;Loracle/fabric/common/NormalizedMessage;Z)Loracle/fabric/common/NormalizedMessage;(Unknown Source)[optimized]

                      at oracle/fabric/CubeServiceEngine.request(CubeServiceEngine.java:380)[optimized]

                      at oracle/integration/platform/blocks/mesh/SynchronousMessageHandler.doRequest(SynchronousMessageHandler.java:139)[optimized]

                      at oracle/integration/platform/blocks/mesh/MessageRouter.request(MessageRouter.java:179)[optimized]

                      at oracle/integration/platform/blocks/mesh/MeshImpl.request(MeshImpl.java:155)[optimized]

                      at sun/reflect/GeneratedMethodAccessor1699.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)[optimized]

                      at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]

                      at java/lang/reflect/Method.invoke(Method.java:597)[optimized]

                      at org/springframework/aop/support/AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)[inlined]

                      at org/springframework/aop/framework/ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)[inlined]

                      at org/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)[optimized]

                      at oracle/integration/platform/metrics/PhaseEventAspect.invoke(PhaseEventAspect.java:59)[optimized]

                      at org/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)[optimized]

                      at org/springframework/aop/framework/JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)[optimized]

                      at $Proxy305.request(Loracle/fabric/common/NormalizedMessage;Loracle/fabric/common/Operation;Loracle/fabric/common/InvocationContext;)Loracle/fabric/common/NormalizedMessage;(Unknown Source)[optimized]

                      at oracle/integration/platform/blocks/soap/WebServiceEntryBindingComponent.doMessageProcessing(WebServiceEntryBindingComponent.java:1281)[optimized]

                      at oracle/integration/platform/blocks/soap/WebServiceEntryBindingComponent.processIncomingMessage(WebServiceEntryBindingComponent.java:859)[optimized]

                      at oracle/integration/platform/blocks/soap/FabricProvider.processMessage(FabricProvider.java:113)[optimized]

                      at oracle/j2ee/ws/server/provider/ProviderProcessor.doEndpointProcessing(ProviderProcessor.java:1187)[optimized]

                      at oracle/j2ee/ws/server/WebServiceProcessor.invokeEndpointImplementation(WebServiceProcessor.java:1081)[optimized]

                      at oracle/j2ee/ws/server/provider/ProviderProcessor.doRequestProcessing(ProviderProcessor.java:581)[optimized]

                      at oracle/j2ee/ws/server/WebServiceProcessor.processRequest(WebServiceProcessor.java:232)[optimized]

                      at oracle/j2ee/ws/server/WebServiceProcessor.doService(WebServiceProcessor.java:192)[inlined]

                      at oracle/j2ee/ws/server/WebServiceServlet.doPost(WebServiceServlet.java:459)[inlined]

                      at oracle/integration/platform/blocks/soap/FabricProviderServlet.doPost(FabricProviderServlet.java:507)[optimized]

                      at javax/servlet/http/HttpServlet.service(HttpServlet.java:727)[optimized]

                      at javax/servlet/http/HttpServlet.service(HttpServlet.java:820)[optimized]

                      at weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)

                      at weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)[inlined]

                      at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:300)[optimized]

                      at weblogic/servlet/internal/TailFilter.doFilter(TailFilter.java:26)[optimized]

                      at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]

                      at oracle/security/jps/ee/http/JpsAbsFilter$1.run(JpsAbsFilter.java:111)[optimized]

                      at jrockit/vm/AccessController.doPrivileged(AccessController.java:254)[inlined]

                      at oracle/security/jps/util/JpsSubject.doAsPrivileged(JpsSubject.java:313)[inlined]

                      at oracle/security/jps/ee/util/JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:413)[inlined]

                      at oracle/security/jps/ee/http/JpsAbsFilter.runJaasMode(JpsAbsFilter.java:94)[inlined]

                      at oracle/security/jps/ee/http/JpsAbsFilter.doFilter(JpsAbsFilter.java:161)[optimized]

                      at oracle/security/jps/ee/http/JpsFilter.doFilter(JpsFilter.java:71)[optimized]

                      at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]

                      at oracle/dms/servlet/DMSServletFilter.doFilter(DMSServletFilter.java:136)[optimized]

                      at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[inlined]

                      at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3715)[inlined]

                      at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)[optimized]

                      at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)[optimized]

                      at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:120)[inlined]

                      at weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServletContext.java:2277)[inlined]

                      at weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext.java:2183)[optimized]

                      at weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:1454)[optimized]

                      at weblogic/work/ExecuteThread.execute(ExecuteThread.java:207)[optimized]

                      at weblogic/work/ExecuteThread.run(ExecuteThread.java:176)

                      at jrockit/vm/RNI.c2java(IIIII)V(Native Method)

                      -- end of trace
                      • 8. Re: Weblogic server + stuck threads
                        846061
                        Regarding

                        Sample of your Weblogic logs showing the java.lang.OutOfMemoryError along with Stack Trace if available

                        Please find below


                        [ACTIVE] ExecuteThread: '20' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <532776f80ba738ab:-48a8d4c8:133068f0cee:-8000-00000000000001a5> <1322080967160> <BEA-080003> <RuntimeException thrown by rmi server: javax.management.remote.rmi.RMIConnectionImpl.invoke(Ljavax.management.ObjectName;Ljava.lang.String;Ljava.rmi.MarshalledObject;[Ljava.lang.String;Ljavax.security.auth.Subject;)
                        javax.management.RuntimeErrorException: java.lang.OutOfMemoryError: allocLargeObjectOrArray: [B, size 1335312.
                        javax.management.RuntimeErrorException: java.lang.OutOfMemoryError: allocLargeObjectOrArray: [B, size 1335312
                                at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:858)
                                at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:869)
                                at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:838)
                                at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
                                at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
                                at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
                                at weblogic.management.mbeanservers.internal.JMXContextInterceptor.invoke(JMXContextInterceptor.java:263)
                                at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
                                at weblogic.management.mbeanservers.internal.SecurityInterceptor.invoke(SecurityInterceptor.java:444)
                                at weblogic.management.jmx.mbeanserver.WLSMBeanServer.invoke(WLSMBeanServer.java:323)
                                at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder$11.run(JMXConnectorSubjectForwarder.java:661)
                                at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder.invoke(JMXConnectorSubjectForwarder.java:654)
                                at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
                                at javax.management.remote.rmi.RMIConnectionImpl_WLSkel.invoke(Unknown Source)
                                at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:668)
                                at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:118)
                                at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
                                at weblogic.work.ExecuteThread.run(ExecuteThread.java:176)
                        Caused By: java.lang.OutOfMemoryError: allocLargeObjectOrArray: [B, size 1335312
                                at java.util.Arrays.copyOf(Arrays.java:2786)
                                at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
                                at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1847)
                          at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
                                at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1847)
                                at java.io.ObjectOutputStream$BlockDataOutputStream.flush(ObjectOutputStream.java:1792)
                                at java.io.ObjectOutputStream.flush(ObjectOutputStream.java:699)
                                at java.io.ObjectOutputStream.close(ObjectOutputStream.java:720)
                                at oracle.dms.jmx.app.ServerAggreMBeanImpl.getTableGroupInBytes(ServerAggreMBeanImpl.java:328)
                                at oracle.dms.jmx.app.ServerAggreMBeanWrapper.invokeOperation(ServerAggreMBeanWrapper.java:71)
                                at oracle.dms.jmx.app.AggreMBeanWrapper.invoke(AggreMBeanWrapper.java:167)
                                at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
                                at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
                                at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
                                at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
                                at weblogic.management.mbeanservers.internal.JMXContextInterceptor.invoke(JMXContextInterceptor.java:263)
                                at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
                                at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
                                at weblogic.management.mbeanservers.internal.SecurityInterceptor.invoke(SecurityInterceptor.java:444)
                                at weblogic.management.jmx.mbeanserver.WLSMBeanServer.invoke(WLSMBeanServer.java:323)
                                at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder$11$1.run(JMXConnectorSubjectForwarder.java:663)
                                at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder$11.run(JMXConnectorSubjectForwarder.java:661)
                                at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
                                at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder.invoke(JMXConnectorSubjectForwarder.java:654)
                                at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
                                at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
                                at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
                                at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1367)
                                at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
                                at javax.management.remote.rmi.RMIConnectionImpl_WLSkel.invoke(Unknown Source)
                                at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:667)
                                at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:522)
                                at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
                                at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
                                at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:518)
                                at weblogic.rmi.internal.wls.WLSExecuteRequest.run(WLSExecuteRequest.java:118)
                                at weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
                                at weblogic.work.ExecuteThread.run(ExecuteThread.java:176)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
                        • 9. Re: Weblogic server + stuck threads
                          846061
                          Hi just wanted to update here,

                          We have EM enabled to monitor the heap usage and they looked pretty much okay
                          No spikes are noticed, the average value for the last 31 days is 1200 MB where as we have set the critical threshold to 2400 MB, so this should be okay right?

                          Edited by: Aravindh on Nov 23, 2011 11:42 PM
                          • 10. Re: Weblogic server + stuck threads
                            PHCharbonneau
                            Hi Aravindh,

                            The error indicates you triggered an OutOfMemoryError during a TLA allocation. TLA stands for Thread Local Area which is basically is a chunk of free space reserved on the heap / nursery and given to a thread for its exclusive use. This could be a problem by itself if you are dealing with large TLA allocations or a symptom that you are running very low on Java Heap space.

                            That being said, before attempting to increase the minimum TLA size, I strongly recommend that you perform the following actions below:

                            1)     Turn on JRockit verbose GC by adding the following parameters:

                            ## For GC and memory analysis.
                            -Xverbose:memory,gc
                                 
                            ## Redirects verbose output to the designated file.
                            -Xverboselog:verbose-jrockit.log
                                 
                            ## Prints a formatted date before each verbose line.
                            -Xverbosetimestamp

                            Verbose GC will allow you to monitor your Java Heap memory utilization day after day and determine if you are dealing with a memory leak vs. a sudden trigger event. Since your failure rate is 5-7 days, chances are you are dealing with a Java Heap leak.
                            Please provide the verbose GC output to this post once generated.

                            2)     Add -XX:+HeapDumpOnOutOfMemoryError within your JRockit JVM start-up arguments. This will trigger a JRockit binary Heap Dump (HPROF format) that you can then analyze (as per my original reply). Binary Heap Dump analysis via MAT tool is very useful to pinpoint source(s) of application / middleware memory leaks.


                            Regards,
                            P-H
                            1 person found this helpful
                            • 11. Re: Weblogic server + stuck threads
                              846061
                              Thanks P-H
                              Am enabling the -XX:-HeapDumpOnOutOfMemoryError and will try to get the root cause, thanks for your help once again