2 Replies Latest reply on Oct 3, 2013 4:56 PM by PHCharbonneau

    Performance Issues in weblogic managed server

    918074

      Hi,

       

      We are facing performance issues in our managed servers in weblogic domain and in logs it is showing this error continuously.

       

       

      <Oct 2, 2013 10:39:27 AM MST> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '47' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "700" seconds working on the request "Workmanager: OIMMDBWorkManager, Version: 0, Scheduled=false, Started=true, Started time: 700360 ms

      ", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:

      Thread-294 "[STUCK] ExecuteThread: '47' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, blocked, priority=1, DAEMON> {

          -- Blocked trying to get lock: java.io.PrintStream@41a53700[not locked]

          java.io.PrintStream.println(PrintStream.java:755)

          com.wyndham.operations.user.eventHandlers.PostCreateOperationsEventHandler.execute(PostCreateOperationsEventHandler.java:60)

          oracle.iam.platform.kernel.impl.OrchProcessData.runPostProcessEvents(OrchProcessData.java:1225)

          oracle.iam.platform.kernel.impl.OrchProcessData.runEvents(OrchProcessData.java:665)

          oracle.iam.platform.kernel.impl.OrchProcessData.executeEvents(OrchProcessData.java:268)

          oracle.iam.platform.kernel.impl.OrchestrationEngineImpl.resumeProcess(OrchestrationEngineImpl.java:810)

          oracle.iam.platform.kernel.impl.OrchestrationEngineImpl.notifyParentProcess(OrchestrationEngineImpl.java:971)

          oracle.iam.platform.kernel.impl.OrchProcessData.runEvents(OrchProcessData.java:665)

          oracle.iam.platform.kernel.impl.OrchProcessData.executeEvents(OrchProcessData.java:268)

          oracle.iam.platform.kernel.impl.OrchestrationEngineImpl.resumeProcess(OrchestrationEngineImpl.java:810)

          oracle.iam.platform.kernel.impl.OrchestrationEngineImpl.resumeChildProcess(OrchestrationEngineImpl.java:880)

          oracle.iam.platform.kernel.impl.OrchProcessData.handleAdditionalChanges(OrchProcessData.java:552)

          oracle.iam.platform.kernel.impl.OrchProcessData.runEvents(OrchProcessData.java:665)

          oracle.iam.platform.kernel.impl.OrchProcessData.executeEvents(OrchProcessData.java:268)

          oracle.iam.platform.kernel.impl.OrchestrationEngineImpl.resumeProcess(OrchestrationEngineImpl.java:815)

          oracle.iam.platform.kernel.impl.OrchestrationEngineImpl.resumeProcess(OrchestrationEngineImpl.java:843)

          oracle.iam.platform.kernel.impl.OrhestrationAsyncTask.execute(OrhestrationAsyncTask.java:97)

          oracle.iam.platform.async.impl.TaskExecutor.executeUnmanagedTask(TaskExecutor.java:99)

          oracle.iam.platform.async.impl.TaskExecutor.execute(TaskExecutor.java:52)

          oracle.iam.platform.async.messaging.MessageReceiver.onMessage(MessageReceiver.java:60)

          sun.reflect.GeneratedMethodAccessor1493.invoke(Unknown Source)

          sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

          java.lang.reflect.Method.invoke(Method.java:575)

          com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)

          com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)

          com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)

          com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:86)

          com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)

          com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)

          com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)

          com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)

          com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:151)

          $Proxy460.onMessage(Unknown Source)

          weblogic.ejb.container.internal.MDListener.execute(MDListener.java:518)

          weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:467)

          weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:327)

          weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4638)

          weblogic.jms.client.JMSSession.execute(JMSSession.java:4134)

          weblogic.jms.client.JMSSession.executeMessage(JMSSession.java:3820)

          weblogic.jms.client.JMSSession.access$000(JMSSession.java:115)

       

      We have performed workload manager tuning in weblogic console as below.

       

      The property ‘Maximum Threads Constraint’ of work managers OIMMDBWorkManager = 80 and OIMUIWorkManager = 20

       

      Please suggest the solution.

        • 1. Re: Performance Issues in weblogic managed server
          Rosario

          Hi,

           

          Seems problem with IO operation in the system. Take strace for the process and your system admin can confirm about why/where the delay.

           

          Regards

          Rosario

          • 2. Re: Performance Issues in weblogic managed server
            PHCharbonneau

            Hi,

             

            If you look more closely at the STUCK Thread stack trace, you will notice that the thread is attempting to acquire a LOCK on the following Object monitor: java.io.PrintStream@41a53700. The native thread state is also in a BLOCKED state.

             

            My primary recommendation is to generate a full JVM Thread Dump (next re-occurence) so we can identify the potential culprit Thread holding such lock. One more thing, please see if you have multiple snapshots of this thread 294?

             

            I also recommend to review the following source code:

             

            com.wyndham.operations.user.eventHandlers.PostCreateOperationsEventHandler.execute(PostCreateOperationsEventHandler.java:60)


            Sharing the same Stream object in this case would trigger thread lock contention given java.io.PrintStream has built-in synchronization ex:


               public void println(String x) {

              synchronized (this) {

                 print(x);

                 newLine();

              }

                }

             

             

            Regards,

            P-H

            http://javaeesupportpatterns.blogspot.com