2 Replies Latest reply: Oct 3, 2013 11:56 AM by PHCharbonneau RSS

    Performance Issues in weblogic managed server

    918074

      Hi,

       

      We are facing performance issues in our managed servers in weblogic domain and in logs it is showing this error continuously.

       

       

      <Oct 2, 2013 10:39:27 AM MST> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '47' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "700" seconds working on the request "Workmanager: OIMMDBWorkManager, Version: 0, Scheduled=false, Started=true, Started time: 700360 ms

      ", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:

      Thread-294 "[STUCK] ExecuteThread: '47' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, suspended, blocked, priority=1, DAEMON> {

          -- Blocked trying to get lock: java.io.PrintStream@41a53700[not locked]

          java.io.PrintStream.println(PrintStream.java:755)

          com.wyndham.operations.user.eventHandlers.PostCreateOperationsEventHandler.execute(PostCreateOperationsEventHandler.java:60)

          oracle.iam.platform.kernel.impl.OrchProcessData.runPostProcessEvents(OrchProcessData.java:1225)

          oracle.iam.platform.kernel.impl.OrchProcessData.runEvents(OrchProcessData.java:665)

          oracle.iam.platform.kernel.impl.OrchProcessData.executeEvents(OrchProcessData.java:268)

          oracle.iam.platform.kernel.impl.OrchestrationEngineImpl.resumeProcess(OrchestrationEngineImpl.java:810)

          oracle.iam.platform.kernel.impl.OrchestrationEngineImpl.notifyParentProcess(OrchestrationEngineImpl.java:971)

          oracle.iam.platform.kernel.impl.OrchProcessData.runEvents(OrchProcessData.java:665)

          oracle.iam.platform.kernel.impl.OrchProcessData.executeEvents(OrchProcessData.java:268)

          oracle.iam.platform.kernel.impl.OrchestrationEngineImpl.resumeProcess(OrchestrationEngineImpl.java:810)

          oracle.iam.platform.kernel.impl.OrchestrationEngineImpl.resumeChildProcess(OrchestrationEngineImpl.java:880)

          oracle.iam.platform.kernel.impl.OrchProcessData.handleAdditionalChanges(OrchProcessData.java:552)

          oracle.iam.platform.kernel.impl.OrchProcessData.runEvents(OrchProcessData.java:665)

          oracle.iam.platform.kernel.impl.OrchProcessData.executeEvents(OrchProcessData.java:268)

          oracle.iam.platform.kernel.impl.OrchestrationEngineImpl.resumeProcess(OrchestrationEngineImpl.java:815)

          oracle.iam.platform.kernel.impl.OrchestrationEngineImpl.resumeProcess(OrchestrationEngineImpl.java:843)

          oracle.iam.platform.kernel.impl.OrhestrationAsyncTask.execute(OrhestrationAsyncTask.java:97)

          oracle.iam.platform.async.impl.TaskExecutor.executeUnmanagedTask(TaskExecutor.java:99)

          oracle.iam.platform.async.impl.TaskExecutor.execute(TaskExecutor.java:52)

          oracle.iam.platform.async.messaging.MessageReceiver.onMessage(MessageReceiver.java:60)

          sun.reflect.GeneratedMethodAccessor1493.invoke(Unknown Source)

          sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

          java.lang.reflect.Method.invoke(Method.java:575)

          com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)

          com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)

          com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)

          com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:86)

          com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)

          com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)

          com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:102)

          com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)

          com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:151)

          $Proxy460.onMessage(Unknown Source)

          weblogic.ejb.container.internal.MDListener.execute(MDListener.java:518)

          weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:467)

          weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:327)

          weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4638)

          weblogic.jms.client.JMSSession.execute(JMSSession.java:4134)

          weblogic.jms.client.JMSSession.executeMessage(JMSSession.java:3820)

          weblogic.jms.client.JMSSession.access$000(JMSSession.java:115)

       

      We have performed workload manager tuning in weblogic console as below.

       

      The property ‘Maximum Threads Constraint’ of work managers OIMMDBWorkManager = 80 and OIMUIWorkManager = 20

       

      Please suggest the solution.

        • 1. Re: Performance Issues in weblogic managed server
          Rosario

          Hi,

           

          Seems problem with IO operation in the system. Take strace for the process and your system admin can confirm about why/where the delay.

           

          Regards

          Rosario

          • 2. Re: Performance Issues in weblogic managed server
            PHCharbonneau

            Hi,

             

            If you look more closely at the STUCK Thread stack trace, you will notice that the thread is attempting to acquire a LOCK on the following Object monitor: java.io.PrintStream@41a53700. The native thread state is also in a BLOCKED state.

             

            My primary recommendation is to generate a full JVM Thread Dump (next re-occurence) so we can identify the potential culprit Thread holding such lock. One more thing, please see if you have multiple snapshots of this thread 294?

             

            I also recommend to review the following source code:

             

            com.wyndham.operations.user.eventHandlers.PostCreateOperationsEventHandler.execute(PostCreateOperationsEventHandler.java:60)


            Sharing the same Stream object in this case would trigger thread lock contention given java.io.PrintStream has built-in synchronization ex:


               public void println(String x) {

              synchronized (this) {

                 print(x);

                 newLine();

              }

                }

             

             

            Regards,

            P-H

            http://javaeesupportpatterns.blogspot.com