4 Replies Latest reply on Aug 23, 2011 6:05 PM by Rajiv Dewan

    WEBLOGIC SERVER STATE IS UNKNOWN

    864512
      Hi,

      We have issue in Production environment.

      we use Weblogic version : 10.3 with Java version:1.6.0_24

      Server state changed from RUNNING to UNKNOWN and we were not able to access the application. We did recycling of the managed servers and in hour time again the state changed from RUNNING to UNKNOWN.

      when we did complete recycle including admin this error didn reapear. ofcource i need to watch for some more time.

      could you please tell me if anyone faced this issue and resolved it.

      and the below error was thrown for both the times.

      ####<Aug 18, 2011 10:22:18 AM EDT> <Info> <RJVM> <servername> <admin> <[ACTIVE] ExecuteThread: '44' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1313677338123> <BEA-000513> <Failure in heartbeat trigger for RJVM: -2068408514382952870S:server name:[-1,-1,18212,18212,-1,-1,18212]:server name:18210,server name:18210:domain:domain002
      java.io.IOException: The connection manager to ConnectionManager for: 'weblogic.rjvm.RJVMImpl@110b5f3 - id: '-2068408514382952870S:server name:[-1,-1,18212,18212,-1,-1,18212]:server name:18210,server name:18210:domain:domain002' connect time: 'Sat Jul 16 05:17:38 EDT 2011'' has already been shut down.
      java.io.IOException: The connection manager to ConnectionManager for: 'weblogic.rjvm.RJVMImpl@110b5f3 - id: '-2068408514382952870S:server name:[-1,-1,18212,18212,-1,-1,18212]:server name:18210,server name:18210:domain:domain002' connect time: 'Sat Jul 16 05:17:38 EDT 2011'' has already been shut down
           at weblogic.rjvm.ConnectionManager.getOutputStream(ConnectionManager.java:1719)
           at weblogic.rjvm.ConnectionManager.createHeartbeatMsg(ConnectionManager.java:1662)
           at weblogic.rjvm.ConnectionManager.sendHeartbeatMsg(ConnectionManager.java:599)
           at weblogic.rjvm.RJVMImpl$HeartbeatChecker.timerExpired(RJVMImpl.java:1584)
           at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)
           at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
           at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
           at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
      >


      thanks in advance.
      Ram
        • 1. Re: WEBLOGIC SERVER STATE IS UNKNOWN
          PHCharbonneau
          Hi Ram,

          As a starting point, please look at the actual logs of your managed server and let me know what you can find. The log is showing that the Admin lost contact with its managed server >> :server name:18210 but you need to look at the server log of the failed managed server.

          There are many scenarios which can lead to an UNKNOWN status. It normally means that your managed server is unhealthy and stopped to accept incodming requests (JVM crash, OutOfMemoryError, too man Stuck Threads, too many sockets / file descriptors vs. your current limit etc.).

          Regards,
          P-H
          http://javaeesupportpatterns.blogspot.com/
          • 2. Re: WEBLOGIC SERVER STATE IS UNKNOWN
            864512
            Hi P-H,

            thanks for the quick update.

            Today again i had the same error in admin server logs and the managed server state become UNKNOWN.

            As per the PROD Admin server logs I found that Failure in heartbeat trigger for RJVM error .

            The same error was thrown yesterday as well.

            BEA weblogic says no action required for this problem. If it is too frequent then we may need to register a case with BEA.

            From BEA Weblogic url (http://download.oracle.com/docs/cd/E12839_01/apirefs.1111/e14397/RJVM.html), following info was given for error <BEA-000513> which we are facing today.
                 Info: Failure in heartbeat trigger for RJVM: id\n t.
            Description      This is usually a temporary failure.
            Cause      This failure indicates a temporary failure in the heartebeat trigger mechanism.
            Action      No action is necessary. This is usually a temporary failure.



            i checked the managed server logs and the following error was thrown. Please help me if you have any clue or suggestions.


            2011-08-19 08:02:15
            Full thread dump Java HotSpot(TM) Server VM (19.1-b02 mixed mode):

            "[STANDBY] ExecuteThread: '17' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=3 tid=0x03894400 nid=0x146 runnable [0x00000000]
            java.lang.Thread.State: RUNNABLE

            "[STANDBY] ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=3 tid=0x03896000 nid=0x145 runnable [0x534ff000]
            java.lang.Thread.State: RUNNABLE
                 at weblogic.cluster.messaging.internal.GroupImpl.getConfigInformation(GroupImpl.java:279)
                 - locked <0x7bc709a8> (a weblogic.cluster.messaging.internal.GroupImpl)
                 at weblogic.cluster.messaging.internal.GroupManagerImpl.findOrCreateGroupMember(GroupManagerImpl.java:183)
                 at weblogic.cluster.messaging.internal.GroupManagerImpl.handleMessage(GroupManagerImpl.java:158)
                 at weblogic.cluster.messaging.internal.ConnectionImpl$1.run(ConnectionImpl.java:116)
                 at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
                 at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
                 at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

            "[ACTIVE] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=3 tid=0x00493c00 nid=0x144 runnable [0x52ffd000]
            java.lang.Thread.State: RUNNABLE
                 at java.util.HashMap.addEntry(HashMap.java:753)
                 at java.util.HashMap.put(HashMap.java:385)
                 at java.util.ListResourceBundle.loadLookup(ListResourceBundle.java:180)
                 - locked <0xed852420> (a oracle.toplink.internal.localization.i18n.TraceLocalizationResource)
                 at java.util.ListResourceBundle.handleGetObject(ListResourceBundle.java:106)
                 at java.util.ResourceBundle.getObject(ResourceBundle.java:368)
                 at java.util.ResourceBundle.getString(ResourceBundle.java:334)
                 at oracle.toplink.internal.localization.ToplinkLocalization.buildMessage(ToplinkLocalization.java:39)
                 at oracle.toplink.internal.localization.TraceLocalization.buildMessage(TraceLocalization.java:14)
                 at oracle.toplink.logging.AbstractSessionLog.formatMessage(AbstractSessionLog.java:834)
                 at oracle.toplink.sessions.DefaultSessionLog.log(DefaultSessionLog.java:152)
                 - locked <0x7dae2a80> (a oracle.toplink.logging.DefaultSessionLog)
                 at oracle.toplink.logging.DefaultSessionLog.log(DefaultSessionLog.java:67)
                 - locked <0x7dae2a80> (a oracle.toplink.logging.DefaultSessionLog)
                 at oracle.toplink.publicinterface.Session.log(Session.java:2284)
                 at oracle.toplink.publicinterface.Session.log(Session.java:3567)
                 at oracle.toplink.publicinterface.Session.log(Session.java:3542)
                 at oracle.toplink.publicinterface.Session.log(Session.java:3521)
                 at oracle.toplink.publicinterface.Session.log(Session.java:3430)
                 at oracle.toplink.jts.AbstractExternalTransactionController.registerSynchronizationListener(AbstractExternalTransactionController.java:121)
                 at oracle.toplink.publicinterface.UnitOfWork.registerWithTransactionIfRequired(UnitOfWork.java:3958)
                 at oracle.toplink.publicinterface.Session.acquireUnitOfWork(Session.java:260)
                 at com.test.common.BaseDBAccessor.createUnitOfWork(BaseDBAccessor.java:256)
                 at com.test.admin.dataaccess.AdminDBAccessor.isUserActive(AdminDBAccessor.java:120)
                 at com.test.admin.ejb.AdminSessionEJBBean.isUserActive(AdminSessionEJBBean.java:61)
                 at sun.reflect.GeneratedMethodAccessor567.invoke(Unknown Source)
                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                 at java.lang.reflect.Method.invoke(Method.java:597)
                 at com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:281)
                 at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:187)
                 at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:154)
                 at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:126)
                 at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:114)
                 at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:176)
                 at com.bea.core.repackaged.springframework.jee.spi.MethodInvocationVisitorImpl.visit(MethodInvocationVisitorImpl.java:15)
                 at weblogic.ejb.container.injection.EnvironmentInterceptorCallbackImpl.callback(EnvironmentInterceptorCallbackImpl.java:54)
                 at com.bea.core.repackaged.springframework.jee.spi.EnvironmentInterceptor.invoke(EnvironmentInterceptor.java:30)
                 at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:176)
                 at com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
                 at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:176)
                 at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:126)
                 at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:114)
                 at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:176)
                 at com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:210)
                 at $Proxy126.isUserActive(Unknown Source)
                 at com.test.admin.ejb.AdminSessionEJBBean_gfe5ds_AdminSessionEJBRemoteImpl.isUserActive(AdminSessionEJBBean_gfe5ds_AdminSessionEJBRemoteImpl.java:637)
                 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                 at java.lang.reflect.Method.invoke(Method.java:597)
                 at weblogic.ejb.container.internal.RemoteBusinessIntfProxy.invoke(RemoteBusinessIntfProxy.java:69)
                 at $Proxy74.isUserActive(Unknown Source)
                 at com.test.admin.facades.AdminFacadeDelegate.isUserActive(AdminFacadeDelegate.java:108)
                 at com.test.common.AuthorizationFilter.doFilter(AuthorizationFilter.java:130)
                 at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)
                 at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3496)
                 at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
                 at weblogic.security.service.SecurityManager.runAs(Unknown Source)
                 at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2180)
                 at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2086)
                 at weblogic.servlet.internal.ServletRequestImpl.run(Unknown Source)
                 at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
                 at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

            "[ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=3 tid=0x00493000 nid=0x143 runnable [0x51cfe000]
            java.lang.Thread.State: RUNNABLE
                 at com.rsa.jsafe.JCMPInt.a(Unknown Source)
                 at com.rsa.jsafe.JCMPInt.modExp(Unknown Source)
                 at com.rsa.jsafe.JCMPInt.modExpCRT(Unknown Source)
                 at com.rsa.jsafe.JA_RSA.b(Unknown Source)
                 at com.rsa.jsafe.JG_AsymmetricCipher.d(Unknown Source)
                 at com.rsa.jsafe.JSAFE_AsymmetricCipher.decryptFinal(Unknown Source)
                 at com.rsa.jsafe.JSAFE_AsymmetricCipher.decryptFinal(Unknown Source)
                 at com.certicom.tls.provider.cipher.JSAFE_RSA.doFinal(Unknown Source)
                 at com.certicom.tls.record.handshake.ServerStateSentHelloDone.handle(Unknown Source)
                 at com.certicom.tls.record.handshake.HandshakeHandler.handleHandshakeMessage(Unknown Source)
                 at com.certicom.tls.record.handshake.HandshakeHandler.handleHandshakeMessages(Unknown Source)
                 at com.certicom.tls.record.MessageInterpreter.interpretContent(Unknown Source)
                 at com.certicom.tls.record.MessageInterpreter.decryptMessage(Unknown Source)
                 at com.certicom.tls.record.ReadHandler.processRecord(Unknown Source)
                 at com.certicom.tls.record.ReadHandler.readRecord(Unknown Source)
                 at com.certicom.tls.record.ReadHandler.readUntilHandshakeComplete(Unknown Source)
                 at com.certicom.tls.interfaceimpl.TLSConnectionImpl.completeHandshake(Unknown Source)
                 - locked <0xed2f8d70> (a com.certicom.tls.interfaceimpl.TLSConnectionImpl)
                 at javax.net.ssl.impl.SSLSocketImpl.startHandshake(Unknown Source)
                 at weblogic.server.channels.DynamicSSLListenThread$1.run(DynamicSSLListenThread.java:130)
                 at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
                 at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
            • 3. Re: WEBLOGIC SERVER STATE IS UNKNOWN
              PHCharbonneau
              Hi Ram,
              That is correct, I'm not too concerned on this Admin server RJVM since this is normal error following disconnect / lost of the managed server.

              The primary focus should be on the root cause which is managed server instability.
              ..................................
              2011-08-19 08:02:15
              Full thread dump Java HotSpot(TM) Server VM (19.1-b02 mixed mode):
              ............................................

              Actually what you saw is not an error but a Weblogic generated JVM Thread Dump following problem detection. JVM Thread Dump provides you with a detailed snapshot of all running Threads of your Java VM and is extremely useful to identify a large list of problems.

              The first 3 Threads of this Thread Dump snapshot is showing that your JVM is facing some potential global contention since CPU / IO related operations are really slowing down from this managed server.

              - Thread # 16 is showing some Weblogic internal Thread race / wait for its intra Cluster communication
              - Thread #13 indicates logging / IO contention within the Oracle Toplink API and originates from your application code >> at com.test.common.BaseDBAccessor.createUnitOfWork(BaseDBAccessor.java:256)
              - Thread #12 indicates slowdown / pause time of your SSL handshake process from incoming SSL requests such as HTTPS

              One of the most common source of global JVM / Threads contention is excessive garbage collection. Global contention can also trigger by too much load (concurrent requests) on this single managed server, excessive logging etc.

              if you can, please analyze the complete Thread Dump (all the Java Threads) and determine if you are facing any culprit Thread triggering this contention vs. no pattern. Otherwise, please post the complete Thread Dump and I will analyze it for you.

              No clear Thread culprit normally indicates a global contention problem patterns such as excessive GC causing too frequent JVM pause time along with high CPU as well.

              Find below my recommendations based on facts we know so far:

              - Investigate immediately your garbage collection process via verbose GC or any other tool of your choice such as JConsole etc. determine your Java Heap utilization and frequency / ratio of young vs. old generations. Also, see if any OutOfMemoryError was generated in your managed server log ~ 5 minutes becoming unresponsive
              - Analyze your complete Thread Dump in order to identify a possible culprit. You can post it here in this Thread and I will analyze it
              - If using Solaris OS, please generate a PRSTAT snapshot (prstat -L <java pid>) in order to generate a detailed breakdown of the internal JVM Threads and its CPU contributions, really useful to pinpoint culprit Thread(s) causing heavy contention
              - Have a look at the CPU utilization of your managed server Java process during problem replication
              - Review your current Toplink logging and ensure your application / Toplink is not generating excessive logging on your system

              Regards,
              P-H
              • 4. Re: WEBLOGIC SERVER STATE IS UNKNOWN
                Rajiv Dewan
                http://download.oracle.com/docs/cd/E14571_01/doc.1111/e14309/plugins.htm#CJACGBJB

                Set ANT_HOME
                Set WLS_HOME

                Build the wlfullclient.jar in Oracle WebLogic server:

                Change directories to WLS_HOME/server/lib.

                Run the following command:

                java -jar ../../../modules/com.bea.core.jarbuilder_1.5.0.0.jar

                Note:
                The exact jar file version can be different based on the WLS. Use the corresponding file with the name as com.bea.core.jarbuilder.