This discussion is archived
3 Replies Latest reply: Sep 25, 2013 6:40 PM by user3490125 RSS

Cluster replication issue in 12c

926005 Newbie
Currently Being Moderated
Hi all,

After the installation of Weblogic Server 12c we have got cluster replication problems in our application test environment.
Although the same configuration used to work with Weblogic Server 10.3 now the secondary server seems to be removed and added to the cluster randomly.
-----
+<[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1332511394471> <BEA-000113> <Removing WL12c_ManagedServer_2 jvmid:4384922070744243395S:172.22.250.219:[10002,10002,-1,-1,-1,-1,-1]:project_domain:WL12c_ManagedServer_2 from cluster view due to PeerGone.>+

+<WL12c_ManagedServer_1> <weblogic.cluster.MessageReceiver> <<WLS Kernel>> <> <> <1332511403456> <BEA-003107> <Lost 2 unicast message(s).>+

+<WL12c_ManagedServer_1> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1332511403457> <BEA-002936> <maximum thread constraint weblogic.unicast.ForwardingWorkManager is reached>+

+<WL12c_ManagedServer_1> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1332511403460> <BEA-000111> <Adding WL12c_ManagedServer_2 with ID 4384922070744243395S:172.22.250.219:[10002,10002,-1,-1,-1,-1,-1]:project_domain:WL12c_ManagedServer_2 to cluster: ClusterWL12c view.>+

+<WL12c_ManagedServer_1> <[STANDBY] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1332511403462> <BEA-000128> <Updating 4384922070744243395S:172.22.250.219:[10002,10002,-1,-1,-1,-1,-1]:project_domain:WL12c_ManagedServer_2 in the cluster.>+

+<WL12c_ManagedServer_1> <ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <> <1332511413393> <BEA-002936> <maximum thread constraint weblogic.unicast.DispatchWorkManager is reached>+
-----

Further I see in the HttpClusterServlet log file that the secondary server hash is occasionally set to *!NONE!*

Environment Details (all servers are running on one physical server):
-     1 Node Server
-     1 Cluster (unicast) with 2 managed servers
-     1 Standalone managed server used as proxy (load balancer) based on HttpClusterServlet.
-     Linux redhat 2.6.18-128
-     Applications have been compiled using the weblogic server JDK

Part of config.xml:

+<server>+
+<name>AdminServer</name>+
+<listen-port>17001</listen-port>+
+<listen-address></listen-address>+
+</server>+
+<server>+
+<name>WL12c_ManagedServer_1</name>+
+<ssl>+
+<enabled>false</enabled>+
+</ssl>+
+<machine>WL12c_Machine</machine>+
+<listen-port>10001</listen-port>+
+<cluster>ClusterWL12c</cluster>+
+<listen-address></listen-address>+
+<server-start>+
+<arguments>-Djava.awt.headless=true</arguments>+
+</server-start>+
+<jta-migratable-target>+
+<name>WL12c_ManagedServer_1</name>+
+<user-preferred-server>WL12c_ManagedServer_1</user-preferred-server>+
+<cluster>ClusterWL12c</cluster>+
+</jta-migratable-target>+
+<weblogic-plugin-enabled>false</weblogic-plugin-enabled>+
+</server>+
+<server>+
+<name>WL12c_ManagedServer_2</name>+
+<ssl>+
+<enabled>false</enabled>+
+</ssl>+
+<machine>WL12c_Machine</machine>+
+<listen-port>10002</listen-port>+
+<cluster>ClusterWL12c</cluster>+
+<listen-address></listen-address>+
+<server-start>+
+<arguments>-Djava.awt.headless=true</arguments>+
+</server-start>+
+<jta-migratable-target>+
+<name>WL12c_ManagedServer_2</name>+
+<user-preferred-server>WL12c_ManagedServer_2</user-preferred-server>+
+<cluster>ClusterWL12c</cluster>+
+</jta-migratable-target>+
+<weblogic-plugin-enabled>false</weblogic-plugin-enabled>+
+</server>+
+<server>+
+<name>WL12c_ProxyServer</name>+
+<ssl>+
+<enabled>false</enabled>+
+</ssl>+
+<machine>WL12c_Machine</machine>+
+<listen-port>9001</listen-port>+
+<cluster xsi:nil="true"></cluster>+
+<web-server>+
+<web-server-log>+
+<number-of-files-limited>false</number-of-files-limited>+
+</web-server-log>+
+</web-server>+
+<listen-address></listen-address>+
+</server>+
+<cluster>+
+<name>ClusterWL12c</name>+
+<cluster-messaging-mode>unicast</cluster-messaging-mode>+
+<weblogic-plugin-enabled>false</weblogic-plugin-enabled>+
+<frontend-host>build-env.rtd</frontend-host>+
+<frontend-http-port>9001</frontend-http-port>+
+<number-of-servers-in-cluster-address>2</number-of-servers-in-cluster-address>+
+</cluster>+

Does anyone have an idea how to solve this issue?
  • 1. Re: Cluster replication issue in 12c
    ArunBodap Pro
    Currently Being Moderated
    This looks like a bug in 12c.
    Please contact support for further investigation

    Arun
  • 2. Re: Cluster replication issue in 12c
    user3490125 Newbie
    Currently Being Moderated

    Seeing the same issue with WebLogic 12c on AIX 7.1, with IBM JDK SR3. 

     

    The same basic config works correctly on Linux and Windows.

     

    Same as OP, we are using unicast cluster messaging with two managed servers on the same OS instance.   We use Apache for the front-end proxy.  However, the nodes are dropped and added from the cluster whenever they attempt to send a cluster message of any kind.   Although they are quickly re-added, application errors result when performing any operation on a node while it is excluded from the cluster.

  • 3. Re: Cluster replication issue in 12c
    user3490125 Newbie
    Currently Being Moderated

    Updated cluster to use Multicast, but no difference in behavior.   I can now run MulticastMonitor and see that the heartbeat messages are sent OK.   And, the monitor sees the session update messages and thinks they are ok.  However, any activity in the application causes the non-active peer to be removed with a "due to Peer Gone" message, then re-added, then updated.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points