This discussion is archived
2 Replies Latest reply: Oct 27, 2013 3:40 PM by Leo_TA RSS

Coherence Warning Causing our production managed servers down/non-functional

karthiksingh_dba Explorer
Currently Being Moderated

Dear Legends,

 

We are facing a number of "Coherence Warning" and we came to know that one issue was in Cluster the servers are starting in Multicast even it has been set to Unicast and after an SR with ORACLE they suggested to add WKA host to be added with the startup scripts. We added and closely monitoring the servers continuously through the weekend, as usual the servers went non-functional in every 5 - 6 days.

This is in our HOST1 where Admin and Managed Server1(soa-server1)

-Dtangosol.coherence.wka1=soams1.com

-Dtangosol.coherence.wka2=soams2.com
-Dtangosol.coherence.localhost=soams1.com

 

This is in our HOST2 where Managed Server2(soa-server2)

-Dtangosol.coherence.wka1=soams1.com

-Dtangosol.coherence.wka2=soams2.com
-Dtangosol.coherence.localhost=soams2.com

 

We were closely looking into the logs and it says

 

<Oct 26, 2013 2:51:50 AM EDT> <Warning> <Coherence> <BEA-000000> <2013-10-26 02:51:50.417/439530.132 Oracle Coherence GE 3.7.1.1 <Warning> (thread=PacketPublisher, member=1): Experienced a 13215 ms communication delay (probable remote GC) with Member(Id=2, Timestamp=2013-10-21 00:56:32.745, Address=10.2.0.35:8088, MachineId=14118, Location=site:,machine:soams2,process:15664, Role=WeblogicServer); 82 packets rescheduled, PauseRate=0.0, Threshold=1976>

<Oct 26, 2013 2:52:35 AM EDT> <Warning> <Coherence> <BEA-000000> <2013-10-26 02:52:35.298/439575.013 Oracle Coherence GE 3.7.1.1 <Warning> (thread=PacketPublisher, member=1): Experienced a 4094 ms communication delay (probable remote GC) with Member(Id=2, Timestamp=2013-10-21 00:56:32.745, Address=10.2.0.35:8088, MachineId=14118, Location=site:,machine:soams2,process:15664, Role=WeblogicServer); 37 packets rescheduled, PauseRate=0.0, Threshold=1878>

<Oct 26, 2013 2:54:04 AM EDT> <Warning> <Coherence> <BEA-000000> <2013-10-26 02:54:04.144/439663.859 Oracle Coherence GE 3.7.1.1 <Warning> (thread=PacketPublisher, member=1): Experienced a 5936 ms communication delay (probable remote GC) with Member(Id=2, Timestamp=2013-10-21 00:56:32.745, Address=10.2.0.35:8088, MachineId=14118, Location=site:,machine:soams2,process:15664, Role=WeblogicServer); 46 packets rescheduled, PauseRate=0.0, Threshold=1696>

<Oct 26, 2013 2:55:04 AM EDT> <Warning> <Coherence> <BEA-000000> <2013-10-26 02:55:04.396/439724.111 Oracle Coherence GE 3.7.1.1 <Warning> (thread=PacketPublisher, member=1): Experienced a 19188 ms communication delay (probable remote GC) with Member(Id=2, Timestamp=2013-10-21 00:56:32.745, Address=10.2.0.35:8088, MachineId=14118, Location=site:,machine:soams2,process:15664, Role=WeblogicServer); 112 packets rescheduled, PauseRate=0.0, Threshold=1612>

<Oct 26, 2013 2:56:55 AM EDT> <Warning> <Coherence> <BEA-000000> <2013-10-26 02:56:55.540/439835.255 Oracle Coherence GE 3.7.1.1 <Warning> (thread=PacketPublisher, member=1): Experienced a 32323 ms communication delay (probable remote GC) with Member(Id=2, Timestamp=2013-10-21 00:56:32.745, Address=10.2.0.35:8088, MachineId=14118, Location=site:,machine:soams2,process:15664, Role=WeblogicServer); 178 packets rescheduled, PauseRate=1.0E-4, Threshold=1532>

<Oct 26, 2013 2:58:09 AM EDT> <Warning> <Coherence> <BEA-000000> <2013-10-26 02:58:09.435/439909.150 Oracle Coherence GE 3.7.1.1 <Warning> (thread=PacketPublisher, member=1): Experienced a 33213 ms communication delay (probable remote GC) with Member(Id=2, Timestamp=2013-10-21 00:56:32.745, Address=10.2.0.35:8088, MachineId=14118, Location=site:,machine:soams2,process:15664, Role=WeblogicServer); 182 packets rescheduled, PauseRate=2.0E-4, Threshold=1456>

 

What would be the reason even after adding the wka?

 

1. But still in logs while starting up am able to see the "-Dtangosol.coherence.clusteraddress=227.7.7.9 -Dtangosol.coherence.clusterport=9778". Is this the issue?

2. Or Else I need to add the "-Dtangosol.coherence.localport.adjust=true -Dtangosol.coherence.localport=8089 -Dtangosol.coherence.wka1.port=8089 -Dtangosol.coherence.wka2.port=8089" ?

 

Any kind of help would be much appreciated. Thanks in advance.

Regards,

Karthik

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points