Forum Stats

  • 3,780,849 Users
  • 2,254,447 Discussions
  • 7,879,484 Comments

Discussions

Federation across networks with private and public DNSes

Chris San Buenaventura
Chris San Buenaventura Member Posts: 17 Green Ribbon
edited Dec 11, 2019 10:59AM in Coherence Support

In our current ActiveActive topology we have two clusters (ClusterA and ClusterB) each hosted in different data centres (DataCentreA and DataCentreB). Cluster communication is via unicast (wka list) running on a reserved port, not the default 7574 port. There's a firewall between the two data centres but the reserved port is open between the two clusters.

Point A: In data centre A, the ClusterA members bind addresses to the internal DNS which isn't visible to data centre B.

Point B: In data centre B, the ClusterB members bind addresses to a global DNS which is also resolvable from data centre A.

TCP communication between ClusterA and ClusterB members works if the global DNS names are used.

Due to the nature of this setup, federation from ClusterA to ClusterB works but not the other way around.

Looking at the logs of a member from ClusterB, when a member hits the NameService from ClusterA, it's given an IPAddress/Port, whereby the IP Address is bound to the internal DNS. ClusterB then tries to connect to ClusterA with this IP Address, which fails as explained in Point A.

I gave "Using a Specific Network Interface for Federation Communication" a stab to no avail. It looks like the instructions here are outdated?

https://docs.oracle.com/en/middleware/fusion-middleware/coherence/12.2.1.4/administer/federating-caches-clusters.html#GUID-2D356F18-34A4-4F4F-A597-C1E1C8A4DB82

Tagged:
Chris San Buenaventura

Best Answer

  • P Fry-Oracle
    P Fry-Oracle Member Posts: 79
    edited Dec 10, 2019 4:43PM Accepted Answer

    Ok, so if I'm following correctly, Cluster A members cannot bind directly to the global IP addresses (e.g. 10.0.0.1) as those are NAT addresses?  In that case would it be possible to use another port in each Cluster A member for federation only?  It is possible to specify a specific IP address and port number to bind to in the federated-scheme (or address provider with the IP address and port number defined in the Coherence operational config).  And then in Cluster B, specify the global IP and the dedicated federation port.  In that case when cluster B members attempt to connect to Cluster A, they will first attempt a NameService connection to the dedicated federation port, which will fail and then they will attempt a direct connection to the federation port, which should succeed.

    This federation direct connection functionality was added just recently in Coherence - versions 12.2.1.3.y (where y is 3 or higher) and 12.2.1.4.0 and later.

    To see an example of how this works, take a look at the cache and operational configuration (and use of system properties) in https://github.com/coherence-community/coherence-demo#enable-federation-on-kubernetes

    Hope this helps,

    Patrick

    Chris San Buenaventura

Answers

  • P Fry-Oracle
    P Fry-Oracle Member Posts: 79
    edited Dec 10, 2019 10:14AM

    Hello Chris,

    The documentation you referenced is accurate.  I'm a little bit confused about what you're trying to do...  If ClusterA members are binding to internal addresses won't they be unreachable from ClusterB?

    Patrick

  • Chris San Buenaventura
    Chris San Buenaventura Member Posts: 17 Green Ribbon
    edited Dec 10, 2019 4:21PM

    Please take note of the interal and global IP addresses. That is the key to my issue.

    Say ClusterA memebrs have following IP addresses:

    Storage Node:

    internal IP: 192.168.0.1

    global IP: 10.0.0.1

    Proxy:

    internal IP: 192.168.0.2

    global IP: 10.0.0.2

    And Cluster B members having IP addresses:

    Storage Node:

    global IP: 10.0.0.3

    Proxy:

    global IP: 10.0.0.4

    Due to the nature of network design, internal IP addresses from ClusterA are NOT accessible from ClusterB, however global IP addresses can be accessed from ClusterB. Further, members of ClusterA can only communicate amongs each other using the internal IP as the global IP is NATt'ed further up the network chain.

    So when a member of ClusterB accesses the NameService from ClusterA (via clusterport or the Specific Network Interface), it's given the internal IP addresses of members from ClusterA. Then it attemps to connect to a member in ClusterA, and fails to do so because the IP address it has received from the NameService is an internal IP.

    See below for a snippet of the ClusterB member trying to connect to ClusterA member

    2019-12-10T21:09:24,637 INFO  [[email protected] 12.2.1.4.0][Coherence] (thread=DistributedCache:DestinationController[ClusterA]Worker:0, member=2) Connecting to service FederatedCache at participant ClusterA with address tmb://192.168.0.1:9000.456

    2019-12-10T21:09:31,655 INFO  [[email protected] 12.2.1.4.0][Coherence] (thread=SelectionService(channels=24, selector=MultiplexedSelector([email protected]), id=1813665703), member=2) Disconnected from destination ClusterA

  • P Fry-Oracle
    P Fry-Oracle Member Posts: 79
    edited Dec 10, 2019 4:43PM Accepted Answer

    Ok, so if I'm following correctly, Cluster A members cannot bind directly to the global IP addresses (e.g. 10.0.0.1) as those are NAT addresses?  In that case would it be possible to use another port in each Cluster A member for federation only?  It is possible to specify a specific IP address and port number to bind to in the federated-scheme (or address provider with the IP address and port number defined in the Coherence operational config).  And then in Cluster B, specify the global IP and the dedicated federation port.  In that case when cluster B members attempt to connect to Cluster A, they will first attempt a NameService connection to the dedicated federation port, which will fail and then they will attempt a direct connection to the federation port, which should succeed.

    This federation direct connection functionality was added just recently in Coherence - versions 12.2.1.3.y (where y is 3 or higher) and 12.2.1.4.0 and later.

    To see an example of how this works, take a look at the cache and operational configuration (and use of system properties) in https://github.com/coherence-community/coherence-demo#enable-federation-on-kubernetes

    Hope this helps,

    Patrick

    Chris San Buenaventura
  • Chris San Buenaventura
    Chris San Buenaventura Member Posts: 17 Green Ribbon
    edited Dec 10, 2019 6:34PM

    Yes and yes to both questions. I'll give this kubernetes example a crack.

    Meanwhile, what we have done is map the internal IP addersses of ClusterA members in the ClusterB box via iptables, works seamlessly.

  • Chris San Buenaventura
    Chris San Buenaventura Member Posts: 17 Green Ribbon
    edited Dec 10, 2019 6:45PM

    Hi Patrick,

    Can you please give example operational and override config xmls for this:

    "It is possible to specify a specific IP address and port number to bind to in the federated-scheme (or address provider with the IP address and port number defined in the Coherence operational config).  And then in Cluster B, specify the global IP and the dedicated federation port."


    Are you referring to this section of the operational config xml? From https://github.com/coherence-community/coherence-demo/blob/master/src/main/resources/cache-config.xml

    pastedImage_2.png

    Alternatively, is it possible to write a custom NameService that would return the equivalent global IP of a member? Is this advised?

    Thanks,

    Chris

  • Chris San Buenaventura
    Chris San Buenaventura Member Posts: 17 Green Ribbon
    edited Dec 11, 2019 12:26AM

    Hey Patrick,

    I sort of combined the instructions from https://docs.oracle.com/en/middleware/fusion-middleware/coherence/12.2.1.4/administer/federating-caches-clusters.html#GUID-2D356F18-34A4-4F4F-A597-C1E1C8A4DB82 and https://github.com/coherence-community/coherence-demo/blob/master/src/main/resources/cache-config.xml .

    Setup an address provider in the override xml and referenced this address provider in the operational config file under the federated scheme. The remote cluster can then hit this address provider via TCP (not NameService). So now working seamlessly.

    override xml

    <cluster-config>

    ...

    <address-providers>
      <address-provider id="FederationAddressProvider">
      <socket-address>
      <address system-property="coherence.localhost"/>
      <port system-property="coherence.local.cluster.federation.port"/>
      </socket-address>
      </address-provider>
      </address-providers>
    </cluster-config>

    operation xml

    <caching-schemes>
      <federated-scheme>
      <scheme-name>transient</scheme-name>
      <service-name system-property="coherence.cache.service.name"/>
      <backing-map-scheme>
      <transient>true</transient>
      <local-scheme/>
      </backing-map-scheme>
      <autostart>true</autostart>
      <address-provider>FederationAddressProvider</address-provider>
      <topologies>
      <topology>
      <name>ActiveActive</name>
      </topology>
      </topologies>
      </federated-scheme>

    ...

    </caching-scemes>

    Thanks for the help!

  • P Fry-Oracle
    P Fry-Oracle Member Posts: 79
    edited Dec 11, 2019 10:59AM

    Glad it worked out!

    Patrick