Configuring Remote Application to Connect to Coherence Cache through HAProxy

3245865

Good morning,

I'm pretty new to Coherence and am looking for some guidance.

I have a functioning Coherence cluster deployed to Kubernetes (bare metal) as a POC. We also have a single HA Proxy instance (outside of Kubernetes) that is configured to talk TCP to each Coherence Pod that is part of the cluster. I am having issues using this instance to proxy 'get' and 'put' requests from a remote application using Coherence Extend. Perhaps I am misunderstanding what Coherence Extend is capable of.

Here is my cluster configuration:

<?xml version="1.0"?>

<cache-config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

   xmlns="http://xmlns.oracle.com/coherence/coherence-cache-config"

   xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-cache-config

   coherence-cache-config.xsd">

   <caching-scheme-mapping>

      <cache-mapping>

         <cache-name>hello-example</cache-name>

         <scheme-name>extend</scheme-name>

      </cache-mapping>

   </caching-scheme-mapping>

   <caching-schemes>

      <distributed-scheme>

         <scheme-name>extend</scheme-name>

         <lease-granularity>member</lease-granularity>

         <backing-map-scheme>

            <local-scheme/>

         </backing-map-scheme>

         <autostart>true</autostart>

      </distributed-scheme>

      <proxy-scheme>

         <service-name>ExtendTcpProxyService</service-name>

         <thread-count>5</thread-count>

         <acceptor-config>

            <tcp-acceptor>

               <local-address>

                  <address>localhost</address>

                  <port>7574</port>

               </local-address>

            </tcp-acceptor>

         </acceptor-config>

         <autostart>true</autostart>

      </proxy-scheme>

   </caching-schemes>

</cache-config>

Here is my client configuration, deployed with my remote application and referenced using -Dtangosol.coherence.cacheconfig=classpath:mycache.xml

<?xml version='1.0'?>

<cache-config xmlns:processing="class://com.oracle.coherence.patterns.processing.config.xml.ProcessingPatternNamespaceHandler"

                          xmlns:element="class://com.oracle.coherence.common.namespace.preprocessing.XmlPreprocessingNamespaceHandler"

                          element:introduce-cache-config="coherence-cache-config.xml">

        <caching-scheme-mapping>

                <cache-mapping>

                        <cache-name>hello-example</cache-name>

                        <scheme-name>remote</scheme-name>

                </cache-mapping>

        </caching-scheme-mapping>

        <caching-scheme>

                <remote-cache-scheme>

                        <scheme-name>remote</scheme-name>

                        <service-name>ExtendTcpCacheService</service-name>

                        <initiator-config>

                                <tcp-initiator>

                                        <remote-addresses>

                                                <socket-address>

                                                        <address>[haproxy ip]</address>

                                                        <port>[haproxy port]</port>

                                                </socket-address>

                                        </remote-addresses>

                                        <connect-timeout>10s</connect-timeout>

                                </tcp-initiator>

                                <outgoing-message-handler>

                                        <request-timeout>5s</request-timeout>

                                </outgoing-message-handler>

                        </initiator-config>

                </remote-cache-scheme>

                <remote-invocation-scheme>

                        <scheme-name>extend-invocation</scheme-name>

                        <service-name>ExtendTcpInvocationService</service-name>

                        <initiator-config>

                                <tcp-initiator>

                                        <remote-addresses>

                                                <socket-address>

                                                        <address>[ha-proxy ip]</address>

                                                        <port>[ha-proxy port]</port>

                                                </socket-address>

                                        </remote-addresses>

                                        <connect-timeout>10s</connect-timeout>

                                </tcp-initiator>

                                <outgoing-message-handler>

                                        <request-timeout>5s</request-timeout>

                                </outgoing-message-handler>

                        </initiator-config>

                </remote-invocation-scheme>

        </caching-scheme>

</cache-config>

The remote Java application is coded to perform 'get' and 'put' operations. In short:

public void put(...) {

  CacheFactory.getCache("hello-example").put(key, value);

}

public String get(...) {

   return (String) CacheFactory.getCache("hello-example").get(key);

}

Any advice would be greatly appreciated - thank you!

Tmiddlet-Oracle

Hi

You definitely shouldn't be using invocation service for get/put. The invocation service is for running code on individual members.

As you are using coherence 12.1.3, Take a look at the following documentation on setting up extend.

https://docs.oracle.com/middleware/1213/coherence/develop-remote-clients/gs_configextend.htm#COHCG4877

Your server side cache config proxy-scheme acceptor is running on 7574 which is the default cluster address for later coherence version, so probably best to change it to another value and expose that on your docker image.

You are using HA proxy, so you must also ensure you set client based load balancing (https://docs.oracle.com/middleware/1213/coherence/develop-remote-clients/gs_configextend.htm#COHCG5168  ) whic means the clietn will do the load balancing and not the service.

Also see JK's blog which talks about Running coherence in docker.

https://thegridman.com/coherence/oracle-coherence-on-docker/#extend

Tim

3245865

Upon further inspection of the documentation, I changed the 'get' method to use an InvocationService defined in the mycache.xml file.

InvocationService invocationService = (InvocationService) CacheFactory.getConfigurableCacheFactory().ensureService("ExtendTcpInvocationService");

Map map = invocationService.query(new AbstractInvocable()

{

   public void run()

  {

  setResult(CacheFactory.getCache("dist-extend").get(key));

  }

}, null);

return (String) map.get(invocationService.getCluster().getLocalMember());

However, now I am receiving the following error when I try and issue a request.

java.lang.IllegalArgumentException: No scheme found for service ExtendTcpInvocationService

        at com.tangosol.net.ExtensibleConfigurableCacheFactory.ensureService(ExtensibleConfigurableCacheFactory.java:302) ~[coherence-12.1.3-0-0.jar:12.1.3.0.0]

        ...

Tmiddlet-Oracle

Hi

You definitely shouldn't be using invocation service for get/put. The invocation service is for running code on individual members.

As you are using coherence 12.1.3, Take a look at the following documentation on setting up extend.

https://docs.oracle.com/middleware/1213/coherence/develop-remote-clients/gs_configextend.htm#COHCG4877

Your server side cache config proxy-scheme acceptor is running on 7574 which is the default cluster address for later coherence version, so probably best to change it to another value and expose that on your docker image.

You are using HA proxy, so you must also ensure you set client based load balancing (https://docs.oracle.com/middleware/1213/coherence/develop-remote-clients/gs_configextend.htm#COHCG5168  ) whic means the clietn will do the load balancing and not the service.

Also see JK's blog which talks about Running coherence in docker.

https://thegridman.com/coherence/oracle-coherence-on-docker/#extend

Tim

3245865

Tim, thank you for your assistance!

I was able to get it working using the documents that you provided. I ended up changing the port I was using, ditching the InvocationService in my code, and removing the remote-invocation-scheme in my client configuration.