What are some best practices for Coherence configuration on Amazon's EC2? Since multicast is disabled, WKA is the only option. When I tried WKA with elastic IP (EIP), cluster would not start and even Oracle's EC2AddressProvider indicates that it uses local addresses internally. Does this mean WKA works only with EC2 local IPs?
Any help on the best practices for cloud deployment would be appreciated- esp. the minimum set of security rules required. When I tried to start the cluster using the local addresses with two nodes, the second node would not join successfully until I opened all UDP ports instead of just 8088 as another was being used to do a handshake.
There is an incubator project in the coherence-common with the name EC2AddressProvider for Amazon cloud integration and more details can be found here: http://coherence.oracle.com/display/INC10/coherence-common
Last time I looked into Coherence in EC2, which was admittedly over a year ago, they didn't look like a good match. Coherence is sensitive to network quality and its generally recommended all servers run of the same switch. Such a requirement can't be fulfilled with EC2 where machines are virtual and could be running in different racks, rooms or even data centres.
If this is just for functional testing, or if EC2 has become more Coherence friendly recently, you might get away with it. But if things haven't changed I'd be careful before running any Production grid up in EC2.
Depending on your usecase Amazon's Elasticache may be a better fit: http://aws.amazon.com/elasticache/
Do you know of anyone who has used the EC2AddressProvider in production? I had seen the [EC2AddressProvider docs|http://coherence.oracle.com/display/INCUBATOR/The+Amazon+EC2+AddressProvider+(1.7.0)] and it seemed to intimate Elastic I.P. addresses are not really supported.
"EC2AddressProvider queries EC2 to determine what instances are up and running and finds all instances that have an "Elastic IP" assigned to it for a given Amazon EC2 client. The instances with an Elastic IP assigned to it will be selected as the WKA nodes in the Coherence cluster. Note: It is *not the "Elastic IP" address itself that becomes the WKA IP, instead the private IP-address* of those instances are used"
How should such address provider should really work in production? For my use case, it's not practical that every instance with an EIP is considered to be a candidate for WKA since I will have other instance types not part of the Coherence cluster.
Our use case for Coherence is limited so our application may tolerate Coherence on the cloud. Our application already runs under less than ideal network infrastructure all around the world and we have to use an extremely high setting for timeout-milliseconds.
Amazon Elasticache unfortunately is not an option as we are an ISV and the cloud use is currently too minor to invest in changing our software.
BTW- Hazelcast which is an open source clustered cache appears to be used predominantly on EC2. As a clustered cache- they would be prey to the same network and timing inconsistencies so I think clustered caches on EC2 can be workable.
I would say it depends on how reliable you need your caches to be and how tolerant your application is of possible loss of nodes and or data. Running Coherence on a less reliable network, especially spread over multiple switches, or worse over a WAN, opens you up to possible problems of nodes dropping out of the cluster due to network issues and timing issues. In the worse case you can split your cluster in half, which would guarantee data loss.
I'm not saying you cannot or should not run Coherence like this, but you need to be aware of what you are getting into.