I am running tests using the StockQuotes example with a cluster of four nodes. I have a question about the behaviour of the cluster as replicas are killed.
I start four nodes, n1, n2, n3, n4. n1 is elected Master. n2 - n4 become replicas.
If I kill n3, n1 remains master and writes are still permitted as there are sufficient replicas to satisfy the SIMPLE_MAJORITY policy. If I then go on to kill n4, n1 still remains master but InsufficientReplicasException is thrown in response to each write. I see in the documentation this is expected behaviour.
In our use case, we would like to know immediately when the number of replicas becomes too few to satisfy the ReplicaAckPolicy, rather than waiting for the next transaction to fail with InsufficientReplicasException. We wonder if there is a way to organise for the Master to transition back to Unknown when this condition arises. If this is not possible, is it possible for the application to learn of this in the other way? I have read about Monitor.startListener() and MonitorChangeListener. Could these help us?
Yes, the various notification mechanisms that inform the user of membership status changes are the right way to go. You've already noticed the Monitor and MonitorChangeListener classes, and I hope the javadoc (start with com.sleepycat.je.rep.monitor.Monitor) and the chapter in the replication guide are informative enough: http://docs.oracle.com/cd/E17277_02/html/ReplicationGuide/monitors.html. Also note the chapter on using com.sleepycat.je.rep.StateChangeListener: http://docs.oracle.com/cd/E17277_02/html/ReplicationGuide/replicawrites.html.
The two mechanisms differ from the perspective of whether the handling is done at the point when the application has access to a ReplicatedEnvironment, or whether it's done by a third party entity, like a load balancer. They're two sides of the same coin. But neither one can totally exempt the application code from needing to handle the possibility of an InsufficientReplicasException.
- It's always possible that the notifications will be delivered and handled later than the exception
- the lack of a response from a replica might be due to a temporary network glitch, and since there hasn't been a true state change, no notification is sent
- if a node fails, it can't send its own notification, of course, and it can take a little while for other nodes to step in, detect the state change, and manufacture a notification
So the notification mechanisms can optimize and reduce the possibility of getting an InsufficientReplicasException, but it can't completely remove it. This does lead to the need to handle a variety of exceptions in the application code. A long time ago, Jeff Alexander wrote up this example of how he made a transaction wrapper to modularize all the exception handling: https://blogs.oracle.com/jhalex/entry/handling_transactions_in_bdb_je. We tend to do something along those lines when we write code that is issuing data operations against a BDB JE HA cluster.