We are using bdb in a 3 way replicated cluster system ( 1 master, 2 replicas / 40+ nodes) we are handling a lot of read traffic and are seeing a very long startup time for nodes in our clusters.
The reason that seems to happen is because on a restart a replica node almost always falls into our TimeConsistencyPolicy and thus leaves the constructor early, but we are still significantly out of date compared to our normal running state.
This means however that the master node has a few gigs of data that need to be synced, and we are already getting in read traffic at that point which becomes incredibly painful.
I am wondering now whether there is a way to set a startup consistency that only applies during the Constructor phase of the replica and during normal operation does fall back on the TimeConsistencyPolicy we configured initially ?
While digging through the code I actually stumbled over the ReplicatedEnvironment constructor that seems to accept a ConsistencyPolicy argument that is only used when the replica joins a group. Is that the intended way to handle a different startup consistency policy generally ??