We are implementing a way to have global lock, so that only one JVM can modify the cache. No using Entry processors is not an option, as the updates will have to happen on a bunch of related caches.
I am using lease granularity of member on both the storage enabled node and client (which is storage disabled) - I could have gone with default, but one the requirement is that unlock can be made on a different thread.
We have put in place, a way to make sure, that more than one thread from the same JVM, doesn't get the lock on the same key - using constructs provided by java.util.concurrent
Here is the scheme used
The issue is, when Thread 1 from JVM1 calls unlock on the key, Thread 2 from JVM1 and Thread 1 from JVM2 are getting lock on the same key (They pretty much at the same mill second).
How do I avoid it, when lock from 2 different JVM's happen at the same time, for the same key.