This content has been marked as final. Show 8 replies
first, let me correct some of your naming convention mistakes.
What you described would most probably be a single grid (cluster) with one or more named caches.
As for what you try to do, if you want to partition your load to multiple JVMs, then you can use multiple approaches:
- a backing map listener for a distributed cache containing the event objects scheduling tasks to be executed in the same JVM (but in a different thread). The task execution in the different thread would have to check that the event key is still owned by the local cluster member and would have to be idempotent
- a partitioned cache with idempotent listeners which check that a task is completed or not (this is similar to what you described)
What are the persistence requirements of your events?
- Do they have to be persistent and do they have to survive the shutdown of the entire cluster?
- Do they have to be persistent but not survive the shutdown of the entire cluster?
- Do they not have to be persistent?
What are the volume and arrival rate of your events?
thanks for your reply!
I expect to receive different types of events, I think that all three categories you mentioned are needed.
Our platform starts with 10.000 registrations, 1.000 concurrent users each firing different all types of events, 1 event per 10 seconds. The number of registrations is expected to grow quickly in the near future. The number of concurrent users will grow relative to that, their event firing behaviour is expected to stay constant.
I'm not sure if I understand the first approach you described. It might have to do with the fact that i don't know about the concept of 'key owning' ;-)
I'm not sure if I understand the first approach youIn the partitioned caches for each key there is a single partition that key belongs to. For any partition, there is exactly one node which owns that partition (this is the node which stores the primary copy of the partition), and transitively owns the entries with the keys belonging to the same partition. There is also a configurable number of backup copies of each partition (provided that there are enough cluster members for hosting each copy in a separate member, as each copy must reside on a different JVM; the copies also reside on different hosts if it is possible to lay them out so).
described. It might have to do with the fact that i
don't know about the concept of 'key owning' ;-)
Backing maps are the maps in the JVM which store the data in the JVM. For distributed caches the backing map stores the primary copies, so only entries with owned keys are found in the backing map of a partitioned cache (backups are stored outside of this backing map). In case of replicated caches the backing map stores all entries.
A backing map listener gets events about manipulation of the entries in the backing map either carried out on behalf of a cache client or triggered by repartitioning or by (configurable) eviction of entries. This means that only events about the owned keys are received by the backing map listener.
However, a backing map listener:
1. Should complete as quickly as it can as it is invoked inline with the original request.
2. Must not invoke any manipulation operations on the same cache service.
Therefore to do any operations on the same cache service as part of handling of the events, you must do that from another thread.
However, if you do something from another thread, the ownership of the key might have changed in the period of time between scheduling the event for the different thread and actually invoking that thread. Therefore to be able to cope with duplicate handling of the events your code should check, that
1. The key of the event is still local, so that you do not try to handle a non-local event, it would be carried out by the new owner.
2. The status of the event is not handled (so that you are able to detect duplicate executions occuring as a result of node failover).
Practically your event handlers should be idempotent.
You should also look at the InvocableMap feature of Coherence (you can find more about it in the Javadoc and at the following page:
I expect to receive different types of events, IFor non-persisted events a simple cache is enough, hosted by a cache service for which the event handler nodes are the storage-enabled nodes. A storage enabled-node for a partitioned cache service is a node which holds primary and backup copies of data. Nodes can also be storage-disabled in a partitioned cache service. These nodes can access the data in the partitioned cache but they don't store copies of them (therefore coming and going of these nodes don't incur loss or rebalancing of the data).
think that all three categories you mentioned are
For events which need to be persisted, you can configure either a write-through or a write-behind cache store. You can read more about it in:
The biggest difference between the two is that with write-through the data is written to the database inline with the request (failure to write the data to DB if so configured results in a failure to putting the data into the cache and an exception in the client). With write-behind data is always written into the cache and it is written to the DB some time later (not inline) and possibly batched up and with writes to the same entry over the configured amount of time coalesced into a single insert/update.
Our platform starts with 10.000 registrations, 1.000I can't comment on this, as only you know what processing cost handling each event incurs and what hardware you have for it. Still, these numbers do not seem to be anything to be frightened about.
concurrent users each firing different all types of
events, 1 event per 10 seconds. The number of
registrations is expected to grow quickly in the near
future. The number of concurrent users will grow
relative to that, their event firing behaviour is
expected to stay constant.
Didn't have time to thank you for your clear answers!
You've helped me very much!
What strategy would you recommend for protecting the service request cache from overblow? I mean, if the request producers are time faster than the request executors (very long peak or some misfunction of the executors) this cache might grow to such extend as to consume too much resources.
What patterns can be used for self-stabilizing the input ratio in distributed & partitioned environment?
Let's just take the case of the total outage of some dependent upon background resource. This means that you can get a totally undeterminable backlog. If you solved this, the temporary buildup of backlog because of insufficient background resource capacity is just a trivial special case of that.
I have to do some digging on this, but in general:
1. Add more storage-enabled JVMs (to have more RAM) and CPUs (so that you don't add more GC to the same amount of CPU power). I know, this cannot be continued indefinitely, but if there are spare boxes which can be quickly provisioned for temporary peaks, this might be a viable route. This is not a solution when the backlog can be unlimited.
2. Coherence has some features which will allow you to have virtually infinite data to be stored within a constant amount of JVMs, by putting additional data to the disk. However I have to check how this really works, I haven't used it so far.
The two main questions that come to my mind for this is
- what kind of events you get when you move some data back to the main memory from the disk as it limits what kind of code you are able to put in the storage node. It might be possible that you can just flow the additional stuff to disk and do the event handling when they can be safely brought back to memory (when the depended upon background resources became available again), or not.
- what order do the data from the disk get back to the memory, can you force an order on it and/or can you achieve ordering consistent with the insertion order (interesting in the case when the order of processing the events and correlation between events is relevant)
3. Throttling the input to the caches in the client. This has to be implemented by you.
4. Do overflow on the cache client side. I expect this has to be implemented by you, too.
I will try to read up on the things related to 2, and get back here when I came up with something. If this is relevant to your use cases, you might also want to bring up this question directly with Oracle via Metalink.
Throttling is exactly what I would like to do. I draw my initial data from the JMS bus and I really want to pull only as much as I can process. It absolutely makes no sense for me to empty the JMS queue and then dump it to slow storage. I really want to have self-regulating system that adjusts according to the throughput of each sub-module, so I might (and do) have more than one service request buffers and I want them all to be self-regulating (have the throttling feature).
In single-node case throttling is trivial :) But when I have partitioned (or even replicated) cache as the request buffer I am not sure what will work. For example - how expensive is to check cache's size and similar.
checking for the size of a cache is not particularly expensive if you don't do it too frequently. It has to contact all storage nodes so it will have some latency, but nothing to worry about.
You can write a task which is scheduled on the JVM the JMS receiver is running in for running every once in a while and checks the size of the cache and if it is above a certain threshold, then you can throttle the JMS receiver (obviously you have to process JMS in a pull mode for this to be easily implementable) and when the backlog falls back, it can open the faucet again.
Just allow some spare memory on the storage nodes so that it can cope with the backlog for at least three-four times the peak-time amount of events during the time between two cache size checks.