This content has been marked as final. Show 4 replies
First, your real problem with multiple consumers is that ModifyOrder can arrive before NewOrder not just at the same time. You know one process may read NewOrder and get stuck in GC pause before it had a chance to even read a message and other node can start process ModifyOrder. I were witnessing such race condition in OMS system causing 6 digit loss to a bank.
So you need to worry about resequencing of messages. If messages for order have sequence numbers it can be easy, if you see a gap in sequence, just wait for missing message.
For a start, I would recommend you to take a look at command pattern from Coherence incubator http://coherence.oracle.com/display/INC10/coherence-command-pattern.
You can associate context objects with account or order ID and pass incoming messages as commands.
BTW If you want account level parallelism why not separate messages at queue level? E.i. create a queue per account and use you MOM capabilities to route messages. This way you would avoid resequencing problem. If you wounder how to distribute multiple message consumers across cluster, you can find solution here http://code.google.com/p/gridkit/wiki/DistributedResourceCoordinator
Edited by: alexey.ragozin on Apr 19, 2012 11:28 AM
Thanks for response. The reason why I want to have multiple consumers(application servers) on FIFO queues is that this completely decouples clients and server application in that client requests are served by any live application servers. If any application server is down, it is transparent to clients as requests are just served by remaining consumer/servers. Otherwise, I need to have additional cluster software like VCS and there is still downtime.
I am thinking to have data affinity by account on cache servers, then cache clients(application servers) listens to common JMS queues and adds to its internal queue if the requests belong to accounts it is responsible for. If not, it adds the request to a coherence cache to which all cache clients monitor. Once notified for new items, cache clients check whether requests belong to accounts it is responsible for, if yes, it adds the request to coherence cache. Once request is processed, it is removed from the cache.
Do you think this will work? Also how command pattern can be applied in this case?
You definitely should not build such complex solution from scratch. Have you considered situation when Coherence moving partition from one node to another? There are lot more nasty detail like this.
Start with command pattern. It should fit your model, revealing your from much of dirty work. I'm still concerned about out of order message handling, though.
BTW I'm not sure about your first remark. Why having multiple queues is problematic?
Regarding command pattern by coherence, for a specific context(associated with a specific account), are commands against that context executed one by one with one thread? How is the thread allocated, any effect to my coherence config such as increasing no of thread in cache? Also, as there are many accounts and one to one mapping, we have many contexts. So coherence create one thread per context?
I dont quite understand how command pattern work in coherence. Thanks in advance.