4 Replies Latest reply: Apr 20, 2012 1:18 AM by 925966 RSS

    How coherence can be used to redistribute requests

    925966
      Hi all,

      I am implementing an OMS system. We are thinking to have multiple OMS instances listening to JMS queues as multiple consumers. However, there may be ordering problem as the NewOrder request may be consumed by instance A which is slow but the ModifyOrder request of the same order is consumed by and processed by instance B before instance A process the NewOrder request. So, we need to have redistribute mechanism probably based on account of the order. I am thinking a way on how Coherence can be used to redistribute the requests among clustered OMS instances?

      Appreciate for any hint. Thanks a lot.

      Henry
        • 1. Re: How coherence can be used to redistribute requests
          alexey.ragozin
          Hi Henry,

          First, your real problem with multiple consumers is that ModifyOrder can arrive before NewOrder not just at the same time. You know one process may read NewOrder and get stuck in GC pause before it had a chance to even read a message and other node can start process ModifyOrder. I were witnessing such race condition in OMS system causing 6 digit loss to a bank.

          So you need to worry about resequencing of messages. If messages for order have sequence numbers it can be easy, if you see a gap in sequence, just wait for missing message.

          For a start, I would recommend you to take a look at command pattern from Coherence incubator http://coherence.oracle.com/display/INC10/coherence-command-pattern.
          You can associate context objects with account or order ID and pass incoming messages as commands.

          BTW If you want account level parallelism why not separate messages at queue level? E.i. create a queue per account and use you MOM capabilities to route messages. This way you would avoid resequencing problem. If you wounder how to distribute multiple message consumers across cluster, you can find solution here http://code.google.com/p/gridkit/wiki/DistributedResourceCoordinator

          Regards,
          Alexey

          Edited by: alexey.ragozin on Apr 19, 2012 11:28 AM
          • 2. Re: How coherence can be used to redistribute requests
            925966
            Hi Alexey,

            Thanks for response. The reason why I want to have multiple consumers(application servers) on FIFO queues is that this completely decouples clients and server application in that client requests are served by any live application servers. If any application server is down, it is transparent to clients as requests are just served by remaining consumer/servers. Otherwise, I need to have additional cluster software like VCS and there is still downtime.

            I am thinking to have data affinity by account on cache servers, then cache clients(application servers) listens to common JMS queues and adds to its internal queue if the requests belong to accounts it is responsible for. If not, it adds the request to a coherence cache to which all cache clients monitor. Once notified for new items, cache clients check whether requests belong to accounts it is responsible for, if yes, it adds the request to coherence cache. Once request is processed, it is removed from the cache.

            Do you think this will work? Also how command pattern can be applied in this case?

            thanks
            Henry
            • 3. Re: How coherence can be used to redistribute requests
              alexey.ragozin
              Hi Henry,

              You definitely should not build such complex solution from scratch. Have you considered situation when Coherence moving partition from one node to another? There are lot more nasty detail like this.

              Start with command pattern. It should fit your model, revealing your from much of dirty work. I'm still concerned about out of order message handling, though.

              BTW I'm not sure about your first remark. Why having multiple queues is problematic?

              Regards,
              Alexey
              • 4. Re: How coherence can be used to redistribute requests
                925966
                Hi Alexey,

                Regarding command pattern by coherence, for a specific context(associated with a specific account), are commands against that context executed one by one with one thread? How is the thread allocated, any effect to my coherence config such as increasing no of thread in cache? Also, as there are many accounts and one to one mapping, we have many contexts. So coherence create one thread per context?

                I dont quite understand how command pattern work in coherence. Thanks in advance.

                Henry