4 Replies Latest reply: Nov 30, 2010 4:39 AM by 814248 RSS

    Processing JMS messages in sequence one after another

    814248
      This is my use case which I wan to implement in OSB.

      1. I have a queue which will receive messages in huge numbers from a single sender.
      2. I have a proxy which will read each message and process the message and send it out.

      In step 1 , I want each message to be read sequentially and when step 2 finishes executing then I want to read the next message from the queue.

      I need to give an acknowledgment to JMS that the message has finished processing and it can make the next message available to the proxy.

      What is the best option to implement this use case ?.

      Regards
        • 1. Re: Processing JMS messages in sequence one after another
          687626
          Weblogic jms' unit of order is the best approach for this.
          http://download.oracle.com/docs/cd/E12840_01/wls/docs103/jms/uoo.html
          • 3. Re: Processing JMS messages in sequence one after another
            819690
            By default Weblogic JMS Queues are FIFO. But that is true only if there is only one consumer of the queue.

            In case of a clustered environment that is not true and in case of OSB even with a single managed server multiple threads are created for a JMS listener Proxy(by default 16). So there will be 16 instances of the Proxy trying to read messages from the Queue and each will be handed out a message and sequential integrity is disturbed.

            There are three ways to overcome this problem:
            1. Unit of Order
            This is suitable only when you want messages of the same group to be processed sequentially while multiple groups can be processed in parallel.
            For e.x. you are getting multiple orders for multiple customers and you want to ensure that the orders of same customer are processed sequentially (in case a customer tries to change the quantity of an order then the latest one should be processed last). In this case you can set UOO as the CustomerID and each customer's order will be processed sequentially but orders of different customers will be still processed in parallel. WLS JMS achieves this by assigning a target queue instance on the cluster for each UOO value. It will assign Q!MS1 to CustID=1, Q!MS2 to CustID=2 and so on. All messages of the same UOO(of the same customer) will go to the same instance on the queue. Furthermore it will ensure that no two messages of the same UOO are released together to the listeners. So if a queue instance has 5 messages each of both Cust=1 and Cust=2, only one message each of Cust1 and Cust2 will be processed by the listeners even if there are 16 listening threads.
            Pros and Cons:
            Overall performance is improved since you can still achieve some parallel processing.
            There will be a slight overhead of processing the UOO headers on the JMS servers.
            Can not ensure sequencing of all the messages. (Although you can achieve that by setting the same UOO on all the messages)
            If the managed server assigned to a UOO is down, if a publisher tried to put the next message for that UOO, it will fail since it will not try to send that message to any other managed server.

            2. Single threaded processing
            If you don't want to process even different groups of messages in parallel and if you want absolute sequencing (i.e. irrespective of the CustomerID you want all the messages to be processed in the sequence they arrive) you will need to process them in a single threaded model. The JMS queue should be deployed on only one managed server of the cluster. The JMS proxy listening to the queue should also be deployed on a single managed server(You will need to change the targeting for the EJB created for this Proxy in the Deployments) and there should be a Work Manager for this Proxy with Maximum Thread Constraint set as 1. In the Connection Factory which the Proxy uses, set the Maximum Messages per session to 1. Another approach for Single threaded could be to set the same value of UOO on all of the messages. This will make all the messages to go to the same Q instance in the cluster and also will make sure that even if Proxy has multiple threads, only one message will be processed at a time.
            Pros and Cons:
            Completely single threaded processing, will take more time since messages will be processed one after the other.
            Load balancing will go awry as all the messages will be processed by only one server.

            3. Using custom implementation
            The most complex way is to create a completely custom implementation. Put all the messages in a DB and then process them one at a time based on timestamp.
            Pros and Cons:
            It will take more effort to implement than the other two approaches.
            It will again need single threaded processing after the messages are put on the DB
            Performance wise there will be more impact because of additional DB calls
            More complex to maintain


            Based on your exact requirements you can choose which approach you want. If you have SOA suite as well them it would be better to move this solution to SOA suite as OESB(Mediator) component of SOA suite has re-sequencing feature.
            • 4. Re: Processing JMS messages in sequence one after another
              814248
              Thanks everyone for your help. Really appreciate it.

              As mentioned above in my use case, only a single consumer will read from the queue and my solution will be hosted in a single server domain.

              I managed to achieve this by changing 2 properties. I created a proxy which reads from the queue and calls a BS where I pass wrong parameters . I set the routing option to retry 3 times every 10 secs. I created a log in the proxy to log the message body and the error time.
              I sent 4 messages to the queue and I check the log the messages are processed in sequential order and the second message starts processing after the first error is logged (as my scenario)

              In the connection factory I change the Client Acknowledge Policy to Previous. (in the Configuration->Client tab).
              In the queue I checked the Auto-generate Unit-of-Order Name option. (in the Configuration->General tab)

              regards.