We have a proxy which polls XML documents from a MQ queue. After some processing of XML data, the proxy is routed to a DB Adapter which stores the data. But before storing the data we have to check that no register for same document is present in database as many XML for same document are received. Currently we do this by querying for some values in database and invoking a delete DB adapter which deletes existing register for matching values. Then the new register is written with last data for the document.
After doing some testing we are getting multiple registers for same document and that should not happen (as existing registers are supposed to be deleted before inserting the latest one). I think this is happening because although messages are obtained from the queue in FIFO order, OSB is consuming messages concurrently so two messages are been processed in parallel and query finds no registers to delete for both executions, getting two registers written to database.
Please, could you suggest me a way of avoiding this issue? I think the only way to archieve this would be that OSB consumes and processes messages from the MQ queue one after another (no processing of a message starts before previous finishes). Our OSB setup consists of a cluster with two nodes. Also note that we will be receiving about 300.000 XML per day in case performance will be impacted.
Looks like a perfect case for using weblogic JMS Unit of order feature.
Unit of order allow sequential processing within messages belonging to same UOO , while allowing parallel execution of messages across different UOO. Since you are in a cluster and a non weblogic jms message producer, you will need some extra components like a MDB targeted to a single managed server polling the MQ [ using foreign jms] . MDB should read the xml of the mq ,set the UOO and publish to a distributed queue. The Distributed queue can be polled by a JMS proxy service where you have the logic of updating the DB.. The UOO should be set to primary key of the registry. This will make weblogic JMS to publish only one message belonging to the same registry key , to the jms proxy service. All other messages belonging to the same registry key will be made pending in the queue until in the in-flight message processing is complete. other messages belonging to different registry key can be processed in parallel.
Other consideration is to make the MDB single threaded by specifying a work manager in its dispatch policy. This will ensures messages are read off MQ and the UOO is set in order it is received.
Downside with this approach is it requires both managed servers in your cluster to be alive all the time requiring some Auto failover setup like using HACMP, weblogic whole server migration etc.
Thank you for your reply. It seems a quite complicated solution though. I have a few questions:
- If I setup the MQ MDB as single thread and target it to a single managed server, do I also need to configure the JMS and unit-of-order? Wouldn't MQ messages be processed in a single thread and so sequentially withouth need of a unit-of-order?
- Also, how do I identify the MDB for the MQ proxy in WLS deployments. What's the name for the MDB?
- My "primary key" for a registry is compound of two fields in the XML. So if I understood correctly, in the MQ proxy I have to group all XML messages which have same values for those XML fields in a unit of order.
Also, I thought of replacing the "query for existing registry and invoke delete db adapter and then invoke store db adapter" with a store db adapter configured with merge operation instead of insert. Would this solve the issue of getting duplicated registers for same entity?
# If I setup the MQ MDB as single thread and target it to a single managed server, do I also need to configure the JMS and unit-of-order? Wouldn't MQ messages be processed in a single thread and so sequentially withouth need of a unit-of-order?
In that case you can do even the db logic within MDB ..you dont need OSB .. but it will have huge performance impact as all message processing are constrained by the single threaded mdb. Using UOO you can get parallel processing for improved throughput , while still maintaining sequential processing within the related group of messages.
# Also, how do I identify the MDB for the MQ proxy in WLS deployments. What's the name for the MDB?
You will have to write your own.
# - My "primary key" for a registry is compound of two fields in the XML. So if I understood correctly, in the MQ proxy I have to group all XML messages which have same values for those XML fields in a unit of order.
Yes .. You can derive the UOO from multiple xml fields.
So I have to write my own MDB for the MQ queue. I thought I could use the OSB MQ proxy for that. I suppose it is not possible to tune the MQ proxy in the way we need as I can't find any deployment that corresponds to the MQ proxy (MDB for JMS proxys are listed in server deployments), that's why I need to write a custom one.
I think I'm going to try to find a different way to implement our requirements to avoid having to configure all these elements. I've thought of writing a database trigger that performs a check when inserting the XML registry. The check will select for a newer record for that document in the table. In that case I will raise and exception preventing the insertion of the older registry. Do you think I would have the same problem with this alternative? That is, that two transactions will be executed at the same time so non of each will see the other registry in the table (as it's not been written yet) so both registrys would be written? I think If I implement this at database level with the trigger this would not happen.