This discussion is archived
7 Replies Latest reply: Apr 11, 2012 7:30 PM by 687626 RSS

OSB: Proxy doesn't consume JMS messages

680075 Pro
Currently Being Moderated
Guys, I'm getting stuck with this one ...

I've got four JMS proxy services. One of them uses Publish to enqueue a message into the three others. This way I get three copies of a single message, each in dedicated JMS queue. The same three JMS proxies are supposed to consume messages from JMS queues and process them. The configuration is the same for all three proxies and all three JMS queues. In fact all was cloned and the difference is only in suffix. So, I've got QueueA, QueueB, QueueC and ProxyA, ProxyB and ProxyC. So if I stick to these names, ProxyMain uses 3x Publish actions to enqueue a message into ProxyA, ProxyB and ProxyC at the same time. This is a proved solution and it has worked for years flawlessly!

However, something weird happened a couple of days ago. All of a sudden, one of those three consuming JMS proxies doesn't consume anything. There's no issue on producing side (the first proxy) as the numbers of messages in all three queues are still increasing. Still, only messages from QueueA and QueueB are being consumed by ProxyA and ProxyB. The third queue (QueueC) doesn't have any consumer attached from ProxyC and I don't know why.

I've already checked everything I know - proxy is enabled, there are no errors in logs related to the proxy or JMS, the issue is not reproducible in test environment with the same configuration, there are no JMS Quota or connection factory restrictions, consuming flag is enabled on queue ...

The lack of errors in logs makes me feel like the ProxyC is connected to something else. However, the same proxy is used for producing messages (from ProxyMain) and messages are incoming to the queue so the endpoint set in proxy must be correct!

Any help, hint or idea is appreciated ...
  • 1. Re: OSB: Proxy doesn't consume JMS messages
    Patrick Taylor Pro
    Currently Being Moderated
    Given that you have validated most of the things already :), I could think of the below options:

    - Do you use any message expiry settings on the messages placed in the queue? If yes, may the messages are expired in the third queue and hence the proxy is not consuming any of the messages.

    - Set the java:alsb-transports-debug to true in the alsbdebug.xml and restart the servers. This should log some more details of the jms transport interactions. See if you can find anything important from the debug logs.

    Thanks,
    Patrick
  • 2. Re: OSB: Proxy doesn't consume JMS messages
    680075 Pro
    Currently Being Moderated
    >
    - Do you use any message expiry settings on the messages placed in the queue? If yes, may the messages are expired in the third queue and hence the proxy is not consuming any of the messages.
    >

    Yes, message expiration is set on queue, but it takes a couple of days for a message to expire. I can see thousands of messages in "Messages Current" under queue monitoring tab, so the expiration is not the reason.

    >
    - Set the java:alsb-transports-debug to true in the alsbdebug.xml and restart the servers. This should log some more details of the jms transport interactions. See if you can find anything important from the debug logs.
    >

    I'll try this setting, but the problem is that since queue contains thousands of messages, I can't simply restart the server because when the proxy eventually wakes up, it will try to flood the backend and most probably kill it. :-) I need to prepare some throttling hack for that first.

    Still, the reason why proxy doesn't consume messages is a mystery.
  • 3. Re: OSB: Proxy doesn't consume JMS messages
    735712 Newbie
    Currently Being Moderated
    The throttling hack can be very simple. In weblogic, create a Work Manager and give it a max thread constraint. In the proxy service enter the workmanager name in the Dispatch Policy attribute.
    This will limit the amount of concurrent threads of the proxy service.

    More info on workmanagers: http://docs.oracle.com/cd/E12840_01/wls/docs103/config_wls/self_tuned.html
  • 4. Re: OSB: Proxy doesn't consume JMS messages
    687626 Expert
    Currently Being Moderated
    The third queue (QueueC) doesn't have any consumer attached from ProxyC and I don't know why
    Try renaming ProxyC to say ProxyC1 and activate the session. This will create a new underlying MDB for the jms proxy and you will see consumers getting set on the queue.. Faced this issue many times in our production environment and easiest workaround we found was to rename the proxy service [ toggling enable/disable proxy doesn't seem to have ny effect).. you can change the name to the original proxy once the consumers are set..                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
  • 5. Re: OSB: Proxy doesn't consume JMS messages
    Anuj Dwivedi Guru
    Currently Being Moderated
    Hisaak,

    Are you using custom work-manager with the proxies or proxies are using default work-manager? I experienced a similar problem earlier where proxy was not getting enough threads to start/initialize itself and hence polling was not working. I think a similar bug was also fixed in PS2. You may consider logging a SR with support if issue is still there.

    Regards,
    Anuj
  • 6. Re: OSB: Proxy doesn't consume JMS messages
    680075 Pro
    Currently Being Moderated
    >
    Are you using custom work-manager with the proxies or proxies are using default work-manager? I experienced a similar problem earlier where proxy was not getting enough threads to start/initialize itself and hence polling was not working. I think a similar bug was also fixed in PS2. You may consider logging a SR with support if issue is still there.
    >

    We use default work manager for majority of proxies, including these JMS proxies. When you faced this, have you spot any error or warning in logs? I ask because this is really frightening - it doesn't work, yet there is no message anywhere.

    And, thanks to atheek for renaming trick. I'll try this one.
  • 7. Re: OSB: Proxy doesn't consume JMS messages
    687626 Expert
    Currently Being Moderated
    I experienced a similar problem earlier where proxy was not getting enough threads to start/initialize itself and hence polling was not working.
    We also experienced a problem similar to this. This occurs when you use same work manager with a max thread constraint in the dispatch policies of 2 more proxies.. This imposes the max thread constraint across all the proxy services.
    In this situation you will see consumers on the individual queues but the messages will be stuck in a 'receive' substate.

    e.g. q1 --> proxy 1 q2 --> proxy 2 . Same WM with max thread 1 is assigned to the dispatch policy of proxy1 and proxy 2. Q1 will have consumer current =1 . Q2 will have consumer current =1. But only a single thread will be used for servicing proxy1 and proxy2 as they share the same WM. In this case you will see messages getting processed in one queue but stuck in a receive state in the other, since the thread is busy with servicing the first proxy.

    Since consumer current on the stuck queue is 1 , you will have an intuition that there is a thread working on it , but in fact it is not.

    The behaviour of thread constraints is best explained in this blog: https://blogs.oracle.com/WebLogicServer/entry/threads_constraints_in_work_ma

    The issue with zero consumers is due to OSB somehow missing the association of the jms proxy service to the underlying mdb ear deployment. Tricks like renaming, creates a new mdb and a new association is created and maintained by OSB. This often leaves the old mdbs as stray deployments which you will have to manually delete from Weblogic admin console..

    Edited by: atheek1 on 11/04/2012 19:27

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points