This discussion is archived
14 Replies Latest reply: Jun 15, 2010 5:13 AM by 807581 RSS

Broker not delivering messages

807581 Newbie
Currently Being Moderated
Hi,

I have a system which has been working fine for a year and is now causing some issues. It is running OpenMQ 4.2 with Java 1.6 (1.6.0_11-b03) on Solaris (SunOS 5.10 Generic_138888-07 sun4v sparc SUNW,Sun-Blade-T6340). No changes have been made to the configuration or the system recently.

The issue is:
1- A producer sends a bunch of messages (around 100) in a queue
2- A consumer listening to the same queue picks up say 96 of the messages and leaves 4 of them in the queue (4 random messages, not in sequence or first or last)

When I try to read these messages left on the queue using a separate script (different JVM than main software) which just connects to the queue and dumps its content on the screen, the call to consumer.receive() does not return any message. Yet listing the queue content (with "imqcmd list") actually shows the 4 messages in the queue and the consumer attached to it :/

When I restart the broker then the 4 messages are delivered correctly and processed by the application.

The broker log (set to default INFO) doesn't show any error or unusual trace, my application logs either.

This problem happened the first time after the broker had been running for 3 months in a row so we though it would probably not happen again if we restart it every month. Yet it now happened again after only 20 days since the last restart of the broker.

Another side problem: when I tried to dump the JVM content using "kill -3" I could not find it in the broker log (set using -Dimq.log.file.filename=broker.log on the command line starting the broker), any idea of how I should proceed?

Thank you very much for any help you could provide

Mickael
  • 1. Re: Broker not delivering messages
    805009 Newbie
    Currently Being Moderated
    The issue is:
    1- A producer sends a bunch of messages (around 100) in a queue
    2- A consumer listening to the same queue picks up say 96 of the messages and leaves 4 of them in the queue (4 random messages, not in sequence or first or last)

    When I try to read these messages left on the queue using a separate script (different JVM than main software) which just connects to the queue and dumps its content on the screen, the call to consumer.receive() >does not return any message. Yet listing the queue content (with "imqcmd list") actually shows the 4 messages in the queue and the consumer attached to it :/
    Did you close the first consumer after it received the 96 messages?

    Nigel
  • 2. Re: Broker not delivering messages
    807581 Newbie
    Currently Being Moderated
    If you didn't change the caching size (default 1000) on the client side, this is the expected behavior. The client reads ahead the cache size of messages for better performance. So the remaining 4 messages are waiting to be read by the client while the client is up and running. While the messages are in the cache for this single client, they are not available to other clients. Right sizing this cache is a balancing act between parallelism and raw performance of a single client. You can:

    1. Shutdown the client before running the other application (or close the connection from within the client)
    2. Resize the cache size (see the admin documentation for that)

    TE

    Edited by: TravelEntity on 10-Jun-2010 2:29 PM

    Edited by: TravelEntity on 10-Jun-2010 2:31 PM
  • 3. Re: Broker not delivering messages
    807581 Newbie
    Currently Being Moderated
    Hi,

    First thanks for your replies.

    @nigeldeakin: No I don't close the first consumer because it keeps on listening until a new messages is posted (loop on consumer.receive() with a timeout of 1000 ms). The same job runs all day and create one consumer at startup that it reuses during the whole day.

    @TravelEntity: I didn't set the cache parameter so it must be set to the default value for 4.2. However I know that even if only one message is posted to the queue the client will pick it up instantly. If I correctly understand what you explained then the consumer would wait for a minimum number of messages before starting to process right? I will have a look at this caching feature you described.

    I realized I should have mentioned some important details in my previous post, let's retry :)

    The issue is:
    1- The producer (only one) sends a bunch of messages (around 100) in a queue
    2- The consumer (only one) listening to the same queue picks up say 96 of the messages and leaves 4 of them in the queue (4 random messages, not in sequence or first or last)
    3- The producer sends another bunch of messages in the same queue
    4- The consumer picks up say 98 of the messages and leaves 2 of them in the queue
    5- mqlist then shows 6 messages in the queue, consumer.receive() returns null if tried from the main client or from a separate queue dump tool

    Thanks again for your help.

    Mickael
  • 4. Re: Broker not delivering messages
    807581 Newbie
    Currently Being Moderated
    If there is less than a 1000 messages, then it would consumer whatever amount of messages there is on the broker, so that is not it.

    From what you are saying, some messages are not consumed (and like you said, the ordering of the message doesn't seem to have an impact). So:

    1. Are you sure the messages made it to the broker? Did the producer really did a commit (or you're using auto-commit)?
    2. Are you using a selector?
    3. Are you using the ttl?
    4. Is there any messages in the dead letter queue (dlq)?
    5. Is the client a J2SE, J2EE or other technology?

    TE
  • 5. Re: Broker not delivering messages
    805009 Newbie
    Currently Being Moderated
    I agree with TravelEntity.
    The issue is:
    1- The producer (only one) sends a bunch of messages (around 100) in a queue
    2- The consumer (only one) listening to the same queue picks up say 96 of the messages and leaves 4 of them in the queue (4 random messages, not in sequence or first or last)
    The basic behaviour here is that the moment this consumer is created (and the connection started) the broker will send all 100 messages to the consumer (assuming you're not using a message selector), where they will be cached in the client until they are delivered to the client application. Whilst they remain in the client they will be unavailable to other consumers. When the first consumer closes, any unconsumed messages will be returned to the broker and become available for delivery to other consumers. The number of messages that are "pre-fetched" to a consumer is defined by the consumerFlowLimit of the connection or destination (see MQ admin guide for details). The default is 1000.

    So this appears to explain why when you created a second consumer, it didn't receive any messages despite there being four messages left on the queue.

    However I don't understand what you mean by "leave 4 random messages in the queue". How were you doing that? Messages are consumed from the queue in order - unless you use a message selector.
    3- The producer sends another bunch of messages in the same queue
    4- The consumer picks up say 98 of the messages and leaves 2 of them in the queue
    5- mqlist then shows 6 messages in the queue, consumer.receive() returns null if tried from the main client or from a separate queue dump tool
    What is mqlist? It's not a MQ tool.

    You can use the MQ command imqcmd to report on how many messages are being cached on the client in this way: in the following sample of its usage, "Count" is the total number of messages on the queue, and "UnAck" tells you how many of those messages are being cached on the client.
    C:\mq\bin>imqcmd list dst  
    
    ---------------------------------------------------------------------------------------------
       Name     Type    State      Producers        Consumers                  Msgs
                                Total  Wildcard  Total  Wildcard  Count  Remote  UnAck  Avg Size
    ---------------------------------------------------------------------------------------------
    INPUT.Q     Queue  RUNNING  0      -         1      -         9268   0       4709   1237.2805
    mq.sys.dmq  Queue  RUNNING  0      -         0      -         0      0       0      0.0
  • 6. Re: Broker not delivering messages
    807581 Newbie
    Currently Being Moderated
    Thanks again for your replies.

    @TravelEntity:

    1-Yes because when the broker is restarted the messages are processed correctly. Yes the producer did an explicit commit of the session.
    2- No the consumer does not use a selector.
    3- No I don't set the ttl.
    4- No there is no message in this queue
    5- The client is a stand alone Java application (no J2EE, no container)

    @nigeldeakin:

    First on the "4 random messages" part, what I mean is that in this case messages are not consumed in order. 4 messages are arbitrarily not consumed, they are not in a sequence, not the first or the last of the list of messages added. Messages posted before or after these 4 are consumed correctly. We are not using a messages selector.

    By mqlist I meant our tool to run the command "imqcmd list dst -t q " sorry for the lack of clarity :)

    I am pretty confident that these messages were not showing in the UNACK columns, unfortunately I did not take a screenshot >.<



    Another important detail: when I tried to restart the job which is running the "faulty" consumer, the current consumer is closed and a new one is created but has the same problem. Restarting the job recreates a whole set of new objects: queue factory, session and consumer.

    Thanks again.
  • 7. Re: Broker not delivering messages
    805009 Newbie
    Currently Being Moderated
    4 messages are arbitrarily not consumed, they are not in a sequence, not the first or the last of the list of messages added. Messages posted before or after these 4 are consumed correctly. We are not using a messages selector.
    What do you mean "not consumed"? How is your first consumer receiving messages? Are you using a MessageListener or calling receive()?

    Nigel
  • 8. Re: Broker not delivering messages
    807581 Newbie
    Currently Being Moderated
    I mean that the call to receive() (from the class javax.jms.MessageConsumer) returns null whereas I can see the messages in the queue.
  • 9. Re: Broker not delivering messages
    805009 Newbie
    Currently Being Moderated
    OK. Are you saying that your first consumer make 100 calls to receive() and that four of those returned null?

    What timeout did you specify?

    Nigel
  • 10. Re: Broker not delivering messages
    807581 Newbie
    Currently Being Moderated
    The timeout passed to the receive() method call is 1000 (ms).

    I am not saying that the "first consumer make 100 calls to receive() and that four of those returned null?".

    What I mean is:

    Say that messages 1 to 100 have been produced in the queue. The consumer calls receive() constantly, thus receiveing one message at a time. It receives messages 1 to 22, then 24 to 57, then 59 to 65, then 67 to 92 and then 94 to 100. The consumer then continues to call the method receive() but the calls returns null. Yet the messages 23, 58, 66 and 93 are left in the queue.

    Thanks
  • 11. Re: Broker not delivering messages
    805009 Newbie
    Currently Being Moderated
    OK. That's odd. Was there another consumer in existence whilst the first consumer did this?

    Was your receive() called from a simple loop, in a single thread?

    Are these persistent messages?

    Were there any error messages in the broker log?

    Nigel
  • 12. Re: Broker not delivering messages
    807581 Newbie
    Currently Being Moderated
    Hi,

    Yes this odd! Especially since the system has been working for more than a year without any problem!

    There is only one consumer and one producer on this queue. The receive call is in a simple while loop in a single thread.

    The messages are persistent and the broker is using filesystem storage.

    Unfortunately there is no error in the broker log when the problem occur nor when it restarts...

    There is no exception raised on the client side either.

    Thanks
  • 13. Re: Broker not delivering messages
    807581 Newbie
    Currently Being Moderated
    So I guess the only question left is, are you using an MQ cluster? To my knowledge, this is the only reason why the messages would arrive out of order. And if this is the case, is the producer experiencing failover? Meaning are some of the messages on one broker and some on the other?

    TE
  • 14. Re: Broker not delivering messages
    807581 Newbie
    Currently Being Moderated
    Hi,

    Sorry for the late reply.

    No we are not using clustering. We have only one broker instance.

    Thanks for helping.