9 Replies Latest reply: Sep 9, 2011 6:12 PM by ak RSS

    Destination Q:mq.sys.dmq could not be found in the store

    807581
      I tried to Google these errors but nothing really came up.
      [26/Jan/2009:09:56:02 CET] ERROR [B3013]: Destination Q:mq.sys.dmq could not be found in the store
      [26/Jan/2009:09:56:02 CET] WARNING Update consumer [consumer:0, type=AUTO_ACKNOWLEDGE] state failed for message 31327-10.13.1.149(ab:2d:72:8:5b:9a)-1558-1232959980070: Destination Q:mq.sys.dmq could not be found in the store
      [26/Jan/2009:09:56:02 CET] ERROR Error in processing ack on 31327-10.13.1.149(ab:2d:72:8:5b:9a)-1558-1232959980070 for [consumer:5284726850505289224, type=AUTO_ACKNOWLEDGE]:
      com.sun.messaging.jmq.jmsserver.util.BrokerException: Destination Q:mq.sys.dmq could not be found in the store
           at com.sun.messaging.jmq.jmsserver.persist.file.MsgStore.getDstMsgStore(MsgStore.java:777)
           at com.sun.messaging.jmq.jmsserver.persist.file.MsgStore.updateInterestState(MsgStore.java:669)
           at com.sun.messaging.jmq.jmsserver.persist.file.FileStore.updateInterestState(FileStore.java:929)
           at com.sun.messaging.jmq.jmsserver.core.PacketReference.acknowledged(PacketReference.java:1697)
           at com.sun.messaging.jmq.jmsserver.core.Session$ackEntry.acknowledged(Session.java:521)
           at com.sun.messaging.jmq.jmsserver.core.Session.ackMessage(Session.java:1309)
           at com.sun.messaging.jmq.jmsserver.core.Session.ackMessage(Session.java:1294)
           at com.sun.messaging.jmq.jmsserver.data.handlers.AckHandler.handleAcks(AckHandler.java:413)
           at com.sun.messaging.jmq.jmsserver.data.handlers.AckHandler.handle(AckHandler.java:246)
           at com.sun.messaging.jmq.jmsserver.data.PacketRouter.handleMessage(PacketRouter.java:181)
           at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.readData(IMQIPConnection.java:1489)
           at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.process(IMQIPConnection.java:644)
           at com.sun.messaging.jmq.jmsserver.service.imq.OperationRunnable.process(OperationRunnable.java:170)
           at com.sun.messaging.jmq.jmsserver.util.pool.BasicRunnable.run(BasicRunnable.java:493)
           at java.lang.Thread.run(Unknown Source)
      [26/Jan/2009:09:56:02 CET] ERROR [B3100]: Unexpected Broker Internal Error : [-------------------------------------------Internal Error: Invalid Acknowledge Packet processing
        notifying client
            Packet: ACKNOWLEDGE(24):37103-10.13.1.149(82:29:9b:6d:53:83)-1475-1232960162481
        Magic/Version: 469754818/301     Size: 116      Type: ACKNOWLEDGE(24)
           Expiration: 0                 Timestamp: 1232960162481
            Source IP: 10.13.1.149(82:29:9b:6d:53:83)  Port: 1475     Sequence: 37103
      Property Offset: 76               Property Size: 0
           Encryption: 0     Priority: 5
                Flags: A                  consumerID: 1354
         TransactionID: 0
             MessageID: 37103-10.13.1.149(82:29:9b:6d:53:83)-1475-1232960162481
            Properties: null
         Message Body: 40 bytes [5284726850505289224:31327-10.13.1.149(ab:2d:72:8:5b:9a)-1558-1232959980070]
      Internal Buffers (useDirect=false):
      Fixed Header Buffer:java.nio.HeapByteBuffer[pos=0 lim=72 cap=72]
      --------------------------------------------]:
      com.sun.messaging.jmq.jmsserver.util.BrokerException: Destination Q:mq.sys.dmq could not be found in the store
           at com.sun.messaging.jmq.jmsserver.persist.file.MsgStore.getDstMsgStore(MsgStore.java:777)
           at com.sun.messaging.jmq.jmsserver.persist.file.MsgStore.updateInterestState(MsgStore.java:669)
           at com.sun.messaging.jmq.jmsserver.persist.file.FileStore.updateInterestState(FileStore.java:929)
           at com.sun.messaging.jmq.jmsserver.core.PacketReference.acknowledged(PacketReference.java:1697)
           at com.sun.messaging.jmq.jmsserver.core.Session$ackEntry.acknowledged(Session.java:521)
           at com.sun.messaging.jmq.jmsserver.core.Session.ackMessage(Session.java:1309)
           at com.sun.messaging.jmq.jmsserver.core.Session.ackMessage(Session.java:1294)
           at com.sun.messaging.jmq.jmsserver.data.handlers.AckHandler.handleAcks(AckHandler.java:413)
           at com.sun.messaging.jmq.jmsserver.data.handlers.AckHandler.handle(AckHandler.java:246)
           at com.sun.messaging.jmq.jmsserver.data.PacketRouter.handleMessage(PacketRouter.java:181)
           at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.readData(IMQIPConnection.java:1489)
           at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.process(IMQIPConnection.java:644)
           at com.sun.messaging.jmq.jmsserver.service.imq.OperationRunnable.process(OperationRunnable.java:170)
           at com.sun.messaging.jmq.jmsserver.util.pool.BasicRunnable.run(BasicRunnable.java:493)
           at java.lang.Thread.run(Unknown Source)
      They seem to occur at regular intervals, they occur about 500 times every minute or so, we do send thousands of messages with a timeout of 100 seconds.

      Any pointers as to what we might be doing wrong would be very much appreciated, thanks.
        • 1. Re: Destination Q:mq.sys.dmq could not be found in the store
          3004
          This looks like bug 6722509. Should be fixed 4.3

          Tom
          • 2. Re: Destination Q:mq.sys.dmq could not be found in the store
            807581
            We are using 4.3 build 7-g as downloaded last Saturday.

            Do I need to provide more information, if so, which?

            We use the RefFSContextFactory, with the bindings on a share.

            OpenMQ is on one server, and two other servers connect to it using providerurl \\servername\OpenMQ\objectstoredir

            Thanks again!

            Edited by: brecht.yperman on Jan 26, 2009 6:03 AM
            • 3. Re: Destination Q:mq.sys.dmq could not be found in the store
              807581
              We don't really need the dead letter queue, so disabling the dead letter queue for all queues might be a workaround not to get these errors anymore?
              • 4. Re: Destination Q:mq.sys.dmq could not be found in the store
                3004
                Well, looks like it may be a new problem. Do you have a test case ? Could you set the property imq.log.level=DEBUGHIGH and try to reproduce the problem and then forward the broker's log file to OpenMQ mailing list users@mq.dev.java.net ?

                Tom
                • 5. Re: Destination Q:mq.sys.dmq could not be found in the store
                  807581
                  Unfortunately I do not have a test case.

                  I have been able to reproduce it today, and I'm sending it to the mailinglist.
                  • 6. Re: Destination Q:mq.sys.dmq could not be found in the store
                    807581
                    When disabling the dead letter queue for this queue, I get other errors:
                    [30/Jan/2009:16:39:11 CET] Unable to process acknowledgement, Ignoring:
                    java.lang.NullPointerException
                         at com.sun.messaging.jmq.jmsserver.core.Session$ackEntry.acknowledged(Session.java:514)
                         at com.sun.messaging.jmq.jmsserver.core.Session.ackMessage(Session.java:1309)
                         at com.sun.messaging.jmq.jmsserver.core.Session.ackMessage(Session.java:1294)
                         at com.sun.messaging.jmq.jmsserver.data.handlers.AckHandler.handleAcks(AckHandler.java:413)
                         at com.sun.messaging.jmq.jmsserver.data.handlers.AckHandler.handle(AckHandler.java:246)
                         at com.sun.messaging.jmq.jmsserver.data.PacketRouter.handleMessage(PacketRouter.java:181)
                         at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.readData(IMQIPConnection.java:1489)
                         at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.process(IMQIPConnection.java:644)
                         at com.sun.messaging.jmq.jmsserver.service.imq.OperationRunnable.process(OperationRunnable.java:170)
                         at com.sun.messaging.jmq.jmsserver.util.pool.BasicRunnable.run(BasicRunnable.java:493)
                         at java.lang.Thread.run(Thread.java:595)
                    [30/Jan/2009:16:39:11 CET] ERROR [B3100]: Unexpected Broker Internal Error : [-------------------------------------------Internal Error: Invalid Acknowledge Packet processing
                      notifying client
                          Packet: ACKNOWLEDGE(24):303-192.168.0.13(df:d9:a5:56:19:89)-2236-1233329951902
                      Magic/Version: 469754818/301     Size: 116      Type: ACKNOWLEDGE(24)
                         Expiration: 0                 Timestamp: 1233329951902
                          Source IP: 192.168.0.13(df:d9:a5:56:19:89)  Port: 2236     Sequence: 303
                    Property Offset: 76               Property Size: 0
                         Encryption: 0     Priority: 5
                              Flags: A                  consumerID: 75
                       TransactionID: 0
                           MessageID: 303-192.168.0.13(df:d9:a5:56:19:89)-2236-1233329951902
                          Properties: null
                       Message Body: 40 bytes [90950674893657346:27-192.168.0.13(8d:35:45:62:c8:10)-2375-1233329761245]
                    Internal Buffers (useDirect=false):
                    Fixed Header Buffer:java.nio.HeapByteBuffer[pos=0 lim=72 cap=72]
                    --------------------------------------------]:
                    com.sun.messaging.jmq.jmsserver.util.BrokerException: Unable to process ack
                         at com.sun.messaging.jmq.jmsserver.core.Session$ackEntry.acknowledged(Session.java:530)
                         at com.sun.messaging.jmq.jmsserver.core.Session.ackMessage(Session.java:1309)
                         at com.sun.messaging.jmq.jmsserver.core.Session.ackMessage(Session.java:1294)
                         at com.sun.messaging.jmq.jmsserver.data.handlers.AckHandler.handleAcks(AckHandler.java:413)
                         at com.sun.messaging.jmq.jmsserver.data.handlers.AckHandler.handle(AckHandler.java:246)
                         at com.sun.messaging.jmq.jmsserver.data.PacketRouter.handleMessage(PacketRouter.java:181)
                         at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.readData(IMQIPConnection.java:1489)
                         at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.process(IMQIPConnection.java:644)
                         at com.sun.messaging.jmq.jmsserver.service.imq.OperationRunnable.process(OperationRunnable.java:170)
                         at com.sun.messaging.jmq.jmsserver.util.pool.BasicRunnable.run(BasicRunnable.java:493)
                         at java.lang.Thread.run(Thread.java:595)
                    Caused by: java.lang.NullPointerException
                         at com.sun.messaging.jmq.jmsserver.core.Session$ackEntry.acknowledged(Session.java:514)
                         ... 10 more
                    We also sometimes receive expired messages from this queue... (the consumer is on the same server as the broker, so no time sync issues here, I guess).
                    • 7. Re: Destination Q:mq.sys.dmq could not be found in the store
                      807581
                      If we send all messages using DeliveryMode.PERSISTENT, we don't seem to get the errors...
                      • 8. Re: Destination Q:mq.sys.dmq could not be found in the store
                        807581
                        After a while, other errors do occur though...
                        [31/Jan/2009:12:10:53 CET] Unable to process acknowledgement, Ignoring:
                        java.lang.NullPointerException
                             at com.sun.messaging.jmq.jmsserver.core.Session$ackEntry.acknowledged(Session.java:514)
                             at com.sun.messaging.jmq.jmsserver.core.Session.ackMessage(Session.java:1309)
                             at com.sun.messaging.jmq.jmsserver.core.Session.ackMessage(Session.java:1294)
                             at com.sun.messaging.jmq.jmsserver.data.handlers.AckHandler.handleAcks(AckHandler.java:413)
                             at com.sun.messaging.jmq.jmsserver.data.handlers.AckHandler.handle(AckHandler.java:246)
                             at com.sun.messaging.jmq.jmsserver.data.PacketRouter.handleMessage(PacketRouter.java:181)
                             at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.readData(IMQIPConnection.java:1489)
                             at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.process(IMQIPConnection.java:644)
                             at com.sun.messaging.jmq.jmsserver.service.imq.OperationRunnable.process(OperationRunnable.java:170)
                             at com.sun.messaging.jmq.jmsserver.util.pool.BasicRunnable.run(BasicRunnable.java:493)
                             at java.lang.Thread.run(Thread.java:595)
                        [31/Jan/2009:12:10:53 CET] ERROR [B3100]: Unexpected Broker Internal Error : [-------------------------------------------Internal Error: Invalid Acknowledge Packet processing
                          notifying client
                              Packet: ACKNOWLEDGE(24):46429-192.168.0.13(f1:ee:30:25:f8:65)-1274-1233400253342
                          Magic/Version: 469754818/301     Size: 116      Type: ACKNOWLEDGE(24)
                             Expiration: 0                 Timestamp: 1233400253342
                              Source IP: 192.168.0.13(f1:ee:30:25:f8:65)  Port: 1274     Sequence: 46429
                        Property Offset: 76               Property Size: 0
                             Encryption: 0     Priority: 5
                                  Flags: A                  consumerID: 13128
                           TransactionID: 0
                               MessageID: 46429-192.168.0.13(f1:ee:30:25:f8:65)-1274-1233400253342
                              Properties: null
                           Message Body: 40 bytes [863880962091099396:8429-192.168.0.13(fe:3b:74:b9:d0:73)-1344-1233397287253]
                        Internal Buffers (useDirect=false):
                        Fixed Header Buffer:java.nio.HeapByteBuffer[pos=0 lim=72 cap=72]
                        --------------------------------------------]:
                        com.sun.messaging.jmq.jmsserver.util.BrokerException: Unable to process ack
                             at com.sun.messaging.jmq.jmsserver.core.Session$ackEntry.acknowledged(Session.java:530)
                             at com.sun.messaging.jmq.jmsserver.core.Session.ackMessage(Session.java:1309)
                             at com.sun.messaging.jmq.jmsserver.core.Session.ackMessage(Session.java:1294)
                             at com.sun.messaging.jmq.jmsserver.data.handlers.AckHandler.handleAcks(AckHandler.java:413)
                             at com.sun.messaging.jmq.jmsserver.data.handlers.AckHandler.handle(AckHandler.java:246)
                             at com.sun.messaging.jmq.jmsserver.data.PacketRouter.handleMessage(PacketRouter.java:181)
                             at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.readData(IMQIPConnection.java:1489)
                             at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.process(IMQIPConnection.java:644)
                             at com.sun.messaging.jmq.jmsserver.service.imq.OperationRunnable.process(OperationRunnable.java:170)
                             at com.sun.messaging.jmq.jmsserver.util.pool.BasicRunnable.run(BasicRunnable.java:493)
                             at java.lang.Thread.run(Thread.java:595)
                        Caused by: java.lang.NullPointerException
                             at com.sun.messaging.jmq.jmsserver.core.Session$ackEntry.acknowledged(Session.java:514)
                             ... 10 more
                        This is a queue with 7 producer which produce a heartbeat every five seconds and 1 consumer reading all those heartbeats every five seconds.

                        We have had so many problems with this, I'm guessing we must be doing something wrong?
                        • 9. Re: Destination Q:mq.sys.dmq could not be found in the store
                          ak
                          These appear to be caused by bugs 6886390 (fixed in 4.4.1 and later releases) and 6886391 (fixed in 4.4x and later releases)