1 Reply Latest reply on Jun 1, 2008 8:34 PM by 666705

    May message be corrupted?

    666705
      Hello guys!
                
                I really need som comment for this problem, and I need it fast.
                We are suppose to deliver a system upgrade into our production environment tomorrow!
                Yesterday we experience a very odd behavior in our production_like_environment that we have not experience before.
                All other test has been fine!
                
                First, please take a look at our system architecture.
                The system look
                
                System A
                Weblogic 8.1.5
                In this we got an MDB registred for listening for messages at a queue at MQ called MQQUEUE_A.
                It finds the queue at MQ through a .bindings jndi
                
                MQ-server A
                This is where the queue MQQUEUE_A resides.
                
                MQ-broker routes messages
                Routes messages between queue MQQUEUE_A and queue MQQUEUE_B.
                
                Mq-server B
                This is where the queue MQQUEUE_B resides.
                This server is our connection point to our collaborate partners.
                
                Firewall
                
                Mq-server C
                This is where the queue MQQUEUE_C resides.
                This server is the server at our collaborate partners.
                
                So the line looks like:
                MQQUEUE_C -> (Firewall) -> MQQUEUE_B -> MQ-broker -> MQQUEUE_A -> MDB.
                
                At thursday night everything was working fine.
                
                At friday morning our collaborate partners posted two messages at MQQUEUE_C as normal, just for a simple last test.
                But they did ot arrive!!!
                
                After some investigation our collaborate partners find out that the channel between MQQUEUE_C -> MQQUEUE_B was in retrying mode. (The two messages was still at the MQQUEUE_C).
                
                Our Firwall-department confirmed that Mq-server C and Mq-server B was talking with each other so nothing was wrong in the Firewall.
                
                Our collaborate partners tried to restart the channel. Voila the messages was sent away.
                ... but they did not go all the way to our MDB.
                
                After some more investigation we saw that the queue depth was 1 at MQQUEUE_A. (But 2 messages was sent!!!)
                In our serverlog at our Weblogic we now was getting logging, every approx. 20 sec, that said something like: "The MDB has reconnect to MQQUEUE_A". In our MQ-server A we was getting logging saying something like "This error might have to due because of connection error".
                
                We tried to restart our Weblogic. During the time the Weblogic was down the queue depth was 2 at MQQUEUE_A. This looks like then that the two messages was luckely sent all the way to our queue MQQUEUE_A!! We gues that the first message was held in the RAM then at MQ-server A.
                
                When our Weblogic was restarted we again get the same error message in the logs, both at Weblogic and and in MQ-server A.
                We then tried to temporary remove the 2 messages and sent a new message from our collaborate partner . That message vent all the way through to our MDB!!!
                
                We then put back the 2 messages, which gave us the same error message again in log files.
                
                Our conclusion is that the 2 messages was in some way corrupted! But we really need to find out why and how to prevent this.
                
                In Weblogic we are pretty sure that the MDB itself was never getting the messages. If I understand it correct our weblogic first get the message from MQ and then hands them over to our MDB. Below is our config for our MDB.
                
                
                ejb-jar.xml:
                <message-driven>
                     <ejb-name>MqToJms</ejb-name>
                     <ejb-class>com.ourcompany.MqToJmsMDB</ejb-class>
                     <transaction-type>Bean</transaction-type>
                     <acknowledge-mode>Auto-acknowledge</acknowledge-mode>
                     <message-driven-destination>
                          <destination-type>javax.jms.Queue</destination-type>
                          <subscription-durability>Durable</subscription-durability>
                     </message-driven-destination>
                </message-driven>
                
                
                weblogic-ejb-jar.xml
                <weblogic-enterprise-bean>
                <ejb-name>MqToJms</ejb-name>
                     <message-driven-descriptor>
                     <pool>
                          <max-beans-in-free-pool>1</max-beans-in-free-pool>
                          <initial-beans-in-free-pool>1</initial-beans-in-free-pool>
                     </pool>
                     <destination-jndi-name>MQQUEUE_A</destination-jndi-name>
                     <initial-context-factory>com.sun.jndi.fscontext.RefFSContextFactory</initial-context-factory>
                     <provider-url>file:path/to/jndi</provider-url>
                     <connection-factory-jndi-name>queue.connection.factory</connection-factory-jndi-name>
                     <jms-polling-interval-seconds>10</jms-polling-interval-seconds>
                </message-driven-descriptor>
                <reference-descriptor>
                </reference-descriptor>
                </weblogic-enterprise-bean>
                
                in folder file:path/to/jndi we got an .bindings file describing the jndi-tree perhaps the error is in that??
                
                If any one got any idea of why we get this mysterious error we really would love to get that.
                So please do not hasitate to guess!!
                Best regards
                Fredrik
        • 1. Re: May message be corrupted?
          666705
          Hi Fredrik,
                    
                    I don't know what the problem was, but it seems you're using MQ from end-to-end -- no WebLogic JMS or even WebLogic JTA, so I'm not sure how WebLogic itself could have made a difference in your scenario.
                    
                    The WebLogic MDB is actually pretty much just a standard MQ client as far as MQ is concerned. In fact, it is pretty much all MQ code running on the WL server except for a thin wrapper. The MQ server pushes the messages to an asynchronous MQ client running on the WebLogic server, which in turn asynchronously invokes an "onMessage()" inside the MDB container, the MDB container then simply calls the "onMessage()" on the MDB application.
                    
                    I recommend sending the full stack traces of all of your exceptions, plus the full error messages, to IBM for analysis. Feel free to post them here as well - especially if the error messages don't look like they were generated by MQ.
                    
                    Tom