5 Replies Latest reply: Jan 28, 2011 10:32 AM by ak RSS

    Stale DB connections?

      Hello Forum,

      We have a pair of MQ 4.4 brokers running on Redhat 5, configured to use our Oracle 10G RAC cluster as their back end. The brokers are configured using the HA connection string provided by our Oracle DBA that works well for our other server processes. Normally everything works great. We noticed a problem when we took down one of our RAC nodes for some updates. Instead of trucking on like I had expected them to, both MQ brokers froze. Hard. I had to 'kill -9' the brokers to get them to restart. Once they restarted they grabed the other DB node without having to touch the configuration. I went back and reviewed the logs around the time this all went down and ran across this:

      [19/Jan/2011:20:45:53 EST] WARNING [B3044]: Internal Error:
      com.sun.messaging.jmq.jmsserver.util.BrokerException: [B4221]: Failed to load all HA cluster information from store
      at com.sun.messaging.jmq.jmsserver.persist.jdbc.BrokerDAOImpl.getAllBrokerInfos(BrokerDAOImpl.java:1077)
      at com.sun.messaging.jmq.jmsserver.persist.jdbc.JDBCStore.getAllBrokerInfos(JDBCStore.java:2771)
      at com.sun.messaging.jmq.jmsserver.persist.Store.getAllBrokerInfos(Store.java:1323)
      at com.sun.messaging.jmq.jmsserver.cluster.ha.HAClusterManagerImpl$HAMap.updateHAMapForState(HAClusterManagerImpl.java:1483)
      at com.sun.messaging.jmq.jmsserver.cluster.ha.HAClusterManagerImpl$HAMap.updateHAMap(HAClusterManagerImpl.java:1470)
      at com.sun.messaging.jmq.jmsserver.cluster.ha.HAClusterManagerImpl.getKnownBrokers(HAClusterManagerImpl.java:458)
      at com.sun.messaging.jmq.jmsserver.data.handlers.HelloHandler.handle(HelloHandler.java:350)
      at com.sun.messaging.jmq.jmsserver.data.PacketRouter.handleMessage(PacketRouter.java:181)
      at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.readData(IMQIPConnection.java:1355)
      at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.process(IMQIPConnection.java:542)
      at com.sun.messaging.jmq.jmsserver.service.imq.OperationRunnable.process(OperationRunnable.java:170)
      at com.sun.messaging.jmq.jmsserver.util.pool.BasicRunnable.run(BasicRunnable.java:493)
      at java.lang.Thread.run(Thread.java:619)
      FT JOIN MQSES41Ccluster1 sTbl ON bTbl.ID = sTbl.BROKER_ID AND sTbl.IS_CURRENT = 1]: Closed Connection
      at com.sun.messaging.jmq.jmsserver.persist.jdbc.DBManager.wrapSQLException(DBManager.java:894)
      at com.sun.messaging.jmq.jmsserver.persist.jdbc.BrokerDAOImpl.getAllBrokerInfos(BrokerDAOImpl.java:1072)
      ... 12 more

      There were a lot of other similar errors. Different queries, but the same root problem: "Closed Connection." It looks like connections closed when the RAC node went offline and instead of throwing them out and getting new ones, MQ is using the same connections until I shut it down and restart. Is this accurate? Is there a setting I should change to correct this behavior?

      Thanks for taking the time to review this and I appreciate any help you can offer.

        • 1. Re: Stale DB connections?
          What is the version of the broker shown in the broker log at startup ?
          • 2. Re: Stale DB connections?
            Sun GlassFish(tm) Message Queue 4.4
            Sun Microsystems, Inc.
            Version: 4.4 Update 1 (Build 7-b)
            Compile: Wed Nov 18 23:21:05 PST 2009

            Copyright (c) 2009 Sun Microsystems, Inc. All rights reserved. Use is
            subject to license terms.
            Java Runtime: 1.6.0_16 Sun Microsystems Inc. /usr/java/jdk1.6.0_16/jre
            [19/Jan/2011:12:35:58 EST] IMQ_HOME=/opt/mq2/mq
            [19/Jan/2011:12:35:58 EST] IMQ_VARHOME=/opt/mq2/var/mq
            [19/Jan/2011:12:35:58 EST] Linux 2.6.18-164.11.1.el5 i386 appsvr0208 (4 cpu) root
            [19/Jan/2011:12:35:58 EST] Java Heap Size: max=504896k, current=193920k
            [19/Jan/2011:12:35:58 EST] Arguments: -bgnd -silent
            [19/Jan/2011:12:35:58 EST] [B1202]: Broker broker02 is running in HA mode as part of the cluster1 cluster
            [19/Jan/2011:12:35:58 EST] [B1004]: Starting the portmapper service using tcp [ 7676, 50, * ] with min threads 1 and max threads of 1
            [19/Jan/2011:12:35:58 EST] [B1060]: Loading persistent data...
            [19/Jan/2011:12:35:58 EST] imq.persist.jdbc.oracle.driver=oracle.jdbc.pool.OracleConnectionPoolDataSource
            [19/Jan/2011:12:35:58 EST] imq.persist.jdbc.oracle.property.implicitCachingEnabled=true
            [19/Jan/2011:12:35:58 EST] imq.persist.jdbc.oracle.property.maxStatements=25
            [19/Jan/2011:12:35:58 EST] Oracle, Oracle Database 10g Enterprise Edition Release - 64bit Production
            With the Partitioning, Real Application Clusters, OLAP, Data Mining
            and Real Application Testing options,
            [19/Jan/2011:12:35:58 EST] Using plugged-in persistent store:
            database connection url=jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=***)(PORT=***))(ADDRESS=(PROTOCOL=TCP)(HOST=***)(PO
            database user=mquser
            • 3. Re: Stale DB connections?
              Please upgrade to MQ 4.4u2 (lastest release is 4.4u2p1) which has JDBC connection validation added (6831996).
              • 4. Re: Stale DB connections?
                We upgraded both brokers over the weekend:

                Open Message Queue 4.4
                Version: 4.4 Update 2 (Build 5-a)
                Compile: Fri May 14 23:24:45 PDT 2010

                Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.

                Today we had an unexpected node failure on our Oracle cluster and we ended up with exactly the same results.


                • 5. Re: Stale DB connections?
                  On getting a JDBC connection from MQ DB connection pool, in enhanced cluster mode, the connection will be validated by default and be destroyed if invalid, then it expects to get a good connection from the underline JDBC driver. Please make sure to set the properties for Oracle driver (imq.persist.jdbc.oracle.property.xxx) for proper faliover for Oracle cluster. Please provide the complete broker logs and broker thread dump.

                  MQ 4.5 latest build (build 27) has fixed the following bugs in MQ DB connection pool: 7014780, 7014791, which likely apply to your case.