6 Replies Latest reply: May 30, 2012 1:57 PM by Ahz RSS

    Too many connections open for WebLogic integration with Oracle AQ JMS

    Ahz
      Is there any document explaining number of database connections required per queue and/or connection factories? I have configured a foreign server with destinations and connection factories to integrate WebLogic with AQ JMS. I have 16 destinations (14 Queues and 2 Topics) and two connection factories (one XAQueueConnectionFactory and one XATopicConnectionFactory) configured. I also have 16 MDBs, one for each destination, configured with max-beans-in-free-pool set to 1. Everything works fine. However, when the application server is started I could see that there are 68 database connections open. That seems too high. When using OC4J we had max-connections set to 30 and it was working fine. Why are so many connections open?
        • 1. Re: Too many connections open for WebLogic integration with Oracle AQ JMS
          153820
          The configuration of AQ JMS queues and connection factories of AQ foreign server itself is "static", which does not trigger any usage of db connection. For example, if you configure an AQ foreign server with a WLS datasource, but there is nobody actually using the foreign server's resources, then there is no db connection from the datasource is used for AQ. But the datasource itself may keep some free db connections in its pool. Depending on its configuration, the number of the connections varies. So if you check the database side, you may see some db connections open. But none of these connections are used.

          MDBs using AQ JMS needs to hold db connection to poll messages. In your case, you have 16 MDBs with max 1 bean instance for each MDB, then each MDB can hold one db connection from the datasource, which makes total 16 db connections that are actively used for polling.

          If there are other components in your application that accesses AQ resources, for example, a servlet sends messages to a queue, the sending session needs a db connection.

          Another usage of db connection is to look up AQ JMS destinations. When a foreign AQ destination is looked up, foreign server internally grabs a connection and does a round trip to AQ server, then immediately releases this connection back to datasource pool. So this is hardly noticeable due to it's short lived. Looking up a connection factory does not need a db connection.

          I suggest you do the following:
          First check the WLS datasource definition and monitor its usage, to see how many db connections are actively used, and how many are just free in the pool. Then check whether there are other components using the same datasource.

          Then check whether there are other components that are accessing AQ resources, in addition to the MDBs.

          Do these MDBs themselves create AQ JMS sessions and do sending? If yes, the MDBs need more db connections, in addition to those 16 connections used for polling.

          Edited by: Longxing on May 10, 2012 4:44 PM
          • 2. Re: Too many connections open for WebLogic integration with Oracle AQ JMS
            Ahz
            That was my assumption. I have 16 MDBs, therefore I was expecting 16+ connections to be open but not 60+. I got a dumpPool and the following three stack traces are repeated many times. From what I see they are all initiated by WebLogic as part of MDB initialization. I need to know whether or not this is normal.


            ####<10-May-2012 10:26:51 o'clock AM EDT> <Info> <JDBC> <WL091133> <DefaultServer> <Thread-39> <weblogic> <> <740451edb30f8977:-72f20ccb:137332c725b:-7ffc-0000000000000015> <1336660011920> <BEA-001519> <Id : "autoCommit=true,enabled=true,isXA=true,isJTS=false,vendorID=0,connUsed=true,doInit=false,'null',destroyed=false,poolname=MyPoolName,appname=null,
            moduleName=null,connectTime=172,dirtyIsolationLevel=false,initialIsolationLevel=2,infected=false,lastSuccessfulConnectionUse=1336593524878,
            secondsToTrustAnIdlePoolConnection=10,currentUser=java.lang.Exception
                 at weblogic.jdbc.common.internal.ConnectionEnv.setup(ConnectionEnv.java:318)
                 at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:344)
                 at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:322)
                 at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:438)
                 at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:317)
                 at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:93)
                 at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:61)
                 at weblogic.jdbc.jta.DataSource.getXAConnectionFromPool(DataSource.java:1670)
                 at weblogic.jdbc.jta.DataSource.refreshXAConnAndEnlist(DataSource.java:1468)
                 at weblogic.jdbc.jta.DataSource.getConnection(DataSource.java:439)
                 at weblogic.jdbc.jta.DataSource.connect(DataSource.java:396)
                 at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:355)
                 at oracle.jms.AQjmsDBConnMgr.getConnection(AQjmsDBConnMgr.java:566)
                 at oracle.jms.AQjmsConnection.createQueueSession(AQjmsConnection.java:646)
                 at weblogic.ejb.container.internal.JMSConnectionPoller.setUpQueueSessions(JMSConnectionPoller.java:1519)
                 at weblogic.ejb.container.internal.JMSConnectionPoller.createJMSConnection(JMSConnectionPoller.java:2047)
                 at weblogic.ejb.container.internal.JMSConnectionPoller.connect(JMSConnectionPoller.java:787)
                 at weblogic.ejb.container.internal.MDConnectionManager.startConnectionPolling(MDConnectionManager.java:243)
                 at weblogic.ejb.container.manager.MessageDrivenManager.start(MessageDrivenManager.java:584)
                 at weblogic.ejb.container.deployer.MessageDrivenBeanInfoImpl.mdManagerStart(MessageDrivenBeanInfoImpl.java:1334)
                 at weblogic.ejb.container.deployer.MessageDrivenBeanInfoImpl.deployMessageDrivenBeans(MessageDrivenBeanInfoImpl.java:1314)
                 at weblogic.ejb.container.deployer.EJBDeployer.deployMessageDrivenBeans(EJBDeployer.java:1669)
                 at weblogic.ejb.container.deployer.EJBDeployer.deployMessageDrivenBeansUsingModuleCL(EJBDeployer.java:1446)
                 at weblogic.ejb.container.deployer.MDBServiceImpl.startService(MDBServiceImpl.java:69)
                 at weblogic.server.ServiceActivator.start(ServiceActivator.java:96)
                 at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
                 at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
                 at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
            ,currentThread=Thread[[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)',5,Pooled Threads],lastUser=null,currentError=null,currentErrorTimestamp=null,JDBC4Runtime=true,supportStatementPoolable=true,needRestoreClientInfo=false,defaultClientInfo={},supportIsValid=true">



            ####<10-May-2012 10:26:51 o'clock AM EDT> <Info> <JDBC> <WL091133> <DefaultServer> <Thread-39> <weblogic> <> <740451edb30f8977:-72f20ccb:137332c725b:-7ffc-0000000000000015> <1336660011936> <BEA-001519> <Id : "autoCommit=true,enabled=true,isXA=true,isJTS=false,vendorID=0,connUsed=true,doInit=false,'null',destroyed=false,poolname=MyPoolName,appname=null,
            moduleName=null,connectTime=156,dirtyIsolationLevel=false,initialIsolationLevel=2,infected=false,lastSuccessfulConnectionUse=1336593516957,
            secondsToTrustAnIdlePoolConnection=10,currentUser=java.lang.Exception
                 at weblogic.jdbc.common.internal.ConnectionEnv.setup(ConnectionEnv.java:318)
                 at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:344)
                 at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:322)
                 at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:438)
                 at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:317)
                 at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:93)
                 at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:61)
                 at weblogic.jdbc.jta.DataSource.getXAConnectionFromPool(DataSource.java:1670)
                 at weblogic.jdbc.jta.DataSource.refreshXAConnAndEnlist(DataSource.java:1468)
                 at weblogic.jdbc.jta.DataSource.getConnection(DataSource.java:439)
                 at weblogic.jdbc.jta.DataSource.connect(DataSource.java:396)
                 at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:355)
                 at oracle.jms.AQjmsDBConnMgr.getConnection(AQjmsDBConnMgr.java:566)
                 at oracle.jms.AQjmsDBConnMgr.<init>(AQjmsDBConnMgr.java:399)
                 at oracle.jms.AQjmsXAConnection.<init>(AQjmsXAConnection.java:112)
                 at oracle.jms.AQjmsXAQueueConnectionFactory.createAllXAConnection(AQjmsXAQueueConnectionFactory.java:320)
                 at oracle.jms.AQjmsXAQueueConnectionFactory.createXAQueueConnection(AQjmsXAQueueConnectionFactory.java:198)
                 at weblogic.ejb.container.internal.JMSConnectionPoller.getXAConnection(JMSConnectionPoller.java:418)
                 at weblogic.ejb.container.internal.JMSConnectionPoller.createJMSConnection(JMSConnectionPoller.java:1959)
                 at weblogic.ejb.container.internal.JMSConnectionPoller.connect(JMSConnectionPoller.java:787)
                 at weblogic.ejb.container.internal.MDConnectionManager.startConnectionPolling(MDConnectionManager.java:243)
                 at weblogic.ejb.container.manager.MessageDrivenManager.start(MessageDrivenManager.java:584)
                 at weblogic.ejb.container.deployer.MessageDrivenBeanInfoImpl.mdManagerStart(MessageDrivenBeanInfoImpl.java:1334)
                 at weblogic.ejb.container.deployer.MessageDrivenBeanInfoImpl.deployMessageDrivenBeans(MessageDrivenBeanInfoImpl.java:1314)
                 at weblogic.ejb.container.deployer.EJBDeployer.deployMessageDrivenBeans(EJBDeployer.java:1669)
                 at weblogic.ejb.container.deployer.EJBDeployer.deployMessageDrivenBeansUsingModuleCL(EJBDeployer.java:1446)
                 at weblogic.ejb.container.deployer.MDBServiceImpl.startService(MDBServiceImpl.java:69)
                 at weblogic.server.ServiceActivator.start(ServiceActivator.java:96)
                 at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
                 at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
                 at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
            ,currentThread=Thread[[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)',5,Pooled Threads],lastUser=null,currentError=null,currentErrorTimestamp=null,JDBC4Runtime=true,supportStatementPoolable=true,needRestoreClientInfo=false,defaultClientInfo={},supportIsValid=true">



            ####<10-May-2012 10:26:52 o'clock AM EDT> <Info> <JDBC> <WL091133> <DefaultServer> <Thread-39> <weblogic> <> <740451edb30f8977:-72f20ccb:137332c725b:-7ffc-0000000000000015> <1336660012373> <BEA-001519> <Id : "autoCommit=true,enabled=true,isXA=true,isJTS=false,vendorID=0,connUsed=true,doInit=false,'null',destroyed=false,poolname=MyPoolName,appname=null,
            moduleName=null,connectTime=156,dirtyIsolationLevel=false,initialIsolationLevel=2,infected=false,lastSuccessfulConnectionUse=1336659788679,
            secondsToTrustAnIdlePoolConnection=10,currentUser=java.lang.Exception
                 at weblogic.jdbc.common.internal.ConnectionEnv.setup(ConnectionEnv.java:318)
                 at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:344)
                 at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:322)
                 at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:438)
                 at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:317)
                 at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:93)
                 at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:61)
                 at weblogic.jdbc.jta.DataSource.getXAConnectionFromPool(DataSource.java:1670)
                 at weblogic.jdbc.jta.DataSource.refreshXAConnAndEnlist(DataSource.java:1468)
                 at weblogic.jdbc.wrapper.JTAConnection.getXAConn(JTAConnection.java:199)
                 at weblogic.jdbc.wrapper.JTAConnection.checkConnection(JTAConnection.java:68)
                 at weblogic.jdbc.wrapper.JTAConnection.getTypeMap(JTAConnection.java:564)
                 at oracle.jms.AQjmsConsumer.dequeue(AQjmsConsumer.java:2053)
                 at oracle.jms.AQjmsConsumer.dequeue(AQjmsConsumer.java:1918)
                 at oracle.jms.AQjmsConsumer.receiveForListener(AQjmsConsumer.java:556)
                 at oracle.jms.AQjmsListenerWorker.dispatchOneMsg(AQjmsListenerWorker.java:287)
                 at oracle.jms.AQjmsListenerWorker.run(AQjmsListenerWorker.java:134)
                 at java.lang.Thread.run(Thread.java:619)
            ,currentThread=Thread[Thread-43,5,Pooled Threads],lastUser=null,currentError=null,currentErrorTimestamp=null,JDBC4Runtime=true,supportStatementPoolable=true,needRestoreClientInfo=false,defaultClientInfo={},supportIsValid=true">

            Edited by: 839959 on May 17, 2012 12:36 PM
            • 3. Re: Too many connections open for WebLogic integration with Oracle AQ JMS
              153820
              The pasted partial stack traces does not give out much info. It may be normal. You cannot tell from it, whether it's one MDB is getting three connections, or three different MDBs with each one getting one connection. Since you have 16 MDBs with each has max 1 bean, can you please count how many these stack traces are? If there are more than 16, it may be an issue.

              Can you please clarify how you observed the 60+ connections? It's not clear from the posting whether these 60+ connections are all for MDBs, or MDBs plus other components, or MDBs plus other components and idle connections in datasource pool. There are many possibilities here.

              There are other things you can try, for example,
              * monitor each MDB to see how many instances it has
              * limit the datasource pool to 16 connections, and assuming the MDBs can grab those connections at start-up, and check exceptions from other components, and see whether there are some other components are competing for the connections.
              • 4. Re: Too many connections open for WebLogic integration with Oracle AQ JMS
                Ahz
                Longxing, thanks for your reply. I observed the number of connections through WebLogic console. Active Connections Current Count attribute of the datasource. As you suspected, some MDBs were not configured properly. I fixed the problem and I now have 32 connections open for 16 MDBs. If I remove one MDB then the "active connection current count" goes down to 30. It seems that it is opening two connections per MDB. One for the message listener to poll the queue and another one for the exception listener to check for exceptions like resource outage??? See the stack trace below. Is that normal behaviour?

                As you suggested, I limited the datasource pool to 20 and the application failed to deploy with "No resources currently available in pool ... to allocate to applications, please increase the size of the pool and retry.."

                <23-May-2012 3:51:31 o'clock PM EDT> <Warning> <JDBC> <BEA-001153> <Forcibly releasing inactive connection "[weblogic.jdbc.wrapper.JTAConnection_weblogic_jdbc_wrapper_XAConnection_oracle_jdbc_driver_LogicalConnection-MyPoolName-1345, oracle.jdbc.driver.LogicalConnection@b782da]" back into the connection pool "MyPoolName", currently reserved by: java.lang.Exception
                     at weblogic.jdbc.common.internal.ConnectionEnv.setup(ConnectionEnv.java:318)
                     at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:344)
                     at weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolImpl.java:322)
                     at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:438)
                     at weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:317)
                     at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:93)
                     at weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolManager.java:61)
                     at weblogic.jdbc.jta.DataSource.getXAConnectionFromPool(DataSource.java:1670)
                     at weblogic.jdbc.jta.DataSource.refreshXAConnAndEnlist(DataSource.java:1468)
                     at weblogic.jdbc.jta.DataSource.getConnection(DataSource.java:439)
                     at weblogic.jdbc.jta.DataSource.connect(DataSource.java:396)
                     at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:355)
                     at oracle.jms.AQjmsDBConnMgr.getConnection(AQjmsDBConnMgr.java:566)
                     at oracle.jms.AQjmsConnection.setExceptionListener(AQjmsConnection.java:416)
                     at weblogic.ejb.container.internal.JMSConnectionPoller.setUpQueueSessions(JMSConnectionPoller.java:1695)
                     at weblogic.ejb.container.internal.JMSConnectionPoller.createJMSConnection(JMSConnectionPoller.java:2047)
                     at weblogic.ejb.container.internal.JMSConnectionPoller.connect(JMSConnectionPoller.java:787)
                     at weblogic.ejb.container.internal.MDConnectionManager.timerExpired(MDConnectionManager.java:172)
                     at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:273)
                     at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
                     at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
                     at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
                • 5. Re: Too many connections open for WebLogic integration with Oracle AQ JMS
                  153820
                  From the stack trace pasted, the particular version of the WebLogic that you use, utilizes AQ JMS message listener for message polling, and sets a ExceptionListener on JMS connection. (I think your MDB is a non-transactional MDB). The AQ JMS message listener uses a db connection of message polling, and the AQ JMS ExceptionListener implementation internally uses a db connection to ping the database periodically. So your MDB uses two connections in your setup.
                  • 6. Re: Too many connections open for WebLogic integration with Oracle AQ JMS
                    Ahz
                    You are right. When the MDB's transaction type is Bean managed or NOT_SUPPORTED then two dedicated connections are reserved for that MDB. When it is set to REQUIRED then no dedicated connection is reserved. Thank you.