1 Reply Latest reply: Nov 13, 2010 4:17 PM by ak RSS

    Open MQ, Oracle failing to obtain new connection after multiple messages

    811392
      I am using:

      Open MQ 4.5, Oracle 11G RAC

      My Goal:

      Running a High Availability cluster

      Problem:

      After a successful setup and about 15 successful send/receive messages, Open MQ fails to obtain a new connection. It seems Open MQ isn't letting go of its past connections and gets stuck in an infinite loop of waiting to obtain new connections, which aren't released. I've played with the connection by adding the args:
      +"-Dimq.persist.jdbc.max_connections=<some larger number> -Dimq.persist.jdbc.min_connections=<some number less than 'some large number'>"+
      Note that using defaults prevents startup as the connection pool, heartbeat, etc. takes more connections than 5, which causes the app to hang and throw exceptions infinitely while waiting for an idle connection to free up.

      Exception from 'log.txt':

      ...
      [08/Nov/2010:14:18:05 PST] Creating new Producer 4250320722553763586 on Q:world for connection 4250320722553763584
      [08/Nov/2010:14:18:05 PST] Created new consumer [consumer:4250320722553803520, type=NONE] on destination Q:world with selector null
      [08/Nov/2010:14:18:05 PST] Attaching Consumer [consumer:4250320722553803520, type=NONE] to Session 4250320722553763585
      [08/Nov/2010:14:18:06 PST] ERROR Error in processing ack on 119-<my ip address hidden for security>(f4:33:86:c8:64:f9)-1840-1289254682299 for [consumer:4250320722553803520, type=AUTO_ACKNOWLEDGE]:
      com.sun.messaging.jmq.jmsserver.util.BrokerException: [B3026]: Failed to get connection to jdbc:oracle:thin:@<host:port:SID hidden for security>
           at com.sun.messaging.jmq.jmsserver.persist.jdbc.comm.CommDBManager.newConnection(CommDBManager.java:671)
           at com.sun.messaging.jmq.jmsserver.persist.jdbc.comm.DBConnectionPool.createConnection(DBConnectionPool.java:456)
           at com.sun.messaging.jmq.jmsserver.persist.jdbc.comm.DBConnectionPool.getConnection(DBConnectionPool.java:492)
           at com.sun.messaging.jmq.jmsserver.persist.jdbc.DBManager.getConnection(DBManager.java:303)
           at com.sun.messaging.jmq.jmsserver.persist.jdbc.comm.CommDBManager.getConnection(CommDBManager.java:692)
           at com.sun.messaging.jmq.jmsserver.persist.jdbc.MessageDAOImpl.hasMessageBeenAcked(MessageDAOImpl.java:1229)
           at com.sun.messaging.jmq.jmsserver.persist.jdbc.JDBCStore.hasMessageBeenAcked(JDBCStore.java:3604)
           at com.sun.messaging.jmq.jmsserver.core.PacketReference.acknowledged(PacketReference.java:1844)
           at com.sun.messaging.jmq.jmsserver.core.Session$ackEntry.acknowledged(Session.java:554)
           at com.sun.messaging.jmq.jmsserver.core.Session.ackMessage(Session.java:1401)
           at com.sun.messaging.jmq.jmsserver.core.Session.ackMessage(Session.java:1376)
           at com.sun.messaging.jmq.jmsserver.data.handlers.AckHandler.handleAcks(AckHandler.java:440)
           at com.sun.messaging.jmq.jmsserver.data.handlers.AckHandler.handle(AckHandler.java:252)
           at com.sun.messaging.jmq.jmsserver.data.PacketRouter.handleMessage(PacketRouter.java:188)
           at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.readData(IMQIPConnection.java:1362)
           at com.sun.messaging.jmq.jmsserver.service.imq.IMQIPConnection.process(IMQIPConnection.java:548)
           at com.sun.messaging.jmq.jmsserver.service.imq.OperationRunnable.process(OperationRunnable.java:176)
           at com.sun.messaging.jmq.jmsserver.util.pool.BasicRunnable.run(BasicRunnable.java:499)
           at java.lang.Thread.run(Thread.java:619)
      Caused by: java.sql.SQLRecoverableException: Io exception: Got minus one from a read call
           at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:101)
           at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:112)
           at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:173)
           at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:229)
           at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:458)
           at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:411)
           at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:490)
           at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:202)
           at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:33)
           at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:465)
           at oracle.jdbc.pool.OracleDataSource.getPhysicalConnection(OracleDataSource.java:275)
           at oracle.jdbc.pool.OracleDataSource.getConnection(OracleDataSource.java:206)
           at oracle.jdbc.pool.OracleConnectionPoolDataSource.getPhysicalConnection(OracleConnectionPoolDataSource.java:139)
           at oracle.jdbc.pool.OracleConnectionPoolDataSource.getPooledConnection(OracleConnectionPoolDataSource.java:88)
           at com.sun.messaging.jmq.jmsserver.persist.jdbc.comm.CommDBManager.newConnection(CommDBManager.java:651)
           ... 18 more

      Exception from test app running hello world example (open-mq-4-5-src/src/share/java/examples/helloworld/helloworldmessage) in loop':

      WARNING: [I500]: Caught JVM Exception: com.sun.messaging.jms.JMSException: [ACKNOWLEDGE_REPLY(25)] [C4036]: A broker error occurred. :[500] [B3026]: Failed to get connection to jdbc:oracle:thin:@<host:port:SID hidden for security> user=guest, broker=localhost:7676(1793)
      Exception in thread "main" com.sun.messaging.jms.JMSException: [ACKNOWLEDGE_REPLY(25)] [C4036]: A broker error occurred. :[500] [B3026]: Failed to get connection to jdbc:oracle:thin:@<host:port:SID hidden for security>user=guest, broker=localhost:7676(1793)
           at com.sun.messaging.jmq.jmsclient.ProtocolHandler.throwServerErrorException(ProtocolHandler.java:4076)
           at com.sun.messaging.jmq.jmsclient.ProtocolHandler.acknowledge(ProtocolHandler.java:2667)
           at com.sun.messaging.jmq.jmsclient.SessionImpl.doAcknowledge(SessionImpl.java:1471)
           at com.sun.messaging.jmq.jmsclient.SessionImpl.autoAcknowledge(SessionImpl.java:1327)
           at com.sun.messaging.jmq.jmsclient.SessionImpl.acknowledge(SessionImpl.java:1202)
           at com.sun.messaging.jmq.jmsclient.MessageConsumerImpl.receive(MessageConsumerImpl.java:498)
           at com.sun.messaging.jmq.jmsclient.MessageConsumerImpl.receive(MessageConsumerImpl.java:460)
           at HelloWorldMessage.main(HelloWorldMessage.java:172)
      Caused by: com.sun.messaging.jms.JMSException: [ACKNOWLEDGE_REPLY(25)] [C4036]: A broker error occurred. :[500] [B3026]: Failed to get connection to jdbc:oracle:thin:@<host:port:SID hidden for security> user=guest, broker=localhost:7676(1793)
           at com.sun.messaging.jmq.jmsclient.ProtocolHandler.throwServerErrorException(ProtocolHandler.java:4061)
           ... 7 more

      Note: I get similar results when I ran 4.4u2.
        • 1. Re: Open MQ, Oracle failing to obtain new connection after multiple messages
          ak
          Note that using defaults prevents startup as the connection pool, heartbeat, etc. takes more connections than 5
          The default setup should not prevent a HA broker startup - that is to get the broker to the state "<brokername>@<host>:<port>" ready as shown in broker log at startup. However if your application uses many concurrent JMS connections, then the default should be adjusted - it will not affect broker startup but affect broker processing application requests.

          Have you checked why the Oracle server throws "java.sql.SQLRecoverableException: Io exception: Got minus one from a read call" ?

          I'v tried running the open-mq-4-5-src/src/share/java/examples/helloworld/helloworldmessage in a loop with a HA broker using an Oracle server (not RAC), no problem. So I'd suggest first to see any Oracle RAC configuration settings could be affecting this.