11 Replies Latest reply on Nov 4, 2015 10:57 AM by 3051442

    XAER_RMERR: 'A resource manager error has occured in the transaction branch' on a clustered environment

    3051442

      Hi,

      We have a problem with Resource Manager on an OSB 11g / WebLogic 10.3.6 installation. We have a clustered deployment on two machines (one hosts Admin Server and one of the managed servers, the second with just one managed server). The problem occurs only on the second managed server (on the second machine).

       

      Any ideas on what could be the cause and how to fix it?

       

      The problem logs:

      Unexpected exception while enlisting XAConnection java.sql.SQLException: XA error: XAResource.XAER_RMERR start() failed on resource 'jdbc/myds_mydomain': XAER_RMERR : A resource manager error has occured in the transaction branch

      oracle.jdbc.xa.OracleXAException

              at oracle.jdbc.xa.OracleXAResource.checkError(OracleXAResource.java:1616)

              at oracle.jdbc.xa.client.OracleXAResource.start(OracleXAResource.java:336)

       

      Stack trace:

      javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection

              at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1387)

              at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1310)

              at org.hibernate.ejb.QueryImpl.getSingleResult(QueryImpl.java:316)

              at com.mysema.query.jpa.impl.AbstractJPAQuery.getSingleResult(AbstractJPAQuery.java:219)

              at com.mysema.query.jpa.impl.AbstractJPAQuery.uniqueResult(AbstractJPAQuery.java:295)

              at com.mysema.query.jpa.impl.AbstractJPAQuery.uniqueResult(AbstractJPAQuery.java:281)

      (...)

      Caused by: org.hibernate.exception.GenericJDBCException: Could not open connection

              at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:54)

              at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:125)

              at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:110)

              at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:221)

              at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:157)

              at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.connection(StatementPreparerImpl.java:56)

              at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:161)

              at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:182)

              at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:159)

              at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1859)

              at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1836)

              at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1816)

              at org.hibernate.loader.Loader.doQuery(Loader.java:900)

              at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:342)

              at org.hibernate.loader.Loader.doList(Loader.java:2526)

              at org.hibernate.loader.Loader.doList(Loader.java:2512)

              at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2342)

              at org.hibernate.loader.Loader.list(Loader.java:2337)

              at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:495)

              at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:357)

              at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:195)

              at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1269)

              at org.hibernate.internal.QueryImpl.list(QueryImpl.java:101)

              at org.hibernate.ejb.QueryImpl.getSingleResult(QueryImpl.java:287)

              ... 281 more

      Caused by: java.sql.SQLException: Unexpected exception while enlisting XAConnection java.sql.SQLException: XA error: XAResource.XAER_RMERR start() failed on resource 'jdbc/myds_mydomain': XAER_RMERR : A resource manager error has occured in the transaction branch

      oracle.jdbc.xa.OracleXAException

              at oracle.jdbc.xa.OracleXAResource.checkError(OracleXAResource.java:1616)

              at oracle.jdbc.xa.client.OracleXAResource.start(OracleXAResource.java:336)

              at weblogic.jdbc.wrapper.VendorXAResource.start(VendorXAResource.java:50)

              at weblogic.jdbc.jta.DataSource.start(DataSource.java:811)

              at weblogic.transaction.internal.XAServerResourceInfo.start(XAServerResourceInfo.java:1250)

              at weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1183)

              at weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceInfo.java:286)

              at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:561)

              at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:488)

              at weblogic.jdbc.jta.DataSource.enlist(DataSource.java:1706)

              at weblogic.jdbc.jta.DataSource.refreshXAConnAndEnlist(DataSource.java:1610)

              at weblogic.jdbc.jta.DataSource.getConnectionInternal(DataSource.java:499)

              at weblogic.jdbc.jta.DataSource.getConnection(DataSource.java:483)

              at weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:526)

              at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:512)

              at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:505)

              at org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider.getConnection(InjectedDataSourceConnectionProvider.java:70)

              at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:292)

      (...)

       

      Best regards.

        • 1. Re: XAER_RMERR: 'A resource manager error has occured in the transaction branch' on a clustered environment
          handat

          Can your second wls server connect to the database via the jdbc resource? Is that the build in hibernate or a custom one that's compaining about the connection problem?

          • 2. Re: XAER_RMERR: 'A resource manager error has occured in the transaction branch' on a clustered environment
            Vijaya_OCP_Moderator-Oracle

            There is a XA Transaction initiated by your application demarcated across various transactional resources. One of them is the Database.

             

            The below error indicates some connection issues with database

             

            =>org.hibernate.exception.GenericJDBCException: Could not open connection

             

            =>oracle.jdbc.xa.OracleXAException

                    at oracle.jdbc.xa.OracleXAResource.checkError(OracleXAResource.java:1616)

             

            1. Try connecting to the Database via External Client like SQL Plus and verify if the connection succeeds.

            2. To check the issue to one of the managed server, turn on the below JTA debug flags on the problematic Server and upload debug logs to this thread.

             

            DebugJTAXA

            DebugJTAJDBC

            DebugJTALifecycle

             

            Thanks,

            Vijaya

            1 person found this helpful
            • 3. Re: XAER_RMERR: 'A resource manager error has occured in the transaction branch' on a clustered environment
              3051442

              Yes, both servers can connect to the database using this JDBC resource. We use built in Hibernate, version 4.2.6.Final.

              • 4. Re: XAER_RMERR: 'A resource manager error has occured in the transaction branch' on a clustered environment
                3051442

                Vijaya_OCP_Moderator-Oracle

                1. DB is up and running and accepts connections.

                 

                2. Logs with flags you suggested are below. What caught my attention is:

                 

                a.

                XAResource.start(bea1-011321006f9e00aee9c4-6a6462632f6578746f726544535f746573746f7362, XAResource.TMNOFLAGS), xaRes:oracle.jdbc.driver.T4CXAResource@1d3c0249-oracle.jdbc.driver.T4CXAConnection@663352b2

                On other environment, where everything runs OK, the xaRes contains only T4CXAResource. This one has T4CXAConnection.

                 

                b.

                (CoordinatorURL=myserver2+10.1.1.2:8011+mydomain+t3+, XAResources={eis/tibjms/Topic, WLStore_mydomain_FileStore_auto_2, WLStore_mydomain__WLS_myserver2, eis/fioranomq/Topic, WSATGatewayRM_myserver2_mydomain, eis/tibjms/Queue, jdbc/mydsNonXA_mydomain, eis/sunmq/Queue, jdbc/myds_mydomain, eis/aqjms/Topic, eis/pramati/Queue, jdbc/bulk_mydomain, eis/tibjmsDirect/Queue, eis/AQ/aqSample, jdbc/bulkxa_mydomain, WLStore_mydomain_WseeFileStore_auto_2, eis/activemq/Queue, eis/aqjms/Queue, eis/tibjmsDirect/Topic, eis/jbossmq/Queue, eis/wls/Topic, eis/Apps/Apps, eis/webspheremq/Queue, eis/wls/Queue},NonXAResources={})

                Exactly the fact that NonXAResources list is empty while I have two NonXA resources (jdbc/mydsNonXA_mydomain and jdbc/bulk_mydomain). The both have driver class oracle.jdbc.OracleDriver and 'Supports Global Transaction' checked.

                 

                Logs:

                ####<Oct 28, 2015 11:05:17 AM CET> <Debug> <JTAXA> <mymachine2> <myserver2> <[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <BEA1-011321006F9E00AEE9C4> <> <1446026717039> <BEA-000000> <ResourceDescriptor[jdbc/myds_mydomain]: getOrCreate gets rd: name = jdbc/myds_mydomain

                resourceType = 2

                registered = true

                scUrls = myserver2+10.1.1.2:8011+mydomain+t3+

                xar = jdbc/myds

                healthy = true

                lastAliveTimeMillis = 1446026708829

                numActiveRequests = 0

                >

                ####<Oct 28, 2015 11:05:17 AM CET> <Debug> <JTAXA> <mymachine2> <myserver2> <[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <BEA1-011321006F9E00AEE9C4> <> <1446026717039> <BEA-000000> <BEA1-011321006F9E00AEE9C4: null: XA.start(rm=jdbc/myds_mydomain, xar=jdbc/myds, flags=TMNOFLAGS)>

                ####<Oct 28, 2015 11:05:17 AM CET> <Debug> <JTAXA> <mymachine2> <myserver2> <[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <BEA1-011321006F9E00AEE9C4> <> <1446026717039> <BEA-000000> <Resource 'jdbc/myds_mydomain' setTransactionTimeout(1800) returned true>

                ####<Oct 28, 2015 11:05:17 AM CET> <Debug> <JTAXA> <mymachine2> <myserver2> <[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <> <1446026717039> <BEA-000000> <startResourceUse, Number of active requests:1, last alive time:0 ms ago.>

                ####<Oct 28, 2015 11:05:17 AM CET> <Debug> <JTAJDBC> <mymachine2> <myserver2> <[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <> <1446026717039> <BEA-000000> < -tx:null- -pool:jdbc/myds- oracleTMNOMIGRATE about to return XAResource.TMNOFLAGS>

                ####<Oct 28, 2015 11:05:17 AM CET> <Debug> <JTAJDBC> <mymachine2> <myserver2> <[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <> <1446026717039> <BEA-000000> < -tx:null- -pool:jdbc/myds- > XAResource.start(bea1-011321006f9e00aee9c4-6a6462632f6578746f726544535f746573746f7362, XAResource.TMNOFLAGS), xaRes:oracle.jdbc.driver.T4CXAResource@1d3c0249-oracle.jdbc.driver.T4CXAConnection@663352b2>

                ####<Oct 28, 2015 11:05:17 AM CET> <Debug> <JTAJDBC> <mymachine2> <myserver2> <[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <> <1446026717039> <BEA-000000> <[[oracle.jdbc.driver.T4CXAConnection@663352b2, owner=jdbc/myds, rmConn=oracle.jdbc.driver.LogicalConnection@25c249b]] setAutoCommit(false)>

                ####<Oct 28, 2015 11:05:17 AM CET> <Debug> <JTAJDBC> <mymachine2> <myserver2> <[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <> <1446026717039> <BEA-000000> <[[oracle.jdbc.driver.T4CXAConnection@663352b2, owner=jdbc/myds, rmConn=oracle.jdbc.driver.LogicalConnection@25c249b]] setAutoCommit returns>

                ####<Oct 28, 2015 11:05:17 AM CET> <Debug> <JTAJDBC> <mymachine2> <myserver2> <[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <0000L2huATjCgo55zRDCiW1MC9RM000093> <1446026717041> <BEA-000000> < -tx:null- -pool:jdbc/myds- < XA error: XAResource.XAER_RMERR

                oracle.jdbc.xa.OracleXAException

                    at oracle.jdbc.xa.OracleXAResource.checkError(OracleXAResource.java:1616)

                    at oracle.jdbc.xa.client.OracleXAResource.start(OracleXAResource.java:336)

                    at weblogic.jdbc.wrapper.VendorXAResource.start(VendorXAResource.java:50)

                    at weblogic.jdbc.jta.DataSource.start(DataSource.java:811)

                    at weblogic.transaction.internal.XAServerResourceInfo.start(XAServerResourceInfo.java:1250)

                    at weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1183)

                    at weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceInfo.java:286)

                    at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:561)

                    at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:488)

                    at weblogic.jdbc.jta.DataSource.enlist(DataSource.java:1706)

                    at weblogic.jdbc.jta.DataSource.refreshXAConnAndEnlist(DataSource.java:1610)

                    at weblogic.jdbc.jta.DataSource.getConnectionInternal(DataSource.java:499)

                    at weblogic.jdbc.jta.DataSource.getConnection(DataSource.java:483)

                    at weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:526)

                    at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:512)

                    at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:505)

                    at org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider.getConnection(InjectedDataSourceConnectionProvider.java:70)

                    at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:292)

                    at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:214)

                    at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:157)

                    at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.connection(StatementPreparerImpl.java:56)

                    at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:161)

                    at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:182)

                    at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:159)

                    at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1859)

                    at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1836)

                    at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1816)

                    at org.hibernate.loader.Loader.doQuery(Loader.java:900)

                    at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:342)

                    at org.hibernate.loader.Loader.doList(Loader.java:2526)

                    at org.hibernate.loader.Loader.doList(Loader.java:2512)

                    at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2342)

                    at org.hibernate.loader.Loader.list(Loader.java:2337)

                    at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:495)

                    at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:357)

                    at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:195)

                    at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1269)

                    at org.hibernate.internal.QueryImpl.list(QueryImpl.java:101)

                    at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:268)

                    at com.mysema.query.jpa.impl.AbstractJPAQuery.getResultList(AbstractJPAQuery.java:196)

                    at com.mysema.query.jpa.impl.AbstractJPAQuery.list(AbstractJPAQuery.java:242)

                 

                ...

                 

                ####<Oct 28, 2015 11:05:17 AM CET> <Debug> <JTAJDBC> <mymachine2> <myserver2> <[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <0000L2huATjCgo55zRDCiW1MC9RM000093> <1446026717043> <BEA-000000> <[[oracle.jdbc.driver.T4CXAConnection@663352b2, owner=null, rmConn=oracle.jdbc.driver.LogicalConnection@25c249b]] clearWarnings()>

                ####<Oct 28, 2015 11:05:17 AM CET> <Debug> <JTAJDBC> <mymachine2> <myserver2> <[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <0000L2huATjCgo55zRDCiW1MC9RM000093> <1446026717043> <BEA-000000> <[[oracle.jdbc.driver.T4CXAConnection@663352b2, owner=null, rmConn=oracle.jdbc.driver.LogicalConnection@25c249b]] clearWarnings returns>

                ####<Oct 28, 2015 11:05:17 AM CET> <Debug> <JTAXA> <mymachine2> <myserver2> <[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <BEA1-011321006F9E00AEE9C4> <0000L2huATjCgo55zRDCiW1MC9RM000093> <1446026717043> <BEA-000000> <BEA1-011321006F9E00AEE9C4: null: XA.start FAILED (rm=jdbc/myds_mydomain, xar=jdbc/myds

                OracleError = 24784

                OracleSQLError = 0

                XAError = -3

                oracle.jdbc.xa.OracleXAException

                    at oracle.jdbc.xa.OracleXAResource.checkError(OracleXAResource.java:1616)

                    at oracle.jdbc.xa.client.OracleXAResource.start(OracleXAResource.java:336)

                    at weblogic.jdbc.wrapper.VendorXAResource.start(VendorXAResource.java:50)

                    at weblogic.jdbc.jta.DataSource.start(DataSource.java:811)

                    at weblogic.transaction.internal.XAServerResourceInfo.start(XAServerResourceInfo.java:1250)

                    at weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1183)

                    at weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceInfo.java:286)

                    at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:561)

                    at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:488)

                    at weblogic.jdbc.jta.DataSource.enlist(DataSource.java:1706)

                    at weblogic.jdbc.jta.DataSource.refreshXAConnAndEnlist(DataSource.java:1610)

                    at weblogic.jdbc.jta.DataSource.getConnectionInternal(DataSource.java:499)

                    at weblogic.jdbc.jta.DataSource.getConnection(DataSource.java:483)

                    at weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:526)

                    at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:512)

                    at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:505)

                    at org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider.getConnection(InjectedDataSourceConnectionProvider.java:70)

                    at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:292)

                    at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:214)

                    at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:157)

                    at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.connection(StatementPreparerImpl.java:56)

                    at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:161)

                    at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:182)

                    at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:159)

                    at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1859)

                    at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1836)

                    at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1816)

                    at org.hibernate.loader.Loader.doQuery(Loader.java:900)

                    at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:342)

                    at org.hibernate.loader.Loader.doList(Loader.java:2526)

                    at org.hibernate.loader.Loader.doList(Loader.java:2512)

                    at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2342)

                    at org.hibernate.loader.Loader.list(Loader.java:2337)

                    at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:495)

                    at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:357)

                    at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:195)

                    at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1269)

                    at org.hibernate.internal.QueryImpl.list(QueryImpl.java:101)

                    at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:268)

                    at com.mysema.query.jpa.impl.AbstractJPAQuery.getResultList(AbstractJPAQuery.java:196)

                    at com.mysema.query.jpa.impl.AbstractJPAQuery.list(AbstractJPAQuery.java:242)

                 

                ...

                 

                ####<Oct 28, 2015 11:05:17 AM CET> <Debug> <JTAXA> <mymachine2> <myserver2> <[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <BEA1-011321006F9E00AEE9C4> <0000L2huATjCgo55zRDCiW1MC9RM000093> <1446026717043> <BEA-000000> <endResourceUse, Number of active requests:0>

                ####<Oct 28, 2015 11:05:17 AM CET> <Debug> <JTAJDBC> <mymachine2> <myserver2> <[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <BEA1-011321006F9E00AEE9C4> <0000L2huATjCgo55zRDCiW1MC9RM000093> <1446026717044> <BEA-000000> < -tx:Xid=BEA1-011321006F9E00AEE9C4(853747657),Status=Active,numRepliesOwedMe=0,numRepliesOwedOthers=0,seconds since begin=0,seconds left=1800,activeThread=Thread[[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)',5,Pooled Threads],SCInfo[mydomain+myserver2]=(state=active),local properties=({weblogic.jdbc.jta.jdbc/myds=[ No XAConnection is attached to this TxInfo ]}),OwnerTransactionManager=ServerTM[ServerCoordinatorDescriptor=(CoordinatorURL=myserver2+10.1.1.2:8011+mydomain+t3+, XAResources={eis/tibjms/Topic, WLStore_mydomain_FileStore_auto_2, WLStore_mydomain__WLS_myserver2, eis/fioranomq/Topic, WSATGatewayRM_myserver2_mydomain, eis/tibjms/Queue, jdbc/mydsNonXA_mydomain, eis/sunmq/Queue, jdbc/myds_mydomain, eis/aqjms/Topic, eis/pramati/Queue, jdbc/bulk_mydomain, eis/tibjmsDirect/Queue, eis/AQ/aqSample, jdbc/bulkxa_mydomain, WLStore_mydomain_WseeFileStore_auto_2, eis/activemq/Queue, eis/aqjms/Queue, eis/tibjmsDirect/Topic, eis/jbossmq/Queue, eis/wls/Topic, eis/Apps/Apps, eis/webspheremq/Queue, eis/wls/Queue},NonXAResources={})],CoordinatorURL=myserver2+10.1.1.2:8011+mydomain+t3+)- -pool:jdbc/myds- < SystemEx error

                javax.transaction.SystemException: start() failed on resource 'jdbc/myds_mydomain': XAER_RMERR : A resource manager error has occured in the transaction branch

                oracle.jdbc.xa.OracleXAException

                    at oracle.jdbc.xa.OracleXAResource.checkError(OracleXAResource.java:1616)

                    at oracle.jdbc.xa.client.OracleXAResource.start(OracleXAResource.java:336)

                    at weblogic.jdbc.wrapper.VendorXAResource.start(VendorXAResource.java:50)

                    at weblogic.jdbc.jta.DataSource.start(DataSource.java:811)

                    at weblogic.transaction.internal.XAServerResourceInfo.start(XAServerResourceInfo.java:1250)

                    at weblogic.transaction.internal.XAServerResourceInfo.xaStart(XAServerResourceInfo.java:1183)

                    at weblogic.transaction.internal.XAServerResourceInfo.enlist(XAServerResourceInfo.java:286)

                    at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:561)

                    at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:488)

                    at weblogic.jdbc.jta.DataSource.enlist(DataSource.java:1706)

                    at weblogic.jdbc.jta.DataSource.refreshXAConnAndEnlist(DataSource.java:1610)

                    at weblogic.jdbc.jta.DataSource.getConnectionInternal(DataSource.java:499)

                    at weblogic.jdbc.jta.DataSource.getConnection(DataSource.java:483)

                    at weblogic.jdbc.common.internal.RmiDataSource.getConnectionInternal(RmiDataSource.java:526)

                    at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:512)

                    at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:505)

                    at org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider.getConnection(InjectedDataSourceConnectionProvider.java:70)

                    at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:292)

                    at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:214)

                    at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:157)

                    at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.connection(StatementPreparerImpl.java:56)

                    at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:161)

                    at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:182)

                    at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:159)

                    at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1859)

                    at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1836)

                    at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1816)

                    at org.hibernate.loader.Loader.doQuery(Loader.java:900)

                    at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:342)

                    at org.hibernate.loader.Loader.doList(Loader.java:2526)

                    at org.hibernate.loader.Loader.doList(Loader.java:2512)

                    at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2342)

                    at org.hibernate.loader.Loader.list(Loader.java:2337)

                    at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:495)

                    at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:357)

                    at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:195)

                    at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1269)

                    at org.hibernate.internal.QueryImpl.list(QueryImpl.java:101)

                    at org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:268)

                    at com.mysema.query.jpa.impl.AbstractJPAQuery.getResultList(AbstractJPAQuery.java:196)

                    at com.mysema.query.jpa.impl.AbstractJPAQuery.list(AbstractJPAQuery.java:242)

                 

                ...

                 

                ####<Oct 28, 2015 11:05:17 AM CET> <Debug> <JTAJDBC> <mymachine2> <myserver2> <[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <BEA1-011321006F9E00AEE9C4> <0000L2huATjCgo55zRDCiW1MC9RM000093> <1446026717047> <BEA-000000> < -tx:Xid=BEA1-011321006F9E00AEE9C4(853747657),Status=Active,numRepliesOwedMe=0,numRepliesOwedOthers=0,seconds since begin=0,seconds left=1800,activeThread=Thread[[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)',5,Pooled Threads],SCInfo[mydomain+myserver2]=(state=active),local properties=({weblogic.jdbc.jta.jdbc/myds=[ No XAConnection is attached to this TxInfo ]}),OwnerTransactionManager=ServerTM[ServerCoordinatorDescriptor=(CoordinatorURL=myserver2+10.1.1.2:8011+mydomain+t3+, XAResources={eis/tibjms/Topic, WLStore_mydomain_FileStore_auto_2, WLStore_mydomain__WLS_myserver2, eis/fioranomq/Topic, WSATGatewayRM_myserver2_mydomain, eis/tibjms/Queue, jdbc/mydsNonXA_mydomain, eis/sunmq/Queue, jdbc/myds_mydomain, eis/aqjms/Topic, eis/pramati/Queue, jdbc/bulk_mydomain, eis/tibjmsDirect/Queue, eis/AQ/aqSample, jdbc/bulkxa_mydomain, WLStore_mydomain_WseeFileStore_auto_2, eis/activemq/Queue, eis/aqjms/Queue, eis/tibjmsDirect/Topic, eis/jbossmq/Queue, eis/wls/Topic, eis/Apps/Apps, eis/webspheremq/Queue, eis/wls/Queue},NonXAResources={})],CoordinatorURL=myserver2+10.1.1.2:8011+mydomain+t3+)- -pool:jdbc/myds- < DataSource.refreshXAConnAndEnlist txTimedOutException :false>

                ####<Oct 28, 2015 11:05:17 AM CET> <Info> <EJB> <mymachine2> <myserver2> <[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <BEA1-011321006F9E00AEE9C4> <0000L2huATjCgo55zRDCiW1MC9RM000093> <1446026717050> <BEA-010227> <EJB Exception occurred during invocation from home or business: weblogic.ejb.container.internal.StatelessEJBHomeImpl@73297096 threw exception: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection>

                • 5. Re: XAER_RMERR: 'A resource manager error has occured in the transaction branch' on a clustered environment
                  957742

                  Hi,

                   

                  Could you please retest the service by enabling the below Transaction properties for the JDBC connection pool. ( Configuration -> Transaction tab )

                   

                  • Rollback Local Tx Upon Connection Close
                  • Keep XA Connection Until Transaction Complete
                  • Keep Connection After Local Transaction
                  • XASetTransactionTimeout
                  • 6. Re: XAER_RMERR: 'A resource manager error has occured in the transaction branch' on a clustered environment
                    3051442

                    I set Rollback Local Tx Upon Connection Close (all others were already set), but the problem still exists.

                    • 7. Re: Re: XAER_RMERR: 'A resource manager error has occured in the transaction branch' on a clustered environment
                      iRAV

                      Just for check.

                       

                      Did you run before setup XA resource:

                       

                      connect sys as sysddba
                      @$ORACLE_HOME/rdbms/admin/xaview.sql
                      grant select on v$xatrans$ to public;
                      grant select on pending_trans$ to public;
                      grant select on dba_2pc_pending to public;
                      grant execute on dbms_system to <user>;
                      
                      • 8. Re: Re: Re: XAER_RMERR: 'A resource manager error has occured in the transaction branch' on a clustered environment
                        3051442

                        Actually, we have run:

                         

                        @$ORACLE_HOME/rdbms/admin/xaview.sql

                        grant all on dba_2pc_pending to <user>;

                        grant all on dba_pending_transactions to <user>;

                        grant all on dba_2pc_neighbors to <user>;

                        grant select on v$xatrans$ to <user>;

                        grant select on pending_trans$ to <user>;

                        grant execute on dbms_system to <user>;

                        grant force any transaction to <user>;

                         

                        The one relevant difference is that you say to grant 3 privs to public, not the user. Is granting them to public necessary?

                        • 10. Re: XAER_RMERR: 'A resource manager error has occured in the transaction branch' on a clustered environment
                          3051442

                          We've found out what was the issue. One of our adapters had an XA data source configured, we changed it to non-XA some time ago since it is used for queries with DBLinks involved, but the configuration didn't propagate to the second server. That is why one of the servers kept having this problem. We're now checking why the configuration did not propagate correctly.

                          • 11. Re: XAER_RMERR: 'A resource manager error has occured in the transaction branch' on a clustered environment
                            3051442

                            The configuration didn't propagate even though we have Staging Mode set to stage. Any ideas what might be missing?