1 Reply Latest reply on Jun 17, 2019 11:56 PM by Tom B-Oracle

    Error Enlisting GXA Transaction

    1001072

      Our production Weblogic 10.3.6 server handles a lot of traffic, mostly during daytime hours. Each evening, between 8 pm and 3 am - when there is very little traffic - we go thru patches where we can't connect to our JMS topic. Here's the exception:

       

      weblogic.jms.common.JMSException: weblogic.messaging.kernel.KernelException: Error enlisting GXA transaction

      at com.ntst.careconnect.jms.commons.CareConnectJMSManager.publishMessage(CareConnectJMSManager.java:116)

      at com.ntst.careconnect.handlers.CareConnectJMSConcurrencyManager.publishMessage(CareConnectJMSConcurrencyManager.java:105)

      at com.ntst.careconnect.hieadapter.HIESubmitDocumentService.addNewRequestToQueue(HIESubmitDocumentService.java:96)

      at com.ntst.careconnect.hieadapter.HIESubmitDocumentService.submitCCDToHIE(HIESubmitDocumentService.java:604)

      at com.ntst.careconnect.hieadapter.HIEAdapterProxy.submitCCDToHIE(HIEAdapterProxy.java:321)

      at com.ntst.careconnect.sdk.HIEEventHandler.handleHIECareRecordChangedEvent(HIEEventHandler.java:286)

      at com.ntst.careconnect.sdk.HIEEventHandler.handleMessage(HIEEventHandler.java:368)

      at com.ntst.careconnect.cf.CareConnectSDKReceiveMessageProcessor.handleMessage(CareConnectSDKReceiveMessageProcessor.java:115)

      at sun.reflect.GeneratedMethodAccessor290.invoke(Unknown Source)

      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

      at java.lang.reflect.Method.invoke(Method.java:606)

      at com.bea.core.repackaged.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:310)

      at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)

      at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)

      at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)

      at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)

      at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)

      at com.oracle.pitchfork.spi.MethodInvocationVisitorImpl.visit(MethodInvocationVisitorImpl.java:34)

      at weblogic.ejb.container.injection.EnvironmentInterceptorCallbackImpl.callback(EnvironmentInterceptorCallbackImpl.java:54)

      at com.oracle.pitchfork.spi.EnvironmentInterceptor.invoke(EnvironmentInterceptor.java:42)

      at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)

      at com.bea.core.repackaged.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)

      at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)

      at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)

      at com.bea.core.repackaged.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)

      at com.bea.core.repackaged.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)

      at com.bea.core.repackaged.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)

      at com.sun.proxy.$Proxy256.handleMessage(Unknown Source)

      at com.ntst.careconnect.cf.CareConnectSDKReceiveMessageProcessor_a9hxcw_ReceiveMessageProcessorImpl.__WL_invoke(Unknown Source)

      at weblogic.ejb.container.internal.SessionRemoteMethodInvoker.invoke(SessionRemoteMethodInvoker.java:40)

      at com.ntst.careconnect.cf.CareConnectSDKReceiveMessageProcessor_a9hxcw_ReceiveMessageProcessorImpl.handleMessage(Unknown Source)

      at sun.reflect.GeneratedMethodAccessor289.invoke(Unknown Source)

      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

      at java.lang.reflect.Method.invoke(Method.java:606)

      at weblogic.ejb.container.internal.RemoteBusinessIntfProxy.invoke(RemoteBusinessIntfProxy.java:90)

      at com.sun.proxy.$Proxy86.handleMessage(Unknown Source)

      at com.ntst.cfsdk.web.AbstractCFSDKServlet.doPost(AbstractCFSDKServlet.java:123)

      at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)

      at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)

      at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)

      at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:301)

      at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:184)

      at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3750)

      at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3714)

      at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)

      at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)

      at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2283)

      at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2182)

      at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1499)

      at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)

      at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

      Caused by: com.ntst.jms.commons.exception.JMSMessageProcessingException: weblogic.jms.common.JMSException: weblogic.messaging.kernel.KernelException: Error enlisting GXA transaction

      at com.ntst.jms.commons.proxy.JMSPublisherProxy.sendWithDelay(JMSPublisherProxy.java:87)

      at com.ntst.careconnect.jms.commons.CareConnectJMSManager.publishMessage(CareConnectJMSManager.java:109)

      ... 51 more

      Caused by: weblogic.jms.common.JMSException: weblogic.messaging.kernel.KernelException: Error enlisting GXA transaction

      at weblogic.jms.dispatcher.DispatcherAdapter.convertToJMSExceptionAndThrow(DispatcherAdapter.java:110)

      at weblogic.jms.dispatcher.DispatcherAdapter.dispatchSyncTran(DispatcherAdapter.java:53)

      at weblogic.jms.client.JMSProducer.toFEProducer(JMSProducer.java:1289)

      at weblogic.jms.client.JMSProducer.deliveryInternal(JMSProducer.java:796)

      at weblogic.jms.client.JMSProducer.sendInternal(JMSProducer.java:541)

      at weblogic.jms.client.JMSProducer.sendWithListener(JMSProducer.java:425)

      at weblogic.jms.client.JMSProducer.send(JMSProducer.java:410)

      at weblogic.jms.client.WLProducerImpl.send(WLProducerImpl.java:1001)

      at weblogic.jms.client.WLProducerImpl.publish(WLProducerImpl.java:130)

      at com.ntst.jms.commons.proxy.JMSPublisherProxy.sendWithDelay(JMSPublisherProxy.java:84)

      ... 52 more

      Caused by: weblogic.jms.common.JMSException: weblogic.messaging.kernel.KernelException: Error enlisting GXA transaction

      at weblogic.jms.dispatcher.Request.handleThrowable(Request.java:87)

      at weblogic.jms.dispatcher.Request.getResult(Request.java:52)

      at weblogic.messaging.dispatcher.Request.wrappedFiniteStateMachine(Request.java:1124)

      at weblogic.messaging.dispatcher.DispatcherImpl.syncRequest(DispatcherImpl.java:185)

      at weblogic.messaging.dispatcher.DispatcherImpl.dispatchSyncTran(DispatcherImpl.java:238)

      at weblogic.jms.dispatcher.DispatcherAdapter.dispatchSyncTran(DispatcherAdapter.java:51)

      ... 60 more

      Caused by: weblogic.jms.common.JMSException: weblogic.messaging.kernel.KernelException: Error enlisting GXA transaction

      at weblogic.jms.dispatcher.Request.handleThrowable(Request.java:87)

      at weblogic.jms.dispatcher.Request.getResult(Request.java:52)

      at weblogic.jms.frontend.FEProducer.sendSingleDestination(FEProducer.java:1247)

      at weblogic.jms.frontend.FEProducer.send(FEProducer.java:1452)

      at weblogic.jms.frontend.FEProducer.invoke(FEProducer.java:1511)

      at weblogic.messaging.dispatcher.Request.wrappedFiniteStateMachine(Request.java:961)

      ... 63 more

      Caused by: weblogic.jms.common.JMSException: weblogic.messaging.kernel.KernelException: Error enlisting GXA transaction

      at weblogic.jms.dispatcher.Request.handleThrowable(Request.java:87)

      at weblogic.jms.dispatcher.Request.getResult(Request.java:52)

      at weblogic.messaging.dispatcher.Request.wrappedFiniteStateMachine(Request.java:1124)

      at weblogic.messaging.dispatcher.DispatcherImpl.dispatchAsyncInternal(DispatcherImpl.java:140)

      at weblogic.messaging.dispatcher.DispatcherImpl.dispatchAsync(DispatcherImpl.java:116)

      at weblogic.messaging.dispatcher.Request.dispatchAsync(Request.java:1303)

      at weblogic.jms.dispatcher.Request.dispatchAsync(Request.java:96)

      at weblogic.jms.frontend.FEProducer.doDispatch(FEProducer.java:933)

      at weblogic.jms.frontend.FEProducer.sendSingleDestination(FEProducer.java:1236)

      ... 66 more

      Caused by: weblogic.jms.common.JMSException: weblogic.messaging.kernel.KernelException: Error enlisting GXA transaction

      at weblogic.jms.backend.BEDestinationImpl.sendIssueMessage(BEDestinationImpl.java:1931)

      at weblogic.jms.backend.BEDestinationImpl.send(BEDestinationImpl.java:2119)

      at weblogic.jms.backend.BEDestinationImpl.wrappedSend(BEDestinationImpl.java:2062)

      at weblogic.jms.backend.BEDestinationImpl.invoke(BEDestinationImpl.java:1550)

      at weblogic.messaging.dispatcher.Request.wrappedFiniteStateMachine(Request.java:961)

      ... 72 more

      Caused by: weblogic.messaging.kernel.KernelException: Error enlisting GXA transaction

      at weblogic.messaging.kernel.internal.KernelImpl.getGXATransaction(KernelImpl.java:572)

      at weblogic.messaging.kernel.internal.QueueImpl.send(QueueImpl.java:329)

      at weblogic.messaging.kernel.internal.TopicImpl.send(TopicImpl.java:155)

      at weblogic.jms.backend.BEDestinationImpl.sendIssueMessage(BEDestinationImpl.java:1884)

      ... 76 more

      Caused by: weblogic.store.gxa.GXAException: Transaction already rolled back, or the transaction manager has timed out this resource.

      at weblogic.store.gxa.internal.GXAResourceImpl.enlist(GXAResourceImpl.java:523)

      at weblogic.messaging.kernel.internal.KernelImpl.getGXATransaction(KernelImpl.java:570)

      ... 79 more

      Caused by: weblogic.transaction.RollbackException: Transaction timed out after 91 seconds

      BEA1-01CD0ABBAC62C29F2042

      at weblogic.transaction.internal.TransactionImpl.throwRollbackException(TransactionImpl.java:1922)

      at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:523)

      at weblogic.transaction.internal.ServerTransactionImpl.enlistResource(ServerTransactionImpl.java:495)

      at weblogic.store.gxa.internal.GXAResourceImpl.enlist(GXAResourceImpl.java:442)

      ... 80 more

      Caused by: weblogic.transaction.internal.TimedOutException: Transaction timed out after 91 seconds

      BEA1-01CD0ABBAC62C29F2042

      at weblogic.transaction.internal.ServerTransactionImpl.wakeUp(ServerTransactionImpl.java:1804)

      at weblogic.transaction.internal.ServerTransactionManagerImpl.processTimedOutTransactions(ServerTransactionManagerImpl.java:1714)

      at weblogic.transaction.internal.TransactionManagerImpl.wakeUp(TransactionManagerImpl.java:1988)

      at weblogic.transaction.internal.ServerTransactionManagerImpl.wakeUp(ServerTransactionManagerImpl.java:1624)

      at weblogic.transaction.internal.WLSTimer.timerExpired(WLSTimer.java:35)

      at weblogic.timers.internal.TimerImpl.run(TimerImpl.java:284)

      at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:550)

      ... 2 more

        • 1. Re: Error Enlisting GXA Transaction
          Tom B-Oracle

          It looks like the message never made it to the topic. 

           

          Here's what seems to be happening based on the servlet invoke and 'Transaction timed out after 91 seconds' in your stack trace:

           

          1. App servlet is invoked
          2. Something is starting a JTA transaction, or, in especially advanced designs, is re-using a cached unfinished transaction.  Most likely, the tx is started as a side effect of the app servlet invoke - for example in your code via a call to 'tx.begin()' or some such or via an EJB annotation.
          3. Transaction times out after 90 seconds
          4. App calls JMS 'publish'
          5. The publish internally tries to enlist in current JTA transaction
          6. The internal enlist fails and throws because the JTA transaction has already timed out and rolled back

           

          It's possible you have app work that naturally takes more than 90 seconds and your tx timeout simply needs to be increased - someone already increased it from 30 (the default) to 90...

           

          HTH,

          Tom