6 Replies Latest reply on Jul 7, 2011 2:23 PM by 832350

    Time Out Error while waiting for response from DB Procedure

    832350
      Hi Gurus,

      We are encountering a problem in our production environment. The system is implemented using AIA foundation pack 2.5 on SOA suite 10.1.3.4.

      We have a BPEL process A which calls an ESB Service which inturn calls BPEL Process B. In process B, we have a DB procedure call which waits for a response from
      a DB procedure. The procedure doesn't reply on time and Process B remains in waiting state to get the response from DB Procedure wherein Process A errors out by showing as "Timed Out Error".

      This issue is intermittent and we have already increased transaction-time outs in transaction-manager.xml to 7200 and ejb-orion-jar.xml to 3600.

      When we encountered this problem, we found out that there are too many connections open and when we bounced the server, everything was restored to nornal but as it is a production env. we can't do it over and over again.

      We have 2 nodes each having max connections as 100 and min. as 0.

      Is there a limit to max no. of connections or can we do something in DB side to ensure that it doesn't happen again ?

      Please suggest.

      Thanks,
      Vikas Manchanda
        • 1. Re: Time Out Error while waiting for response from DB Procedure
          Anuj Dwivedi--Oracle
          We have 2 nodes each having max connections as 100 and min. as 0.
          100 is a considerably large value in terms of DB connection. Do you really need these many open connections at a time? I would suggest you to reduce the number to somewhere 30 - 40 and see how it affects the processing. Contact your DBA as well to know about the DB capacity and it's performance. Check for message size and network interruption as well.

          Regards,
          Anuj
          • 2. Re: Time Out Error while waiting for response from DB Procedure
            832350
            Hi Anuj,

            Thanks for your reply but help me in understanding this. The max limit of 100 would be reached only when 1 thread is waiting for response from DB and there are multiple requests but this problem has appeared also when we have ample no. of connections available.

            Do you think anything else could be the reason. How would reducing the no. of connections help.??

            Regards,
            Vikas Manchanda
            • 3. Re: Time Out Error while waiting for response from DB Procedure
              Anuj Dwivedi--Oracle
              Hi Vikas,

              It may be that number of connections open reached to a large value which database could not fulfill and hence timeout errors. That's why I mentioned to check with DBA.

              One more check. Check the data-sources.xml and see if you find any setting with name "abandoned-connection-timeout" for these connection pool? If yes, then it's a known issue.

              Regards,
              Anuj
              • 4. Re: Time Out Error while waiting for response from DB Procedure
                832350
                Hi Anuj,

                I don't think it is a problem with connection reaching to max numbers because this issue is coming on very intermittent basis.we don't have any other processes using
                the same connection pool and this issue is coming even when there is no load on the server. This is recent trace from the production environment. Also i don't have any thing called "abandoned-connection-pool" in my data-sources.xml.

                <2011-07-07 13:09:16,101> <ERROR> <default.collaxa.cube> <BaseCubeSessionBean::logError> Error while invoking bean "delivery": Waiting for response has timed out. The conversation id is null. Please check the process instance for detail.
                com.oracle.bpel.client.delivery.ReceiveTimeOutException: Waiting for response has timed out. The conversation id is null. Please check the process instance for detail.
                at com.collaxa.cube.ejb.impl.DeliveryBean.request(DeliveryBean.java:109)
                at sun.reflect.GeneratedMethodAccessor113.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:592)
                at com.evermind.server.ejb.interceptor.joinpoint.EJBJoinPointImpl.invoke(EJBJoinPointImpl.java:35)
                at com.evermind.server.ejb.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:119)
                at com.evermind.server.ejb.interceptor.system.DMSInterceptor.invoke(DMSInterceptor.java:52)
                at com.evermind.server.ejb.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:119)
                at com.evermind.server.ejb.interceptor.system.JAASInterceptor$1.run(JAASInterceptor.java:31)
                at com.evermind.server.ThreadState.runAs(ThreadState.java:693)
                at com.evermind.server.ejb.interceptor.system.JAASInterceptor.invoke(JAASInterceptor.java:34)
                at com.evermind.server.ejb.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:119)
                at com.evermind.server.ejb.interceptor.system.TxRequiredInterceptor.invoke(TxRequiredInterceptor.java:50)
                at com.evermind.server.ejb.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:119)
                at com.evermind.server.ejb.interceptor.system.DMSInterceptor.invoke(DMSInterceptor.java:52)
                at com.evermind.server.ejb.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:119)
                at com.evermind.server.ejb.InvocationContextPool.invoke(InvocationContextPool.java:55)
                at com.evermind.server.ejb.StatelessSessionEJBObject.OC4J_invokeMethod(StatelessSessionEJBObject.java:87)
                at DeliveryBean_RemoteProxy_4bin6i8.request(Unknown Source)
                at com.collaxa.cube.ws.soap.oc4j.SOAPRequestProvider.processNormalOperation(SOAPRequestProvider.java:451)
                at com.collaxa.cube.ws.soap.oc4j.SOAPRequestProvider.processBPELMessage(SOAPRequestProvider.java:274)
                at com.collaxa.cube.ws.soap.oc4j.SOAPRequestProvider.processMessage(SOAPRequestProvider.java:120)
                at oracle.j2ee.ws.server.provider.ProviderProcessor.doEndpointProcessing(ProviderProcessor.java:956)
                at oracle.j2ee.ws.server.WebServiceProcessor.invokeEndpointImplementation(WebServiceProcessor.java:349)
                at oracle.j2ee.ws.server.provider.ProviderProcessor.doRequestProcessing(ProviderProcessor.java:466)
                at oracle.j2ee.ws.server.WebServiceProcessor.processRequest(WebServiceProcessor.java:114)
                at oracle.j2ee.ws.server.WebServiceProcessor.doService(WebServiceProcessor.java:96)
                at oracle.j2ee.ws.server.WebServiceServlet.doPost(WebServiceServlet.java:194)
                at javax.servlet.http.HttpServlet.service(HttpServlet.java:763)
                at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
                at com.evermind.server.http.ResourceFilterChain.doFilter(ResourceFilterChain.java:64)
                at oracle.security.jazn.oc4j.JAZNFilter$1.run(JAZNFilter.java:400)
                at java.security.AccessController.doPrivileged(Native Method)
                at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)
                at oracle.security.jazn.oc4j.JAZNFilter.doFilter(JAZNFilter.java:414)
                at com.evermind.server.http.ServletRequestDispatcher.invoke(ServletRequestDispatcher.java:623)
                at com.evermind.server.http.ServletRequestDispatcher.forwardInternal(ServletRequestDispatcher.java:370)
                at com.evermind.server.http.HttpRequestHandler.doProcessRequest(HttpRequestHandler.java:871)
                at com.evermind.server.http.HttpRequestHandler.processRequest(HttpRequestHandler.java:453)
                at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:313)
                at com.evermind.server.http.AJPRequestHandler.run(AJPRequestHandler.java:199)
                at oracle.oc4j.network.ServerSocketReadHandler$SafeRunnable.run(ServerSocketReadHandler.java:260)
                at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:303)
                at java.lang.Thread.run(Thread.java:595)
                Caused by: com.oracle.bpel.client.delivery.ReceiveTimeOutException: Waiting for response has timed out. The conversation id is null. Please check the process instance for detail.
                at com.collaxa.cube.engine.delivery.DeliveryHandler.initialRequestAnyType(DeliveryHandler.java:576)
                at com.collaxa.cube.engine.delivery.DeliveryHandler.initialRequest(DeliveryHandler.java:465)
                at com.collaxa.cube.engine.delivery.DeliveryHandler.request(DeliveryHandler.java:134)
                at com.collaxa.cube.ejb.impl.DeliveryBean.request(DeliveryBean.java)

                Please suggest.

                Thanks,
                Vikas Manchanda
                • 5. Re: Time Out Error while waiting for response from DB Procedure
                  Anuj Dwivedi--Oracle
                  Being a production issue, raise a SR with support. May be timeouts needs to be tuned as well as DB.

                  Regards,
                  Anuj
                  • 6. Re: Time Out Error while waiting for response from DB Procedure
                    832350
                    Thanks Anuj, I was considering that as a last option. Now that we have tried everything, I would raise an SR with Oracle Support.

                    Regards,
                    Vikas Manchanda