1 2 Previous Next 20 Replies Latest reply on Feb 5, 2013 6:42 AM by 989148

    OSB problem posting >5MB body to a business service endpoint

    225953
      Hello,

      I have an https endpoint they results in a Broken pipe exception after running for about 10mins if I send a SOAP body greater than 5MB. If I send the same request to this endpoint using SOAPUI (no OSB involvement) it works fine and completes in <1min, it only fails through OSB. Also if the request is less than 5MB it works fine through OSB and again completes in <1min.

      I've tried calling a proxy service (through test console and SOAPUI) and the business service (through test console) and all fail with the same broken pipe exception. Below is the constant state of the thread during this 10min period.

      "[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=3 tid=0x02fa6400 nid=0x16 runnable [0xb19fd000..0xb19ffaf0]
      java.lang.Thread.State: RUNNABLE
           at java.net.SocketOutputStream.socketWrite0(Native Method)
           at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
           at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
           at com.certicom.io.OutputSSLIOStream.write(Unknown Source)
           at com.certicom.tls.record.WriteHandler.flushOutput(Unknown Source)
           at com.certicom.tls.record.WriteHandler.write(Unknown Source)
           at com.certicom.io.OutputSSLIOStreamWrapper.write(Unknown Source)
           at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
           - locked <0xe70af568> (a java.io.BufferedOutputStream)
           at weblogic.net.http.HttpOutputStream.write(HttpOutputStream.java:22)
           at weblogic.utils.io.UnsyncByteArrayOutputStream.writeTo(UnsyncByteArrayOutputStream.java:104)
           at weblogic.net.http.HttpURLConnection.writeRequests(HttpURLConnection.java:162)
           - locked <0xcefd08f8> (a weblogic.net.http.SOAPHttpsURLConnection)
           at weblogic.net.http.HttpURLConnection.writeRequestForAsyncResponse(HttpURLConnection.java:492)
           at weblogic.net.http.AsyncResponseHandler.writeRequestAndRegister(AsyncResponseHandler.java:190)
           at weblogic.net.http.AsyncResponseHandler.writeRequestAndRegister(AsyncResponseHandler.java:152)
           at com.bea.wli.sb.transports.http.HttpOutboundMessageContext.send(HttpOutboundMessageContext.java:313)
           at com.bea.wli.sb.transports.http.HttpTransportProvider.sendMessageAsync(HttpTransportProvider.java:564)
           at sun.reflect.GeneratedMethodAccessor310.invoke(Unknown Source)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
           at java.lang.reflect.Method.invoke(Method.java:597)
           at com.bea.wli.sb.transports.Util$1.invoke(Util.java:82)
      ............


      I am running OSB 10.3.0 on SPARC Solaris using Sun JVM.

      Any ideas/suggestions?


      Many thanks,

      Mike.
        • 1. Re: OSB problem posting >5MB body to a business service endpoint
          687626
          A quick search on google shows that broken pipe exception are thrown if the tcp connection is closed from some one other than the client. Could the network be the bottleneck..I think you are running OSB from the data center, while SOAPUI is deployed in your local env. Could the network path taken by the two being different be the reason for the difference you are observing ? Could there be any active firewalls between OSB and endpoint which stops connection after 10 mins ?
          • 2. Re: OSB problem posting >5MB body to a business service endpoint
            225953
            Hello,

            thank you for your response but I don't believe the network is the problem. I'm running the tests from the same network and go through the same firewalls. The other factor that makes me think it is NOT a network issue is that the request succeeds if the body is less than <5MB (a 4.5MB body will work absolutely fine).

            Thanks,

            Mike.
            • 3. Re: OSB problem posting >5MB body to a business service endpoint
              Eric Elzinga
              Can you try using the streaming option on the proxy service?
              I was able to send messages of like 50mb to http endpoints (with streaming though)

              It could be some timeout setting in wls.
              http://forums.adobe.com/thread/584822 (http post)
              http://stackoverflow.com/questions/1307154/weblogic-transaction-timeout-how-to-set-in-admin-console-in-weblogic-as-8-1 (jta)

              Can you play a bit with these kind of settings to see what they do for you?
              Any other stacktraces in the wls logging ?

              Edited by: Eric Elzinga (IT-Eye) on Mar 29, 2010 11:42 PM
              • 4. Re: OSB problem posting >5MB body to a business service endpoint
                225953
                Hello,

                I've tried content streaming but it made no difference. Maybe I've misunderstood content streaming but I thought you use this when body variable in the pipeline pair of a proxy is too big to manipulate in memory. In my scenario this is not a problem (I can report on the body fine just before I route to the business service) the only issue occurs when I call the business service with a request greater than 5MB.

                I've already adjusted the WLS timeouts/message sizes as follows:

                Max Post Size = -1 (although I think this is only used for POST to WLS rather than the other way around)
                Max Message Size = 50000000
                JTA timeout = 60

                The only stack trace is for the broken pipe:

                ####<30-Mar-2010 08:35:57 o'clock BST> <Debug> <AlsbTransports> <int-app-04> <RSPCA_ESB_Server1> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <> <1269934557943> <BEA-000000> <LoadBalanceFailoverListener.sendMessageToServiceAsync
                com.bea.wli.sb.transports.TransportException: Broken pipe
                at com.bea.wli.sb.transports.TransportException.newInstance(TransportException.java:195)
                at com.bea.wli.sb.transports.http.HttpOutboundMessageContext.send(HttpOutboundMessageContext.java:370)
                at com.bea.wli.sb.transports.http.HttpTransportProvider.sendMessageAsync(HttpTransportProvider.java:564)
                at sun.reflect.GeneratedMethodAccessor327.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at com.bea.wli.sb.transports.Util$1.invoke(Util.java:82)
                at $Proxy60.sendMessageAsync(Unknown Source)
                at com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageAsync(LoadBalanceFailoverListener.java:148)
                at com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToServiceAsync(LoadBalanceFailoverListener.java:543)
                at com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToService(LoadBalanceFailoverListener.java:478)
                at com.bea.wli.sb.transports.TransportManagerImpl.sendMessageToService(TransportManagerImpl.java:544)
                at com.bea.wli.sb.transports.TransportManagerImpl.sendMessageAsync(TransportManagerImpl.java:422)
                at com.bea.wli.sb.pipeline.PipelineContextImpl.doDispatch(PipelineContextImpl.java:583)
                at com.bea.wli.sb.pipeline.PipelineContextImpl.dispatch(PipelineContextImpl.java:498)
                at stages.routing.runtime.RouteRuntimeStep.processMessage(RouteRuntimeStep.java:128)
                at com.bea.wli.sb.pipeline.StatisticUpdaterRuntimeStep.processMessage(StatisticUpdaterRuntimeStep.java:41)
                at com.bea.wli.sb.stages.StageMetadataImpl$WrapperRuntimeStep.processMessage(StageMetadataImpl.java:339)
                at com.bea.wli.sb.pipeline.RouteNode.doRequest(RouteNode.java:106)
                at com.bea.wli.sb.pipeline.Node.processMessage(Node.java:67)
                at com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:866)
                at com.bea.wli.sb.pipeline.Router.processMessage(Router.java:191)
                at com.bea.wli.sb.pipeline.MessageProcessor.processRequest(MessageProcessor.java:75)
                at com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:508)
                at com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:506)
                at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
                at weblogic.security.service.SecurityManager.runAs(Unknown Source)
                at com.bea.wli.sb.security.WLSSecurityContextService.runAs(WLSSecurityContextService.java:55)
                at com.bea.wli.sb.pipeline.RouterManager.processMessage(RouterManager.java:505)
                at com.bea.wli.sb.transports.TransportManagerImpl.receiveMessage(TransportManagerImpl.java:371)
                at com.bea.wli.sb.transports.http.HttpTransportServlet$RequestHelper$1.run(HttpTransportServlet.java:279)
                at com.bea.wli.sb.transports.http.HttpTransportServlet$RequestHelper$1.run(HttpTransportServlet.java:277)
                at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
                at weblogic.security.service.SecurityManager.runAs(Unknown Source)
                at com.bea.wli.sb.transports.http.HttpTransportServlet$RequestHelper.securedInvoke(HttpTransportServlet.java:276)
                at com.bea.wli.sb.transports.http.HttpTransportServlet$RequestHelper.service(HttpTransportServlet.java:237)
                at com.bea.wli.sb.transports.http.HttpTransportServlet.service(HttpTransportServlet.java:133)
                at weblogic.servlet.FutureResponseServlet.service(FutureResponseServlet.java:24)
                at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
                at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
                at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
                at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292)
                at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:175)
                at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3498)
                at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
                at weblogic.security.service.SecurityManager.runAs(Unknown Source)
                at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2180)
                at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2086)
                at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1406)
                at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
                at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
                java.net.SocketException: Broken pipe
                at java.net.SocketOutputStream.socketWrite0(Native Method)
                at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
                at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
                at com.certicom.io.OutputSSLIOStream.write(Unknown Source)
                at com.certicom.tls.record.WriteHandler.flushOutput(Unknown Source)
                at com.certicom.tls.record.WriteHandler.write(Unknown Source)
                at com.certicom.io.OutputSSLIOStreamWrapper.write(Unknown Source)
                at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
                at weblogic.net.http.HttpOutputStream.write(HttpOutputStream.java:22)
                at weblogic.utils.io.UnsyncByteArrayOutputStream.writeTo(UnsyncByteArrayOutputStream.java:104)
                at weblogic.net.http.HttpURLConnection.writeRequests(HttpURLConnection.java:162)
                at weblogic.net.http.HttpURLConnection.writeRequestForAsyncResponse(HttpURLConnection.java:492)
                at weblogic.net.http.AsyncResponseHandler.writeRequestAndRegister(AsyncResponseHandler.java:190)
                at weblogic.net.http.AsyncResponseHandler.writeRequestAndRegister(AsyncResponseHandler.java:152)
                at com.bea.wli.sb.transports.http.HttpOutboundMessageContext.send(HttpOutboundMessageContext.java:313)
                at com.bea.wli.sb.transports.http.HttpTransportProvider.sendMessageAsync(HttpTransportProvider.java:564)
                at sun.reflect.GeneratedMethodAccessor327.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at com.bea.wli.sb.transports.Util$1.invoke(Util.java:82)
                at $Proxy60.sendMessageAsync(Unknown Source)
                at com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageAsync(LoadBalanceFailoverListener.java:148)
                at com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToServiceAsync(LoadBalanceFailoverListener.java:543)
                at com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToService(LoadBalanceFailoverListener.java:478)
                at com.bea.wli.sb.transports.TransportManagerImpl.sendMessageToService(TransportManagerImpl.java:544)
                at com.bea.wli.sb.transports.TransportManagerImpl.sendMessageAsync(TransportManagerImpl.java:422)
                at com.bea.wli.sb.pipeline.PipelineContextImpl.doDispatch(PipelineContextImpl.java:583)
                at com.bea.wli.sb.pipeline.PipelineContextImpl.dispatch(PipelineContextImpl.java:498)
                at stages.routing.runtime.RouteRuntimeStep.processMessage(RouteRuntimeStep.java:128)
                at com.bea.wli.sb.pipeline.StatisticUpdaterRuntimeStep.processMessage(StatisticUpdaterRuntimeStep.java:41)
                at com.bea.wli.sb.stages.StageMetadataImpl$WrapperRuntimeStep.processMessage(StageMetadataImpl.java:339)
                at com.bea.wli.sb.pipeline.RouteNode.doRequest(RouteNode.java:106)
                at com.bea.wli.sb.pipeline.Node.processMessage(Node.java:67)
                at com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:866)
                at com.bea.wli.sb.pipeline.Router.processMessage(Router.java:191)
                at com.bea.wli.sb.pipeline.MessageProcessor.processRequest(MessageProcessor.java:75)
                at com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:508)
                at com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:506)
                at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
                at weblogic.security.service.SecurityManager.runAs(Unknown Source)
                at com.bea.wli.sb.security.WLSSecurityContextService.runAs(WLSSecurityContextService.java:55)
                at com.bea.wli.sb.pipeline.RouterManager.processMessage(RouterManager.java:505)
                at com.bea.wli.sb.transports.TransportManagerImpl.receiveMessage(TransportManagerImpl.java:371)
                at com.bea.wli.sb.transports.http.HttpTransportServlet$RequestHelper$1.run(HttpTransportServlet.java:279)
                at com.bea.wli.sb.transports.http.HttpTransportServlet$RequestHelper$1.run(HttpTransportServlet.java:277)
                at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
                at weblogic.security.service.SecurityManager.runAs(Unknown Source)
                at com.bea.wli.sb.transports.http.HttpTransportServlet$RequestHelper.securedInvoke(HttpTransportServlet.java:276)
                at com.bea.wli.sb.transports.http.HttpTransportServlet$RequestHelper.service(HttpTransportServlet.java:237)
                at com.bea.wli.sb.transports.http.HttpTransportServlet.service(HttpTransportServlet.java:133)
                at weblogic.servlet.FutureResponseServlet.service(FutureResponseServlet.java:24)
                at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
                at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
                at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
                at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292)
                at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:175)
                at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3498)
                at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
                at weblogic.security.service.SecurityManager.runAs(Unknown Source)
                at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2180)
                at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2086)
                at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1406)
                at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
                at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

                Thanks,

                Mike.
                • 5. Re: OSB problem posting >5MB body to a business service endpoint
                  687626
                  Can you try turning on network snoop to see the packets exchanged? In solaris i think you can use the snoop command . Could give you some indication why it is taking more than 10 mins for the transfer of 5 MB . Pipe broken error seems to occur when the other end of the tcp connection closes the connection. So seems like there could be some setting on the end system which is breaking the connection after 10 mins ...
                  1 person found this helpful
                  • 6. Re: OSB problem posting >5MB body to a business service endpoint
                    225953
                    Hello,

                    Snoop has highlighting some interesting goings on. It seems when sending a payload > 5MB after about 10secs I get ZeroWindow (need to Google what this is in a minute) warning and then the destination sends a RST. At this point OSB re-establishes the the connections and sends the data again. This keeps repeating itself again and again until it eventually gives up.

                    Thanks,

                    Mike.
                    • 7. Re: OSB problem posting >5MB body to a business service endpoint
                      687626
                      Looks like the problem is at target app which is receiving the message so slowly which eventually results in filling of the tcp buffer at their side and making the available window size to 0
                      • 8. Re: OSB problem posting >5MB body to a business service endpoint
                        225953
                        I think it might be the other way around, OSB receive buffer is full but not processing what's in it for some reason. Sorry I didn't give you all the details before, the zerowindow message is reported by the OSB server and then the destination server keeps sending zerowindowprobes until it eventually gives up and resets the connection.

                        Mike.
                        • 9. Re: OSB problem posting >5MB body to a business service endpoint
                          225953
                          I've just setup OSB 10.3.0 on Windows using the Sun JVM and JRockit and experience the same thing. Maybe a JVM bug? I'll try updating to the latest at see what happens.

                          Mike.

                          Edited by: mikerobins on Apr 1, 2010 3:36 AM
                          • 10. Re: OSB problem posting >5MB body to a business service endpoint
                            FabioDouek
                            Hi Mike,

                            Try disabling "Use Chunked Streaming Mode" in your Business Service.
                            I had some issues when running performance test with this option enabled. After disabling it the issue was sorted out.

                            Please let me know if this helps.
                            Fabio Douek
                            • 11. Re: OSB problem posting >5MB body to a business service endpoint
                              225953
                              Thanks for the suggestion but this is already off and I don't think it can even be enabled for https endpoints.

                              Mike.
                              • 12. Re: OSB problem posting >5MB body to a business service endpoint
                                FabioDouek
                                Hi Mike,

                                I would be interested in trying to reproduce the issue. If you want to export your project and send me via email, I can try to reproduce (just remove any confidential information and unnecessary resources).

                                Just some points to check before:
                                1) You mentioned that if you send the requests for about 10 minutes with message size>5mb, the following requests will fail.
                                1.1) Only the requests with message size greater than 5MB or any smaller requests from OSB?
                                1.2) How many messages did you send during this 10 minutes period? How many parallel threads?
                                1.3) Is there any hogging threads in WLS?
                                1.4) At this stage (after the 10 minutes), if you see the socket connections to your backend from your unix shell, do you see any connection? How many? In what state?
                                1.5) What kind of backend are you hitting?
                                1.6) Are you using a single server for this test? How much memory do you have allocated to the JVM (min/max)?

                                Regards,
                                Fabio Douek.
                                • 13. Re: OSB problem posting >5MB body to a business service endpoint
                                  225953
                                  Hi Fabio,

                                  1.1) Any message sent to this business service that is around 5MB or greater (I cannot be 100% on the exact tipping point) will return an broken pipe exception after 10mins or so. Smaller messages e.g. a 4.5MB body will complete successfully in ~30secs.
                                  1.2) I only invoke the business service once but as I mentioned earlier, a packet analyser shows the target reseting the TCP connection and then OSB trying to resend the entire message. Again this only happens with a larger message, smaller ones go through fine with no TCP resets and resends.
                                  1.3) Only the one thread is considered stuck whilst trying to write to the socket (see first post for the detail) but cpu utilisation is basically zero when I call this business service.
                                  1.4) Will need to check this but I think only one with a state of ESTABLISHED (will update this message once I've checked)
                                  1.5) Just an .Net service on IIS - think its running on Windows 2003 VM machine (again will update this message once I've checked). I'd love to blame this but the fact messages of all shapes and sizes work fine from SOAPUI means it's probably an OSB/WLS/JVM issue.
                                  1.6) I've tried a clustered and single server setup. Clustered setup has 2GB heap per managed server (min and max fixed to 2GB). Single server had 1GB (min 512MB and max 1GB)

                                  I've also tried taking my Windows install of OSB outside our corporate network (just direct internet connection to the endpoint) to rule out any firewall, NAT etc issues on our network and got the same problem :(


                                  Thanks,

                                  Mike.
                                  • 14. Re: OSB problem posting >5MB body to a business service endpoint
                                    687626
                                    Hi Mike,

                                    You mentioned that the reason for tcp connection reset is due to a zero window condition. Can you check out which side of the connection gets its buffer full. Since OSB is sending data to the endpoint , data is flowing into the endpoint only. Apart from the ACK's for the packet send by OSB, nothing should be coming back to OSB. So its hard to understand why OSB is reporting its side buffer is full even though no real data is flowing towards it.

                                    The packet analyser should also tell the available window size on either side of the connections. So if you track through all the packets exchanged in an interaction you would better understand where the bottlenck is.

                                    As a next step of your troubleshooting may be you can use Wireshark in your windows install and see the TCP interactions for the 2 scenarios:
                                    1. Through OSB - where you get the pipe broken error
                                    2. Through SOAPUI - where the request is working successful.

                                    This would help you understand the difference. Analyse the ACK flows in 2 scenarios, the available window size on 2 sides etc.

                                    If you feel this could be an OSB/WLS issue, I think it is time to contact the support.
                                    1 2 Previous Next