3 Replies Latest reply on Dec 5, 2009 2:18 AM by 450961

    Internal Server Errors when using JMS

    738871
      We have a system where there are multiple JMS listeners. The OSB is set to listen to a directory where we place files - it is currently set to read 500 files every 10 seconds. When putting lots of files into this directory we get continual errors:
      Starting with:
      <Dec 3, 2009 4:29:28 PM EST> <Error> <OSB Kernel> <BEA-382004> <Failed to process request message for service ProxyService FileIngestion/IngestionEndpoints/ONdocIngest: com.bea.wli.sb.context.BindingLayerException: General runtime error: Broken pipe
      com.bea.wli.sb.context.BindingLayerException: General runtime error: Broken pipe
      at com.bea.wli.sb.context.ContextUtils.createBindingLayerException(ContextUtils.java:920)
      at com.bea.wli.sb.pipeline.PipelineContextImpl.doDispatch(PipelineContextImpl.java:595)
      at com.bea.wli.sb.pipeline.PipelineContextImpl.dispatchSync(PipelineContextImpl.java:466)
      at stages.transform.runtime.WsCalloutRuntimeStep$WsCalloutDispatcher.dispatch(WsCalloutRuntimeStep.java:1401)
      at stages.transform.runtime.WsCalloutRuntimeStep.processMessage(WsCalloutRuntimeStep.java:236)
      Truncated. see log file for complete stacktrace
      com.bea.wli.sb.transports.TransportException: Broken pipe
      at com.bea.wli.sb.transports.TransportException.newInstance(TransportException.java:195)
      at com.bea.wli.sb.transports.http.HttpOutboundMessageContext.send(HttpOutboundMessageContext.java:370)
      at com.bea.wli.sb.transports.http.HttpTransportProvider.sendMessageAsync(HttpTransportProvider.java:564)
      at sun.reflect.GeneratedMethodAccessor634.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      Truncated. see log file for complete stacktrace
      java.net.SocketException: Broken pipe
      at java.net.SocketOutputStream.socketWrite0(Native Method)
      at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
      at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
      at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
      at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
      Truncated. see log file for complete stacktrace
      >


      Then lots of:
      <Dec 3, 2009 4:32:14 PM EST> <Warning> <EJB> <BEA-010065> <MessageDrivenBean threw an Exception in onMessage(). The exception was:
      javax.ejb.EJBException: nested exception is: com.bea.wli.sb.transports.TransportException: Internal Server Error.
      javax.ejb.EJBException: nested exception is: com.bea.wli.sb.transports.TransportException: Internal Server Error
      at com.bea.wli.sb.transports.poller.listener.PolledMessageListenerMDB.__onMessage(PolledMessageListenerMDB.java:55)
      at com.bea.wli.sb.transports.poller.listener.PolledMessageListenerMDB.access$000(PolledMessageListenerMDB.java:31)
      at com.bea.wli.sb.transports.poller.listener.PolledMessageListenerMDB$1.run(PolledMessageListenerMDB.java:41)
      at com.bea.wli.sb.transports.poller.listener.PolledMessageListenerMDB$1.run(PolledMessageListenerMDB.java:39)
      at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
      Truncated. see log file for complete stacktrace


      The OSB stays up for a while, but eventually becomes un-responsive (cannot get to the homepage, java process using 100% cpu) - typically after handling ~800,000 messages. These messages are very small - typically 16 bytes.

      There are about 5 active queues and about 10 JMS listener processes.

      What can we do to make OSB more stable?
      Thanks!

      Joe Obernberger
      Northrop Grumman IT
      http://www.lovehorsepower.com
        • 1. Re: Internal Server Errors when using JMS
          Tom B-Oracle
          The exceptions don't look they are specific to JMS. Can you try posting to an OSB newsgroup?
          • 2. Re: Internal Server Errors when using JMS
            738871
            Thank you Tom - do you have an newsgroup/forum in mind? The problem starts when we start using JMS (pretty much all we use the OSB for):

            ####<Dec 3, 2009 3:47:27 PM EST> <Error> <OSB Kernel> <telesto.it-protect.com> <xbusServer> <[ACTIVE] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <BEA1-2F991BD24277D06D1480> <> <1259873247965> <BEA-382004> <Failed to process request message for service ProxyService FileIngestion/IngestionEndpoints/ONdocIngest: com.bea.wli.sb.context.BindingLayerException: General runtime error: Broken pipe
            com.bea.wli.sb.context.BindingLayerException: General runtime error: Broken pipe
            at com.bea.wli.sb.context.ContextUtils.createBindingLayerException(ContextUtils.java:920)
            at com.bea.wli.sb.pipeline.PipelineContextImpl.doDispatch(PipelineContextImpl.java:595)
            at com.bea.wli.sb.pipeline.PipelineContextImpl.dispatchSync(PipelineContextImpl.java:466)
            at stages.transform.runtime.WsCalloutRuntimeStep$WsCalloutDispatcher.dispatch(WsCalloutRuntimeStep.java:1401)
            at stages.transform.runtime.WsCalloutRuntimeStep.processMessage(WsCalloutRuntimeStep.java:236)
            at com.bea.wli.sb.pipeline.debug.DebuggerRuntimeStep.processMessage(DebuggerRuntimeStep.java:74)
            at com.bea.wli.sb.stages.StageMetadataImpl$WrapperRuntimeStep.processMessage(StageMetadataImpl.java:346)
            at com.bea.wli.sb.pipeline.PipelineStage.processMessage(PipelineStage.java:84)
            at com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:866)
            at com.bea.wli.sb.pipeline.Pipeline.processMessage(Pipeline.java:141)
            at com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:866)
            at com.bea.wli.sb.pipeline.PipelineNode.doRequest(PipelineNode.java:55)
            at com.bea.wli.sb.pipeline.Node.processMessage(Node.java:67)
            at com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:866)
            at com.bea.wli.sb.pipeline.Router.processMessage(Router.java:191)
            at com.bea.wli.sb.pipeline.MessageProcessor.processRequest(MessageProcessor.java:75)
            at com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:508)
            at com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:506)
            at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
            at weblogic.security.service.SecurityManager.runAs(Unknown Source)
            at com.bea.wli.sb.security.WLSSecurityContextService.runAs(WLSSecurityContextService.java:55)
            at com.bea.wli.sb.pipeline.RouterManager.processMessage(RouterManager.java:505)
            at com.bea.wli.sb.transports.TransportManagerImpl.receiveMessage(TransportManagerImpl.java:371)
            at com.bea.wli.sb.transports.file.FileTask.process(FileTask.java:108)
            at com.bea.wli.sb.transports.poller.listener.PolledMessageListenerMDB.__onMessage(PolledMessageListenerMDB.java:52)
            at com.bea.wli.sb.transports.poller.listener.PolledMessageListenerMDB.access$000(PolledMessageListenerMDB.java:31)
            at com.bea.wli.sb.transports.poller.listener.PolledMessageListenerMDB$1.run(PolledMessageListenerMDB.java:41)
            at com.bea.wli.sb.transports.poller.listener.PolledMessageListenerMDB$1.run(PolledMessageListenerMDB.java:39)
            at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
            at weblogic.security.service.SecurityManager.runAs(Unknown Source)
            at weblogic.security.Security.runAs(Security.java:41)
            at com.bea.wli.sb.transports.poller.listener.PolledMessageListenerMDB.onMessage(PolledMessageListenerMDB.java:39)
            at weblogic.ejb.container.internal.MDListener.execute(MDListener.java:466)
            at weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:371)
            at weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:327)
            at weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4547)
            at weblogic.jms.client.JMSSession.execute(JMSSession.java:4233)
            at weblogic.jms.client.JMSSession.executeMessage(JMSSession.java:3709)
            at weblogic.jms.client.JMSSession.access$000(JMSSession.java:114)
            at weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:5058)
            at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:516)
            at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
            at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

            Despite all these exceptions, it runs, but eventually fails.

            It is running on Linux, and top shows:
            top - 11:29:04 up 27 days, 17:35, 5 users, load average: 1.08, 0.73, 0.63
            Tasks: 294 total, 1 running, 293 sleeping, 0 stopped, 0 zombie
            Cpu(s): 6.4%us, 0.1%sy, 0.0%ni, 93.5%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
            Mem: 32949628k total, 32835820k used, 113808k free, 234224k buffers
            Swap: 2031608k total, 216k used, 2031392k free, 4028112k cached

            PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
            20718 root 18 0 27.1g 22g 30m S 103.8 70.4 825:02.20 java


            Joe Obernberger
            Northrop Grumman IT

            Edited by: user3409757 on Dec 4, 2009 11:29 AM
            • 3. Re: Internal Server Errors when using JMS
              450961
              What Tom is pointing out is that the source of the exception appears to be here:

              com.bea.wli.sb.pipeline.PipelineContextImpl.doDispatch(PipelineContextImpl.java:595)

              which isn't part of the WL JMS implementation. That code is just making use of JMS (it's running in an onMessage routine) and it is that code (not JMS) that is throwing an exception.

              For OSB issues you can post here:
              SOA Suite


              -Jeff