0 Replies Latest reply: Jan 28, 2013 9:20 AM by 987572 RSS

    WL Server Stuck Thread on Network Read

      I am experiencing an issue with the Weblogic deployment at my organisation.

      Firstly an introduction; My Application works by receiving a Web Service (SOAP) Request from a client, the Web Service request is then processed and a internal message packet is generated and sent to a C++ Application ( which is listening on a particular port). The C++ application then processes the information, and returns the response to the web service Layer, which then process the Web Service request back to the calling client.

      Currently we are having a situation where some packets are sent to the C++ application as expected. But then we are having situations where some packets NEVER reach the C++ side and the calling client time outs the entire request ( configured timeout is set for 180 seconds) and on the WebLogic Layer we have noticed a significant amount of Stuck ( and Hogging ) Threads. It should be noted that even after a stuck thread is recorded some requests are still being processed normally.

      A deeper investigation into the Weblogic Thread dump shows the Weblogic thread is stuck at:

      _at sun/nio/ch/IOUtil.read(Ljava/io/FileDescriptor;Ljava/nio/ByteBuffer;JLsun/nio/ch/NativeDispatcher;Ljava/lang/Object;)I(IOUtil.java)[inlined]_+

      _at sun/nio/ch/SocketChannelImpl.read(SocketChannelImpl.java:236)[optimized]_+

      +_^-- Holding lock: java/lang/Object@0x1a27db60[thin lock]_+

      _at com/reso/number/service/comm/SoaSocket.receive(SoaSocket.java:128)[optimized]_+

      _at com/reso/number/service/event/OTEvent.submitEvent(TEvent.java:219)[optimized]_+

      _at com/reso/number/service/event/SIMQueryEvent.process(QQEvent.java:368)[optimized]_+

      It seems that a write is sent to the C++ Application and the thread is 'WAITING' for the response.

      Now, the problem is that due to the Infrastructure in place and architectural setup ( We are using Solaris Zone for Node deployment and we have the WebLogic AS on the same node/zone as the C++ application) we are unable to do a network trace to determine whether WebLogic actually sends the request.

      Now my questions: i) Are there any known ways to trace the networks packets sent by Java ( maybe via the JVM) ii) Could Garbage collection cause the symptoms being observed? iii)Any recommended ways to resolve this kind of problem?

      Thanks for all the help.