I am experiencing an issue with the Weblogic deployment at my organisation.
Firstly an introduction; My Application works by receiving a Web Service (SOAP) Request from a client, the Web Service request is then processed and a internal message packet is generated and sent to a C++ Application ( which is listening on a particular port). The C++ application then processes the information, and returns the response to the web service Layer, which then process the Web Service request back to the calling client.
Currently we are having a situation where some packets are sent to the C++ application as expected. But then we are having situations where some packets NEVER reach the C++ side and the calling client time outs the entire request ( configured timeout is set for 180 seconds) and on the WebLogic Layer we have noticed a significant amount of Stuck ( and Hogging ) Threads. It should be noted that even after a stuck thread is recorded some requests are still being processed normally.
A deeper investigation into the Weblogic Thread dump shows the Weblogic thread is stuck at:
+_^-- Holding lock: java/lang/Object@0x1a27db60[thin lock]_+
It seems that a write is sent to the C++ Application and the thread is 'WAITING' for the response.
Now, the problem is that due to the Infrastructure in place and architectural setup ( We are using Solaris Zone for Node deployment and we have the WebLogic AS on the same node/zone as the C++ application) we are unable to do a network trace to determine whether WebLogic actually sends the request.
Now my questions: i) Are there any known ways to trace the networks packets sent by Java ( maybe via the JVM) ii) Could Garbage collection cause the symptoms being observed? iii)Any recommended ways to resolve this kind of problem?
Thanks for all the help.