I'm observing a strange behavior on Weblogic 10.3.4 under Linux RedHat 5.4
My application works well on unitary test but when I run load tests I have some HTTP connection leak.
* I've configured a Work manager with 10 as max thread constraint and 250 as Capacity.
* I have a cluster of 6 managed server.
* I run a test with 200 injector threads.
* During the run, my max thread constraint is reached and I use the capacity constraint (up to 15 threads in backlog per server)
* At the end, I have up to 30 threads blocked forever. In fact, I have an apache reverse proxy and it hangs the connections an return 502 status to my clients.
* On Weblogic side, the 30 errors threads correspond to 30 pending request still in the server backlog.
* On system side, as the reverse proxy has hanged the connection, the 30 requests correpond to 30 CLOSE_WAIT sockets.
What I understand is, when the max thread constraint is reached, the pending requests with established socket are stored and given to an executor thread when one is available. But sometimes, it seems to have a connection leak in weblogic as the connection is maintened and never served.
When doing a thread dump after the run, all my executor thread are turned back to the thread pool, so no thread is working...
I am under linux and I use the PosixSocketMuxer (native). I've tried with JavaSocketMuxer. And it works fine. My performance are worse with JavaSocketMuxer but at least, it release all the connections. The leak seems to be inside the PosixSocketMuxer.
I would like to know if anybody has encountered such problem and if it is possible to set something at weblogic or system level.
A workaround is to remove the min Thread Constraints of the work manager.
It seems to be a bug on work manager when a min thread constraints under certain conditions (min thread constraints = 0 maybe...).
For my software I don't really need min thread constraints so I've removed it.
But Weblogic behavior doesn't seems correct on 10.3.4 version.