Java APIs

Hello,

A while ago I opened a topic on {message:id=4635424} , to discuss what could cause the dreaded exception +"java.net.ConnectException: Connection timed out: connect"+ , and whether it would make sense to increase the client timeout (was set to 20s).
The answers to the first question were "the server may be too busy", "the host may be down", "there may be a network problem".
The answer to the second question was "No, if the server and network are healthy, 20 seconds is amply enough".

I am still facing the following exception reproducibly, in a round of performance tests of a WebService application.
A set of test clients exercise WS calls over HTTPS targetting the server configuration.
The server has an Apache reverse-proxy that receives the incoming flow, unwraps the SSL layer, and forwards regular HTTP WebService calls to a back-end Glassfish server hosting a JEE application.

Since a recent modification of the client code (each client now has two threads that invoke the same web service endpoint), the test client has started facing ConnectException reproducibly (~0.1% of requests). I quote an example of these exceptions below.

1) From earlier reading on this forum, I thought this indicates that the server-side (in this case, Apache) has not accepted, or not timely accepted, the connection request issued by the client.
However, I see nothing in my server-side logs or monitoring page that indicates that the Apache server is too busy to service the connection requests. The Apache status page shows lots of non-busy processes.

2) Moreover, someone from my team pointed out that if the server could not accept the connection request, I should face +"java.net.ConnectException: Connection refused: connect"+ instead of +"java.net.ConnectException: Connection timed out: connect"+ (indeed Connection refused is what I observe if shut down the Apache server, and also what I observe on a dummy JAX-WS client/server test with an undersized server-side pool).


Further research gave no authoritative answers (TCP tutorial or RFC) as to the difference; I don't know whether those messages are OS-specific or API-specific. I found the following pages though:
http://www.realvnc.com/pipermail/vnc-list/2005-November/052929.html (seems to indicate that "connection timed out" points out a network problem as opposed to "connection refused" indicating a non-working server)
http://answers.yahoo.com/question/index?qid=20090415213118AAsbWzJ (I'm not sure this answer is very reliable)

3) Someone from another team mentioned that maybe the client-side "connection request timeout" might have elapsed before the Apache-to-backend timeout, which could explain that Apache doesn't see nor log any problem. However, to only even know that it's supposed to forward the request to the back-end Glassfish server, Apache should have "accepted" the socket, right? So how could the connection request itself time out?

4) If the "network problem" hypothesis is right (nasty firewall, bandwidth limit,...), how does the client code have an impact on the problem (remember, the problem, which used to be very infrequent, is almost systematically reproducible since a client code update).

5) How else would you advise to investigate this issue?

Thanks in advance,

J.

P.S.: stack trace of intermittent "Connection timed out" problem.

Caused by: java.net.ConnectException: Connection timed out: connect
                at java.net.PlainSocketImpl.socketConnect(Native Method)
                at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
                at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
                at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
                at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
                at java.net.Socket.connect(Socket.java:519)
                at com.sun.net.ssl.internal.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:550)
                at com.sun.net.ssl.internal.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:141)
                at sun.net.NetworkClient.doConnect(NetworkClient.java:163)
                at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
                at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
                at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:271)
                at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:328)
                at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:172)
                at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:816)
                at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:158)
                at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:881)
                at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:230)
                at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:107)
                ... 22 more