APEX

Not if it's running on Windows. Correct. Correct. Far more than enough. A couple of seconds is enough to transit an IP packet several times around the world. No. Connection timeout means there has been no response whatsoever. This is due to, in decreasing order of probability:

1. There has been a temporary network outage for the duration of the timeout, for example a router going down and up.

2. The host is down.

3. The server is running on a non-Windows platform and its backlog queue is full, caused by accept() running behind incoming connections. This is extremely unlikely given that minimum actual backlog queue sizes on non-Windows platforms are of the order of 500 and have been for a decade. That is correct for a server running on Windows only. This suggestion makes no sense. If the client got a connection timeout there was no response to the connection attempt, so there wasn't any back-end activity whatsoever. By parallelizing the operations you are presumably increasing network load. Maybe you are doing so far more than you think. Sniff the network between the client and whatever it is failing to connect to and observe the SYN and SYN-ACK packets.

I would also trawl your client code for HttpURLConnection.disconnect() calls and remove them. They inhibit client-side connection pooling, which mitigates most problems of this sort considerably.

I thought you could get a timeout if you have a firewall which doesn't allow traffic through.

Correct, I should have included that. There were firewalls on the market that issued RSTs in this situation but I think everybody now agrees that this is incorrect behaviour. The other possible firewall action is an ICMP Destination Unreachable.

Hello Esmond,

thanks for your help.
As a general question first, do you have a reference that describe when such or such exception's "message" is raised?
As I said in the OP, I haven't found this in the TCP RFCs, and I fear it's system-specific; in this case it's not clear whether the message comes from java.net API, from the Windows' socket API layer,...
The server is running on Linux (RedHat). The client is on Windows.
TBH, you were the one that suggested those reasons :o)
Still, revisited below.
OK.
In my case I find this unlikely, given the "client code version" effect (unless the router restart duration was so pinpointedly timed that it consistently impacts the "small traffic" client and consistently not the "higher traffic" one.
Moreover, my network admin team claims there were no such incidents.
In my case, that seems unlikely, as parallel requests did succeed within seconds of some requests failing.
Ah,ah, that's something to chew on.
What do you call the "backlog queue"? There are a handful of "backlog" mentioned in, e.g., this Red-Hat-specific doc:
net.ipv4.tcp_max_syn_backlog "Length of the backlog queue for each socket (default is 128)"
net.core.netdev_max_backlog "Maximum number of packets that can be queued on input when a network interface receives packets faster than the kernel can process them (default is 300)"

Additionally, if I happened to find out that the "backlog queue" indeed enqueues lots of connection requests, doesn't that add delay to the connection request-accept duration?
Of course, if the queue fills up, the client will eventually face the exception, no matter what its timeout setting is... I should definitely monitor this queue's status.
I'm not sure I understand: I do witness +"java.net.ConnectException: Connection refused: connect"+ if I shut down the Apache server on the Linux host. Admittedly, I haven't proven I get them if I let the server up and overflow its capacity, on Linux. Does your remark "correct for a server running on Windows only" apply only to the "overloaded server" case?
Thanks for confirming.
I did another test with the "parallel calls" disabled, and faced the same issues.
I'm still on investigating the changes between the the client versions (I fear a JAX-WS version change has sneaked into this as well, sorry if I unwantingly misrepresent the "small changes").
Err..., thanks, but what would I do with the results? Shouldn't I, instead, or in addition, monitor the backlog queue you mentioned above?
Yes. But the client has no access to the HttpURLConnection (encapsulated in JAX-WS stubs). I don't know if the JAX-WS client layer performs any disconnection (why would it).
However, that raises another question: why do I even have connection requests with HTTP1.1 enabled on both ends? From a cursory look at the Apache doc, I seem to understand that the number of requests over the same socket is bounded in the default configuration.

I'll post updates if I find anything.

Regards,

J.

Thanks for the suggestion.

I did a ping -r 5 serverIp from two Windows client, and the connection is direct (no hops). That guarantee that there is no intermediate machine between the client sand the server, correct?

Additionally, I did a netstat -o | find "<Apache port>" on the Windows clients, and only the PIDS of Java processes are listed.
Does that guarantee that this is not Windows' software firewall that opens the socket with the Apache server?

They don't specify APIs or exceptions at all, just the protocol, in fact one of the curiousities of TCP/IP is the complete lack of a formal document relating the Sockets API to the protocol. However you can figure it out like this:

- ICMP UNREACH -> NoRouteToHostException
- RST -> ConnectException 'connection refused'
- nothing -> connection timeout. It is indeed. Windows will reject a connection when the backlog queue is full with an RST. Most other systems just ignore it and rely on the connect() API to retry, which it does three times in normal implementations ... if all 3 fail with no response you get the timeout exception. Everything except the timeout exception emanates from the TCP/IP stack. Timeout exception originates from Java. That's because there is no timed-connect API at the Sockets level, you have to synthethize it with a non-blocking connect() followed by a select() followed by another connect() if the select() didn't time out. That is also the reason for the existence of SocketChannel.finishConnect(). I suggested the ones that were correct :-| The queue of incoming connections which have already been accepted by the TCP/IP stack (SYN-SYN/ACK-ACK) but which haven't been returned yet as sockets via the accept() method, because the server is running behind. The length of the queue is specified in the Sockets listen() call, or in Java in the ServerSocket constructor, unless you default it. No. That's what it's for. The connection is completed by the stack before or while the application calls accept(). You can't. There is no API to tell you anything whatsoever about the backlog queue, not even its actual or maximum size. You might be able to dig around in a Linux/Unix /dev/proc way, that's it. Of course you do. Nothing is listening at the port so the TCP/IP stack rejects the connection. I was referring to your colleague's theory about the server not being able to accept the connection, which I assumed means the server process is still running and its backlog queue is full. If he means something else, OK. Correct. Analyze them. Post them here as a last resort: somebody might look ;-) If you can, yes, but as well, not instead. No idea what Apache does, but Java pools HTTP 1.1 connections and reuses them if required within a short time, I think 15 seconds. This is the implementation of HTTP 1.1 Connection: keep-alive.