This discussion is archived
1 2 Previous Next 18 Replies Latest reply: Feb 8, 2012 10:36 PM by 911421 RSS

ObjectInputStream.readObject() hangs forever during socket communication

911421 Newbie
Currently Being Moderated
I have encountered a problem of socket communication on linux system, the communication process is like below: client send a message to ask the server to do a compute task, and wait for the result message from server after the task completes.

But the client would hangs up to wait for the result message if the task costs a long time such as about 40 minutes even though from the server side, the result message has been written to the socket to respond to the client, but it could normally receive the result message if the task costs little time, such as one minute. Additionally, this problem only happens on customer environment, the communication process behaves normally in our testing environment.

I have suspected the cause to this problem is the default timeout value of socket is different between customer environment and testing environment, but the follow values are identical on these two environment, and both Client and server.

getSoTimeout:0
getReceiveBufferSize:43690
getSendBufferSize:8192
getSoLinger:-1
getTrafficClass:0
getKeepAlive:false
getTcpNoDelay:false
the codes on CLient are like:

Message msg = null;
ObjectInputStream in = client.getClient().getInputStream();
//if no message readObject() will hang here
while ( true ) {
try {
Object recObject = in.readObject();
System.out.println("Client received msg.");
msg = (Message)recObject;
return msg;
}catch (Exception e) {
e.printStackTrace();
return null;
}
}
the codes on server are like,

ObjectOutputStream socketOutStream = getSocketOutputStream();
try {
MessageJobComplete msgJobComplete = new MessageJobComplete(reportFile, outputFile );
socketOutStream.writeObject(msgJobComplete);
}catch(Exception e) {
e.printStackTrace();
}
in order to solve this problem, i have added the flush and reset method, but the problem still exists:

ObjectOutputStream socketOutStream = getSocketOutputStream();
try {
MessageJobComplete msgJobComplete = new MessageJobComplete(reportFile, outputFile );
socketOutStream.flush();
logger.debug("AbstractJob#reply to the socket");
socketOutStream.writeObject(msgJobComplete);
socketOutStream.reset();
socketOutStream.flush();
logger.debug("AbstractJob#after Flush Reply");
}catch(Exception e) {
e.printStackTrace();
logger.error("Exception when sending MessageJobComplete."+e.getMessage());
}
so do anyone knows what the next steps i should do to solve this problem. I guess the cause is the environment setting, but I do not know what the environment factors would affect the socket communication?

And the socket using the Tcp/Ip protocal to communicate, the problem is related with the long time task, so what values about tcp would affect the timeout of socket communication?

After my analysis about the logs, i found after the message are written to the socket, there were no exceptions are thrown/caught. But always after 15 minutes, there are exceptions in the objectInputStream.readObject() codes snippet of Server Side which is used to accept the request from client. However, socket.getSoTimeout value is 0, so it is very strange that the a Timed out Exception was thrown.

{2012-01-09 17:44:13,908} ERROR java.net.SocketException: Connection timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:146)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:312)
at sun.security.ssl.InputRecord.read(InputRecord.java:350)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:809)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:766)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:94)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:69)
at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2265)
at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2558)
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2568)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368)
so why the Connection Timed out exceptions are thrown?

Edited by: 908418 on 2012-1-17 下午11:19
  • 1. Re: ObjectInputStream.readObject() hangs forever during socket communication
    sabre150 Expert
    Currently Being Moderated
    If this is a time out issue then why not send a heartbeat from client to server and server to client every N seconds. I do this on one program just so that the server knows that the client has not died. In this heartbeat I include a very very crude estimate (nearly as crude as MS estimates of time remaining) of the time remaining before the task is complete, a counter and the current client time.
  • 2. Re: ObjectInputStream.readObject() hangs forever during socket communication
    911421 Newbie
    Currently Being Moderated
    thanks for your reply, what your advised would be the next improvement of my codes. but i have to seek the root cause of this problem, i found it is very, very strange. I have not set the timeout for the socket, but there is a "connection timed out" exception, and the client could receive the response if the time costs little.
  • 3. Re: ObjectInputStream.readObject() hangs forever during socket communication
    EJP Guru
    Currently Being Moderated
    it is very strange that the a Timed out Exception was thrown.
    It wasn't. You didn't get a read timeout. That would have been a SocketTimeoutException. Instead you got a SocketException with a message: 'Connection timed out', which would normally be associated with a ConnectException.

    It is strange that this thread is titled 'hangs forever' when nothing of the sort happened.
  • 4. Re: ObjectInputStream.readObject() hangs forever during socket communication
    911421 Newbie
    Currently Being Moderated
    thanks for your reply, the socket used between client and server is created using SSLContext, such as

    public static ServerSocket getSecureServerSocket(int port) throws IOException, BindException {

              System.setProperty("javax.net.ssl.keyStore", CERTIF_PATH);
              System.setProperty("javax.net.ssl.keyStorePassword", CERTIF_PASSWORD);
              ServerSocketFactory serverFactory = SSLServerSocketFactory.getDefault();
              return serverFactory.createServerSocket(port);
         }

    public static Socket getSecureSocket(String host, int port) {
    SSLContext context = null;
              try {
                   context = SSLContext.getInstance("SSL");
                   context.init(null, trustAllCerts, new java.security.SecureRandom());
                   System.setProperty("javax.net.ssl.keyStore", CERTIF_PATH);
                   System.setProperty("javax.net.ssl.keyStorePassword", CERTIF_PASSWORD);
                   SocketFactory socketFactory = context.getSocketFactory();
                   return socketFactory.createSocket(host, port);
              } catch (KeyManagementException e) {
                   e.printStackTrace();
              } catch (NoSuchAlgorithmException e) {
                   e.printStackTrace();
              }
    }


    Yes, you are right, and From the JAVA API document about the setSoTimeout method, if this method is set a no-zero value, when the times expires, only the SocketTimeoutException was thrown, not the SocketException:Connection timed out, so, this exception should not relate with the setSotimeoutMethod.

    public void setSoTimeout(int timeout) throws SocketException
    Enable/disable SO_TIMEOUT with the specified timeout, in milliseconds. With this option set to a non-zero timeout, a read() call on the InputStream associated with this Socket will block for only this amount of time. If the timeout expires, a java.net.SocketTimeoutException is raised, though the Socket is still valid. The option must be enabled prior to entering the blocking operation to have effect. The timeout must be > 0. A timeout of zero is interpreted as an infinite timeout.

    Edited by: 908418 on 2012-1-17 下午11:10
  • 5. Re: ObjectInputStream.readObject() hangs forever during socket communication
    911421 Newbie
    Currently Being Moderated
    It is also strange that this thread is titled 'hangs forever' when nothing of the sort happened.

    To be specifically, the saying that "But the client would hangs up to wait for the result message if the task costs a long time such as about 40 minutes even though from the server side, the result message has been written to the socket to respond to the client"

    there are both ObjectInputStream.readObject() and ObjectOutputStream.writeObject() methods invoked on Client and server.

    when server invokes objectOutputStream.writeObject() to responds to the client, the corresponding objectInputStream.readObject() method on client could not receive, but hangs for ever. this objectOutputStream and objectInputStream is produced in one connection to the socket.

    also there are objectOutputStream.writeObject() method on client to write request to the server where the ojectInputStream.readObject() is used to accept the request.

    the strange case is that after the objectOutputStream.writeObject() was invoked on server, 15minutes later, the ojectInputStream.readObject() on server throws the java.net.SocketException: Connection timed out.
  • 6. Re: ObjectInputStream.readObject() hangs forever during socket communication
    EJP Guru
    Currently Being Moderated
    When you got the SocketException on the server you should have closed the socket as a matter of course: a Socket is useless after any exception except a SocketTimeoutException. Then the client would have got an EOFException instead of hanging forever, and your (and our) attention would have been directed to the real problem, which is the 'connection timed out' SocketException. Which however I cannot explain. One reason for that is that you haven't posted the relevant code.
  • 7. Re: ObjectInputStream.readObject() hangs forever during socket communication
    796440 Guru
    Currently Being Moderated
    Crosspost: http://www.coderanch.com/t/564782/sockets/java/ObjectInputStream-readObject-hangs-forever-during#2564727
  • 8. Re: ObjectInputStream.readObject() hangs forever during socket communication
    911421 Newbie
    Currently Being Moderated
    yes, what causes the java.net.SocketExcetpion:Connection timed out, is one of two points of my question, the other is that why the client side could not receive the replied message from the server, even the server has written the message using method objectOutputStream.writeObject(), what weird is that the writeObject() doesnot throw any exceptions.
  • 9. Re: ObjectInputStream.readObject() hangs forever during socket communication
    EJP Guru
    Currently Being Moderated
    The connection timeout is so odd I am suspecting a kernel bug. The only way you can get that error is by calling connect() at the C level, and servers don't do that, they get their connections from accept().

    The write didn't thrown an exception because of socket buffers. A subsequent write would.
  • 10. Re: ObjectInputStream.readObject() hangs forever during socket communication
    911421 Newbie
    Currently Being Moderated
    why the socket would buffer the exception? why not throw exceptions once the outputstream or inputstream is invalid
  • 11. Re: ObjectInputStream.readObject() hangs forever during socket communication
    EJP Guru
    Currently Being Moderated
    why the socket would buffer the exception?
    It didn't buffer the exception. It buffered the data that was written. The error doesn't occur until that data has failed to be written, which happens asynchronously after the write method returns and the kernel's internal TCP timers and retry counts have expired, so it can't be reported as an exception immediately.
  • 12. Re: ObjectInputStream.readObject() hangs forever during socket communication
    911421 Newbie
    Currently Being Moderated
    thanks. the specific value of kernel's internal TCP timers* is located at /proc/sys/net/ipv4? do you know which one?

    and i am very strange that what causes the Exception "java.net.SocketException: Connection timed out" ?

    if the socket or connection is broken, why not throw: " java.net.SocketException: Connection reset "
    or "java.net.SocketException: Broken Pipe", or other exceptions.

    it is said that "java.net.SocketException: Connection timed out" means that the TCP connection
    could not be established due to a timeout. Actually, in our case, the socket is not closed using socket.close method,and its timeout value is kept its default value:0 which means infinite limit.

    The 'connection timed out' often means that IP or Port is occupied, is there any possible that the port of Client socket is occupied because some Pc network configuring policy? Or is is possible that the port would be closed automatically because of a long time no received response?

    Edited by: shawn on 2012-2-2 上午1:30
  • 13. Re: ObjectInputStream.readObject() hangs forever during socket communication
    EJP Guru
    Currently Being Moderated
    thanks. the specific value of kernel's internal TCP timers* is located at /proc/sys/net/ipv4? do you know which one?
    It's not as simple as 'one'. That's why I said 'timers and retry counts', both plural. There is a retry count, a modeled RTT timer per connection, a retransmit timeout parameter, etc. When TCP has decided the remote peer is dead based on a combination of factors it will put the socket into the error state. Don't start thinking that there is a single thing there that you can just tweak. Write timeouts are caused by problems at the other end, or in the network.
    and i am very strange that what causes the Exception "java.net.SocketException: Connection timed out" ?
    I agree, and I have already said so. It is extremely strange. You should never see that error on an established connection.

    It is so strange that I am wondering whether you are observing it correctly. The only time I've ever seen anything like this was on the first write to a localhost connection when the target wasn't running, and it's about 20 years ago so I have no idea whether current TCP stacks still behave like that. Was there any prior I/O on this connection, or is this the first? And is this connection to the localhost, or a remote host?
    if the socket or connection is broken, why not throw: " java.net.SocketException: Connection reset "
    or "java.net.SocketException: Broken Pipe", or other exceptions.
    It does, of course.
    Actually, in our case, the socket is not closed using socket.close method, and its timeout value is kept its default value:0 which means infinite limit.
    As I said above in my first reply, this is irrelevant. The socket timeout is a read timeout. Connection timeouts are specified in java in the connect() method, and in any case they are never infinite: about a minute is both the default and the maximum. But if you weren't doing a connect when you got the connect timeout this is also irrelevant.
    The 'connection timed out' often means that IP or Port is occupied
    No it doesn't. It never means that. It means no response was received from the remote peer when connecting. If you weren't connecting this is also irrelevant.
    is there any possible that the port of Client socket is occupied because some Pc network configuring policy?
    Doubly irrelevant. If the port was 'occupied' you wouldn't have got the connection at all.
    Or is is possible that the port would be closed automatically because of a long time no received response?
    An intervening firewall might close the connection after a longish interval of no traffic. Nobody closes ports except the application (and the OS when the application exits).
  • 14. Re: ObjectInputStream.readObject() hangs forever during socket communication
    911421 Newbie
    Currently Being Moderated
    thanks very much.
    EJP wrote:
    It is so strange that I am wondering whether you are observing it correctly. The only time I've ever seen anything like this was on the first write to a localhost connection when the target wasn't running, and it's about 20 years ago so I have no idea whether current TCP stacks still behave like that. Was there any prior I/O on this connection, or is this the first? And is this connection to the localhost, or a remote host?
    The original codes have added the try/catch to the socket or Object*putStream, if there are other exceptions, they would be logged to the log files. the connection used in my application is to a remote host, and both peers are running.
    The 'connection timed out' often means that IP or Port is occupied
    No it doesn't. It never means that. It means no response was received from the remote peer when connecting. If you weren't connecting this is also irrelevant.
    Yes, and the message is written to an established connection, and this connection may be still valid, because there are no other exceptions.
    An intervening firewall might close the connection after a longish interval of no traffic. Nobody closes ports except the application (and the OS when the application exits).
    If the connection is closed, why the no other exceptions are detected to notify the application. I think the firewall on the network may be one of the cause. And because the result of a computing task costing little time could be normally responded to the client, I suspect that the cause may be at the environment or network, i can not be sure whether there are routers or firewall PC would drop the messages, or automatically close the transferring port.
1 2 Previous Next

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points