This discussion is archived
8 Replies Latest reply: May 14, 2012 8:24 PM by 934824 RSS

Infamous SocketException: Broken Pipe

934824 Newbie
Currently Being Moderated
Hello,

As with many of the other users on this forum I have been plagued by the Broken Pipe SocketException, very rarely, which seems to essentially block on the java.net.SocketOutputStream.socketWrite0(Native Method) for quite a long time (presumably the length of some timeout) during a flush. It is also typically accompanied by a "java.net.SocketException: Connection timed out" on the socket itself.

I've read the many responses by EJP that this means:

"This is caused by writing to a connection when the other end has already closed it.

So you have a poorly defined or implemented application protocol."

My main question is: Is it normal for these exceptions to occur, and we just need to write some layer that handles this?

And a follow up: If so, would that layer be something along the lines of:
- a separate thread on the server for each output stream, and if this thread blocks during flush (for whatever reason), either:
- use a watchdog thread to close the output if some internal timeout has exceeded, or:
- just let Java's connection timeout eventually occur and let the output thread die

Or am I way off base here?

One more thing: Any other advice / best architectural practices / help regarding this problem / reading suggestions would be extremely appreciated.

Thanks.
  • 1. Re: Infamous SocketException: Broken Pipe
    tschodt Pro
    Currently Being Moderated
    931821 wrote:
    My main question is: Is it normal for these exceptions to occur?
    Depends.

    If your applications arbitrarily use [url http://docs.oracle.com/javase/6/docs/api/java/net/Socket.html#close%28%29]Socket.close() when a socket connection is no longer interesting, yes, you will likely get these exceptions.

    But if all your applications use [url http://docs.oracle.com/javase/6/docs/api/java/net/Socket.html#shutdownOutput%28%29]Socket.shutdownOutput() when they are done with a socket and only use [url http://docs.oracle.com/javase/6/docs/api/java/net/Socket.html#close%28%29]Socket.close() in response to an EOS condition
    you should only see these exceptions if an application terminates abruptly.
    Any other advice / best architectural practices / help regarding this problem / reading suggestions would be extremely appreciated.
  • 2. Re: Infamous SocketException: Broken Pipe
    EJP Guru
    Currently Being Moderated
    I don't know what 'plagued very rarely' means, or what you mean by 'accompanied', as you can't get two exceptions at the same time, but it isn't 'normal', it is caused by errors in your application. That's what's meant by 'application protocol error'. You have written something that the other end isn't consuming. Your client and server are out of whack. It's a bug.

    EDIT: tschodt's suggestion will certainly bullet-proof you against this problem but it isn't really necessary, it's just necessary to consume all the output that the peer is sending before closing. Or, don't send it ;-)
  • 3. Re: Infamous SocketException: Broken Pipe
    934824 Newbie
    Currently Being Moderated
    Thanks for the replies. By rarely I mean with hundreds of active concurrent connections, I get a 'broken pipe' exception only once every few days. I have never been able to reproduce this in my development environment.

    My setup is 1 thread per socket connection, and then I wrap the socket Input/OutputStreams like so:

    DataOutputStream out_ = new DataOutputStream(new BufferedOutputStream(socket_.getOutputStream()));

    this "out_" gets data packed into it and flush()'d by a main thread that serves all the connections. I have a try/catch block around out_.flush(). This occasionally blocks for a very long time returning the following eventually:

    java.net.SocketException: Broken pipe
         at java.net.SocketOutputStream.socketWrite0(Native Method)
         at java.net.SocketOutputStream.socketWrite(Unknown Source)
         at java.net.SocketOutputStream.write(Unknown Source)
         at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
         at java.io.BufferedOutputStream.flush(Unknown Source)
         at java.io.DataOutputStream.flush(Unknown Source)

    By "accompanied" I mean the Connection Timed Out exception happens on the "in_" side of the socket at the same time as the broken pipe on the "out_" side.

    I see what you are saying: Do not flush send data to the client when the connection is closed. So are you saying I could simply check if (socket_.isClosed()) before attempting to flush() or packByte() and this would vanish? Can the client not disconnect in the middle of a flush without informing the server? (say, power failure on client end) Would this not cause the flush() to block still?

    If flush() cannot be guaranteed not to block (under the above circumstances) it would seem to me that these flush()es and packByte()s must be on their own thread (per connection) rather than on the main thread.

    Thanks for your help. I feel like I am near a solution here.

    Edited by: Jesse on May 4, 2012 9:24 AM
  • 4. Re: Infamous SocketException: Broken Pipe
    EJP Guru
    Currently Being Moderated
    No. It is rare for any question beginning 'so you are saying' to have the answer 'yes'. So rare that I cannot remember a single example here in fifteen years on this forum. And this case is no exception.

    Socket.isClosed() only tells you whether you have closed the socket. It doesn't (can't) tell you whether the peer has closed the connection. The only way you can discover that is by reading an EOS or getting a connection reset/broken pipe when writing. I will say once more that writing data when the other end has closed the connection is a bug in your code. You have misimplemented your applcation protocol. Or else you are just streaming large amounts of data and the peer giving up is part of the application protocol (for example, the peer is a browser and the user navigates away), in which case the correct action is to catch the exception and give up on the client. If you are the client, maybe you've hit an upload limit at the server. But if the peer is supposed to receive what you are sending, there is a bug somewhere, and trying to fix it with bandaids as you suggest is not the answer.

    There is something seriously wrong with your statement that the peer gets a connection timeout when this side gets a broken pipe. Connection timeout happens when creating a connection: in fact it happens instead of creating a connection. So there is no connection yet, so no peer at all, so it is impossible for the other end to be writing to the connection at all, because there is no connection and no other end. Do you mean a read timeout?
  • 5. Re: Infamous SocketException: Broken Pipe
    934824 Newbie
    Currently Being Moderated
    EJP wrote:
    Or else you are just streaming large amounts of data and the peer giving up is part oft the application protocol(for example, the peer is a browser and the user navigates away), in which case th correct action is to catch the exception and give up on the client.
    I believe this might be the case. The clients can (and typically do) receive the information I am sending. It is absolutely possible that they can navigate away during the transfer. It is fine for the purposes of the application to give up on this client. To give up on the client and not have flush() block my main thread, I would need a separate thread per client, for handling the outputstream, correct? ie. one for receive, another for send, per client.
    There is something seriously wrong with your statement that the peer gets a connection timeout when this side gets a broken pipe. Connection timeout happens when creating a connection: in fact it happens instead of creating a connection. So there is no connection yet, so no peer at all, so it is impossible for the other end to be writing to the connection at all, because there is no connection and no other end. Do you mean a read timeout?
    I get:

    ThreadID-1310 java.net.SocketException: Connection timed out
    ThreadID-1310 java.net.SocketException: Broken pipe

    On the server-side thread that holds the socket for this client. The Connection timed out happens from the Socket InputStream during a readByte(), Broken pipe happens from the Socket OutputStream during a flush().

    Init code:
    socket_ = serverSocket_.accept();
    in_ = new DataInputStream(new BufferedInputStream(socket_.getInputStream()));
    out_ = new DataOutputStream(new BufferedOutputStream(socket_.getOutputStream()));
    mainThread.add(this); // the output main thread with some synchronization
    start(); // the read thread below
    Receive in the thread that throws Connection Timed out:
    run() { 
      while(running) {
        try { 
          byte index = in_.readByte();
        } catch (SocketException e) {
          System.err.println(getId() + ": " + e);
        }
      }
    }
    Send code in main thread that throws broken pipe:
        try { 
          callingThread_.out_.flush()
        } catch (SocketException e) {
          System.err.println(callingThread_.getId() + ": " + e);
        }
      }
    Both exceptions happen back to back. The Main thread keeps a back-pointer to the calling thread As you can see, both in_ and out_ are from the same socket on the original object. The user has absolutely been connected for a while before this occurs.

    Thanks again for the insight.
  • 6. Re: Infamous SocketException: Broken Pipe
    EJP Guru
    Currently Being Moderated
    When you get any exception on a socket other than a SocketTimeoutException when reading, you must close the socket. I don't see your code doing that.

    I don't get the part about the back pointer or the calling thread but I suggest you print out the socket itself (Socket.toString()) along with the exception. That will tell you for sure what socket is involved, without relying on your application to hook itself together properly. I also suggest you print the stack trace, not just the message. You might get a surprise.

    What platform are you running on?
  • 7. Re: Infamous SocketException: Broken Pipe
    934824 Newbie
    Currently Being Moderated
    Oh, I'm sorry I should have mentioned I trimmed down the code I pasted here for brevity. But, that's the jist. The current implementation actually does close the socket for all exceptions, though you are right it should definitely allow SocketTimeoutException without shutting down the socket as I have certainly disconnected people from the application even when they just had a brief interruption in connectivity. It also prints the stack trace.

    I would have posted the entirety of the code, but, honestly it is a lot of code and wouldn't want to impose. I'm sure it would be time consuming to troubleshoot, though you are obviously very knowledgeable on the subject and I would welcome your help if you had the time.

    The server is CentOS release 5.8 (Final) linux distro.

    Do you think, then, that 2 threads per client, one for 'input' one for 'output', so that even if a thread dies, it does not disrupt the main server, sounds reasonable?
  • 8. Re: Infamous SocketException: Broken Pipe
    934824 Newbie
    Currently Being Moderated
    To give this some closure: I put the 'sends' inside their own thread (each socket has 1 thread for incoming and 1 for outgoing traffic) and then I simply allow the broken pipes to occur and close the connection when they do. This fixed my problem and server has been running stable with over 50,000 active users for a whole week!

    Thanks for all the helpful responses.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points