This discussion is archived
9 Replies Latest reply: May 16, 2013 11:12 AM by 810618 RSS

Behaviour when experiencing packet loss

1008861 Newbie
Currently Being Moderated
I'm writing this as a none Java expert but as someone interested in solving this particular 'problem' so please ask any pertinent questions and I'll provide answers if my initial post is lacking in detail.

We had an issue recently when one of our internet links was experiencing up to 10% packet loss. As a consequence of various timeouts and out of order packets the remote host closed the connection and subsequent requests from the client were responded to with a TCP RST. The thing that's confusing at the moment was that, upon receiving the RST, the client would respond with another request devoid of all HTTP headers ending with a 'Bad Request' at the server end.

This is a standalone Java program sending simple SOAP requests to an upstream server. We've seen this behaviour when using versions 1.6.0_26, 1.6.0_41 and 1.7.0_21.

To simplify:

1 - connection
2 - RST seen
3 - client resends **without anything coded to specify this**

Edited by: 1005858 on May 14, 2013 8:33 AM
  • 1. Re: Behaviour when experiencing packet loss
    jtahlborn Expert
    Currently Being Moderated
    it's possible that you are hitting this awful java "feature" that HttpURLConnection will automatically retry a post if it receives a response which is not parseable as a valid http response. we discovered this little gem ourselves only recently. you can disable this "feature" using the "sun.net.http.retryPost" networking property, detailed here:

    http://docs.oracle.com/javase/6/docs/technotes/guides/net/properties.html
  • 2. Re: Behaviour when experiencing packet loss
    EJP Guru
    Currently Being Moderated
    A truly appalling feature. What about idempotence? Why isn't it documented properly in HttpURLConnection where it belongs?
  • 3. Re: Behaviour when experiencing packet loss
    810618 Newbie
    Currently Being Moderated
    Hmmm, this is interesting.

    [http://www.coderanch.com/t/490463/sockets/java/Timeout-retry-URLHTTPRequest] talks about it some more.

    I am unable to recreate this (using 1.6.0_45). In fact, I get a null with:

         System.out.println(System.getProperty("sun.net.http.retryPost"));

    So I explicitly set it at startup time:

    -Dsun.net.http.retryPost=true

    And still nothing. I forced three conditions: read timeout, connection refused (wrong port), and IO Exception (I guess - I just shut the server off?). I also tried one where I sent back an undefined response code.

    I could see the RST's but no retry after them from the client.

    I'm obviously missing something (by the fact that I don't even see that property).

    Can someone let me know what I'm doing wrong in my attempt to recreate this?
  • 4. Re: Behaviour when experiencing packet loss
    gimbal2 Guru
    Currently Being Moderated
    KevinRyan wrote:
    -Dsun.net.http.retryPost=true
    don't you mean false, to disable it?
    And still nothing.
    Still nothing what? The property is still not set, or you simply get the same results?


    EDIT: wait a minute, you're not even the OP :s
  • 5. Re: Behaviour when experiencing packet loss
    EJP Guru
    Currently Being Moderated
    I am unable to recreate this (using 1.6.0_45). In fact, I get a null with:

         System.out.println(System.getProperty("sun.net.http.retryPost"));
    Did somebody say it would be set? Most system properties don't work like that. The variables into which they are read have default values.
  • 6. Re: Behaviour when experiencing packet loss
    810618 Newbie
    Currently Being Moderated
    gimbal2 wrote:
    KevinRyan wrote:
    -Dsun.net.http.retryPost=true
    don't you mean false, to disable it?
    No, I mean true to enable it - I'm trying to recreate the problem.

    >
    And still nothing.
    Still nothing what? The property is still not set, or you simply get the same results?


    EDIT: wait a minute, you're not even the OP :s
    Sorry, I didn't intend to hijack this thread - I just wanted to understand the problem which might even had helped the OP.

    Edited by: KevinRyan on May 16, 2013 9:53 AM
  • 7. Re: Behaviour when experiencing packet loss
    810618 Newbie
    Currently Being Moderated
    EJP wrote:
    I am unable to recreate this (using 1.6.0_45). In fact, I get a null with:

         System.out.println(System.getProperty("sun.net.http.retryPost"));
    Did somebody say it would be set? Most system properties don't work like that. The variables into which they are read have default values.
    The link that jtahlborn provided discusses this and says the property defaults to 'true'. Since the property value was null, I would assume it is behaving as you describe and defaulting the variable itself. However, in order to try to recreate this problem, with as little unknowns as possible, I explicitly set the property to true.

    The thing that I am trying to determine is why I could not get my system to behave as the OP's and jtahlborn's. I could not get it to retry the request even though I could see the RST in the exchange.
  • 8. Re: Behaviour when experiencing packet loss
    jtahlborn Expert
    Currently Being Moderated
    KevinRyan wrote:
    The thing that I am trying to determine is why I could not get my system to behave as the OP's and jtahlborn's. I could not get it to retry the request even though I could see the RST in the exchange.
    what did your client code look like?
  • 9. Re: Behaviour when experiencing packet loss
    810618 Newbie
    Currently Being Moderated
    jtahlborn wrote:
    what did your client code look like?
    The test snippet:
        public static void main(String[] args) {
         System.out.println(System.getProperty("sun.net.http.retryPost"));
         String response;
         try {
             response = HttpClientLink.query
              ("http://mccctest.evla.nrao.edu:8899/test",
               "Now is the time for all good men to come to the aid\n");
         }catch(HttpLinkException e) {
             response = e.getMessage();
         }
         System.out.println("response: " + response);
    
         for(int i=0; i<100; i++) {
             Log.info("tick: " + new java.util.Date());
             try{Thread.sleep(10000);}catch(InterruptedException e){}
         }
        }
    }
    Where the HttpURLConnection is done:
        public static String query(String urlString, String query, 
                             int readTimeout) throws HttpLinkException {
         URL url = null;
         boolean windowOpen = window != null && window.isVisible();
         try {
                url = new URL(urlEncode(urlString));
             String ip = url.getHost();
             int port = url.getPort();
                HttpURLConnection con = (HttpURLConnection)url.openConnection();
             con.setConnectTimeout(readTimeout);
             con.setReadTimeout(readTimeout);
    
                StringBuffer responseBuf = new StringBuffer();
    
                if (query != null) {
              con.setRequestMethod("POST");
              con.setDoOutput(true);
              try {
                  OutputStream out = con.getOutputStream();
                  out.write(query.getBytes());
                  out.flush();
                  out.close();
              }catch(IOException e) {
                  throw new IOException("Error writing POST text");
              }
             }//end (post method processing)
    
             if (con.getResponseCode() == con.HTTP_OK ||
              con.getResponseCode() == con.HTTP_ACCEPTED ||
              con.getResponseCode() == con.HTTP_CREATED) {
              DataInputStream in = new DataInputStream
                  (new BufferedInputStream(con.getInputStream()));
              byte[] buf = new byte[32768];
              int nBytes;
              while((nBytes = in.read(buf)) > -1) {
                  responseBuf.append(new String(buf, 0, nBytes));
              }
              in.close();
             }
             else {
              responseBuf.append("HTTP Error -- " +
                           String.valueOf(con.getResponseCode()) +
                           ": " + con.getResponseMessage() + "\n");
             }
                return responseBuf.toString();
            }catch(MalformedURLException e) {
                throw new HttpLinkException("HttpClientLink.query Exception: " +
                                    "Malformed URL: '" + url + "' -- " +
                                    e.getMessage());
            }
         catch(IllegalStateException e) {
                throw new HttpLinkException("HttpClientLink.query Exception: " +
                                    "Illegal State, Trying 'setDoOutput' " +
                                    "while already connected to url: '" +  url + 
                                    "' -- " + e.getMessage());
         }
         catch(java.lang.RuntimeException e) {
             /*
              * got this when http host = null
              */
                throw new HttpLinkException("HttpClientLink.query Exception: " +
                             "Illegal Argument (check IP Address)" +
                             "\n" + e.getMessage());
         }
            catch(IOException e) {
                throw new HttpLinkException
              ("HttpClientLink.query Exception: " + 
               "\nIO Failure with URL: '" + url + "'" +
               "\nquery: '" + query + "' -- \n" +
               e.getMessage() + 
               edu.nrao.widar.util.Log.getStackTrace(e, "widar"));
            }
        }

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points