This discussion is archived
1 2 Previous Next 28 Replies Latest reply: Sep 9, 2012 10:43 PM by EJP RSS

Jre7 Socket problem

960486 Newbie
Currently Being Moderated
Hello All,

I am experiencing a strange problem while using the jre7 for an application. This application is an application which creates an SSL socket to connect to a server. Sending and receiving short strings using a Buffered(Writer/Reader) works without any problem but when I want to send a long( 200+ characters) Base64 encoded string something really weird happens:

e.g. I want to send the following:

TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIGFkaXBpc2NpbmcgZWxpdC4g
SW50ZWdlciBpZCBpcHN1bSBhIGVzdCBlbGVtZW50dW0gc3VzY2lwaXQgbmVjIGVsZWlmZW5kIGZl
bGlzLiBQaGFzZWxsdXMgc2VkIGxlY3R1cyBpZCB0b3J0b3IgY29uc2VjdGV0dXIgcG9ydHRpdG9y
Lg==

This arrives at the server as follows:

T00000000000000000000000000000000000000000000000000000000000000000000000000
0000000000000000000000000000000000000000000000000000000000000000000000G9yZW
0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIGFkaXBpc2NpbmcgZWxpdC4gSW50ZWd
lciB

For some reason there are nulls included and the last piece is discarded (if you send 512 bytes you receive 512 bytes at the other end).

What is causing this? is this a problem in the JRE version 7, did something change or is this something I do wrong? The code has been working fine for years with jre 6 versions.


Thanks in advance!
  • 1. Re: Jre7 Socket problem
    sabre150 Expert
    Currently Being Moderated
    957483 wrote:
    What is causing this?
    You most probably have a bug in your code but without a view of your code it is difficult to tell what is likely to be wrong.
  • 2. Re: Jre7 Socket problem
    jtahlborn Expert
    Currently Being Moderated
    In most cases like this, it is an application error. i would recommend showing the sending/receiving code here.
  • 3. Re: Jre7 Socket problem
    960486 Newbie
    Currently Being Moderated
    Here is the send code (It works with jre6 but not with jre7):

    BUFFER_SIZE is 512, any larger will give problems while sending to mobile devices.
    public boolean send( byte[] data ) {
        boolean sent = false;
        int bytesRemaining = 0;
        int length = data.length;
        int offset = 0;
        int toSend = 0;
        OutputStream outputStream = null;
    
        bytesRemaining = length;
    
        System.out.println( String.format( "Start sending %d bytes", length ) );
              
        try {
            outputStream = socket.getOutputStream();
    
            while ( bytesRemaining > 0 ) {
                toSend = ( bytesRemaining > BUFFER_SIZE ? BUFFER_SIZE : bytesRemaining );
    
                System.out.println( String.format( "Sending %d bytes, offset: %d bytes, remaining: %d bytes", toSend, offset, bytesRemaining ) );
                        
                outputStream.write( data, offset, toSend );
                        
                outputStream.flush();
    
                offset += toSend;
                        
                bytesRemaining -= toSend;
            }
    
            System.out.println( String.format( "Finished sending %d bytes", length ) );
                   
            sent = true;
        } catch ( Exception ex ) {
            Logger.getLogger( SocketHandler.class.getName() ).log( Level.SEVERE, null, ex );
        }
    
        return sent;
    }
    It takes a byte array as argument, this is usually a Base64 encoded zip archive with one or more entries (200 - 3000 bytes). This is broken up into pieces of BUFFER_SIZE and send to the other side where it is assembled and processed. As said, it works in jre6 but there are null characters when using jre7.

    Receiving these encoded files using jre7 works fine.

    Edited by: 957483 on Sep 6, 2012 11:02 PM
  • 4. Re: Jre7 Socket problem
    DrClap Expert
    Currently Being Moderated
    Did you consider the possibility that the data you are sending already had the nulls in it before you sent it?

    Also, I'm no networking expert, but I don't see how your sending loop differs from just these two lines with no loop:
    outputStream.write( data);
    outputStream.flush();
    But I don't expect that's the source of the problem.
  • 5. Re: Jre7 Socket problem
    sabre150 Expert
    Currently Being Moderated
    DrClap wrote:
    Did you consider the possibility that the data you are sending already had the nulls in it before you sent it?
    ++

    >
    Also, I'm no networking expert, but I don't see how your sending loop differs from just these two lines with no loop:
    outputStream.write( data);
    outputStream.flush();
    ++

    >
    But I don't expect that's the source of the problem.
    ++
  • 6. Re: Jre7 Socket problem
    960486 Newbie
    Currently Being Moderated
    Yes, i did check if the data already contained nulls. It did not, it is magicaly added and received by the receiving side.I have traced it all the way to this function.

    I have tried to use BufferedWriter and BufferedReader as weell as sending it byte for byte but no luck, the latter makes things even worse.

    The receiving size alsi reports receiving correctly sized blocks but it e.g contains one character followed by null characters the following block contains the remaining part. received bytes also reports the block size.
  • 7. Re: Jre7 Socket problem
    sabre150 Expert
    Currently Being Moderated
    957483 wrote:
    Yes, i did check if the data already contained nulls. It did not, it is magicaly added and received by the receiving side.I have traced it all the way to this function.
    Sorry but something does not make sense. You say the zeros are "magicaly added and received by the receiving side" but you only show code for the transmitting side! What am I missing?

    Annanotherthing :- Each of those added zeros are are the ASCII character '0' and not the byte 0x00 . It seems more likely to me that in Java bytes are filled with 0x00 (the default when creating byte arrays) than with '0' so one more indication that they are added prior to encoding or as part of the encoding. Which Base64 encoder are you using and how?
  • 8. Re: Jre7 Socket problem
    960486 Newbie
    Currently Being Moderated
    Well... what you are missing is the same as what I am missing. The code works flawless when using JRE6, except in JRE7. Ive checked the input to this function and it does not contain any null bytes. I use the Apache commons codec Base64 class to encode/decode the Base64 encoded data.

    Ive added the receiving code below:
         public byte[] receive( int nrOfBytesToReceive ) {
              byte[] data = null;
              byte[] buffer = null;
              int offset = 0;
              int received = 0;
              int remaining = nrOfBytesToReceive;
              int toReceive = 0;
              InputStream inputStream = null;
    
              try {
                   inputStream = socket.getInputStream();
    
                   System.out.println( String.format( "Start receiving %d bytes", nrOfBytesToReceive ) );
                   
                   while ( remaining > 0 ) {
                        toReceive = ( remaining > BUFFER_SIZE ? BUFFER_SIZE : remaining );
    
                        buffer = new byte[toReceive];
    
                        received = inputStream.read( buffer );
    
                        if ( data == null ) {
                             data = new byte[nrOfBytesToReceive];
                        }
                        
                        System.out.println( String.format( "Received %d bytes, offset %d bytes, remaining: %d bytes", received, offset, remaining ) );
                        
                        Util.print( buffer, received );
                        
                        System.arraycopy( buffer, 0, data, offset, received );
    
                        offset += received;
    
                        remaining -= received;
                   }
                   
                   System.out.println( String.format( "Finished receiving %d bytes", nrOfBytesToReceive ) );
              } catch ( Exception ex ) {
                   Logger.getLogger( SocketHandler.class.getName() ).log( Level.SEVERE, null, ex );
              }
    
              return data;
         }
    "received" is the number of bytes received and this corresponds to the number of bytes sent. Util.print() prints the individual bytes.
  • 9. Re: Jre7 Socket problem
    EJP Guru
    Currently Being Moderated
    You can replace the entire can of worms with:
    public byte[] receive( int nrOfBytesToReceive ) throws IOException {
        byte[] buffer = new byte[nrOfBytesToReceive];
        new DataInputStream(socket.getInputStream()).readFully(buffer);
        return buffer;
    }
  • 10. Re: Jre7 Socket problem
    sabre150 Expert
    Currently Being Moderated
    957483 wrote:
    Well... what you are missing is the same as what I am missing.
    Not quite - we are missing access to the whole of your code. I use Java sockets all the time with both 1.6 and 1.7 and see nothing like this at all. I find it hard to believe that you have found what would be a very serious bug in the Socket library that nobody else has found and I still think the problem is in your code. The significant redundancy commented on by others in both your send and receive just add to my certainty that the problem is in your code.

    You have not commented on my observation that the added bytes are the character '0' and not 0x00 . Since sockets are inherently binary and know nothing about characters this still indicates to me that the problem is either before or during Base64 encoding. Since you are using Apache commons codec I am more inclined to think that the problem is before encoding.

    Edited by: sabre150 on Sep 7, 2012 1:13 PM

    I'm at a loss as to why you need to Base64 encode since the sample data in your original post decodes to
    "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Integer id ipsum a est elementum suscipit nec eleifend felis. Phasellus sed lectus id tortor consectetur porttitor." . It is rare to even need to Base64 encode binary data when sending it over sockets and even more rare to need to Base64 encode character data.

    Edited by: sabre150 on Sep 7, 2012 1:31 PM

    Thinking about this a bit more. I speculate that you are reading the data to be sent from a file and are assuming that FileInputStream.read(byte[] ) always reads all the bytes.
  • 11. Re: Jre7 Socket problem
    960486 Newbie
    Currently Being Moderated
    This code has been working for more than three consecutive years without any problem, all java 1.6 versions work without any problems. It is only since java 1.7 which started to give problems.

    My first example is an example of how I receive data, in this example the data is indeed text, but like i said in the beginning, in real life it contains a zip archive with one to n images. It is obvious that this is a waste of forum space and would certainly not contribute anything to my explanation, therefore I used a piece of lorem ipsum in the example. What I tried to explain in my first post is that when I send the data the receiving side first receives the first character followed by 0s then the second block received contains the remainder of the first part.

    For example when you have a transfer of 1280 bytes you will have a transfer of two blocks of 512 bytes and one of 256 bytes:

    Iteration
    1 512
    2 512
    3 256

    so the receiving end should receive this in the same order as well, if the receiver accidentally overwrites its send buffer would never be able to send the first byte followed by 511 zeroes.

    I cannot send it as a whole stream because the receiving end is mostly on a mobile network (2/3g) and some of these networks break the connection for unknown reasons. Breaking the whole stream in blocks of 512 bytes solved this problem and made the whole connection stable, therefore the redundancy in the code. For the sockets this does not matter, before sending the data I send a command which specifies the size of the data which will be sent so the receiving end always knows how much data to read before the transfer is complete.

    To complete the picture:

    Sender <----> Relay <----> Receiver

    The reason for the relay is because the receiver is often behind a firewall with a private ip range and the need of authentication.

    The Base64 encoding just makes it easy to debug, to send and receive, to convert to a character data etc, it works fine and is impossible to change and most certainly is not causing the problem.

    Data is generated through a program, put in a zip archive, encoded and send to the receiver. When the receiver sends similar data to the sender it is received correctly ( the receivers are still java 1.6)

    I will have a very good look later to see whether your advice about the encoded data is indeed wrong, i tested it earlier and that seemed to be fine.
  • 12. Re: Jre7 Socket problem
    sabre150 Expert
    Currently Being Moderated
    957483 wrote:
    This code has been working for more than three consecutive years without any problem, all java 1.6 versions work without any problems.
    Not really relevant if you are relying on something that is not guaranteed.
    It is only since java 1.7 which started to give problems.

    My first example is an example of how I receive data, in this example the data is indeed text, but like i said in the beginning, in real life it contains a zip archive with one to n images. It is obvious that this is a waste of forum space and would certainly not contribute anything to my explanation, therefore I used a piece of lorem ipsum in the example. What I tried to explain in my first post is that when I send the data the receiving side first receives the first character followed by 0s then the second block received contains the remainder of the first part.

    For example when you have a transfer of 1280 bytes you will have a transfer of two blocks of 512 bytes and one of 256 bytes:

    Iteration
    1 512
    2 512
    3 256
    So how do you read these blocks? I envisage you assuming that a InputStream.read(byte[]) or InputStream.read(byte[], int offset, int len) read the number of bytes requested. Since the contract for these does not guarantee this and that these methods return the number of bytes actually read then this is a strong candidate for the source of your problem. The blocks of '0' chars makes me believe that is is exactly what you are assuming. Seeing the code would help.

    If your code reading the data is anything like the following (based on what you have published so far) then it accounts for the symptoms you are seeing -
           // VERY VERY VERY flawed code
            // VERY VERY VERY flawed code
            // VERY VERY VERY flawed code
            // VERY VERY VERY flawed code
            // VERY VERY VERY flawed code
            // VERY VERY VERY flawed code
            // VERY VERY VERY flawed code
            // VERY VERY VERY flawed code
            final int BUFFER_SIZE = 512;
            File f = new File("some file");
            long bytesRemaining = f.lastModified();
            InputStream is = new FileInputStream(f);
            while (bytesRemaining > 0)
            {
                int toRead = (bytesRemaining > BUFFER_SIZE ? BUFFER_SIZE : (int) bytesRemaining);
                byte[] buffer = new byte[toRead];
                is.read(buffer); // This is where the major flaw is
                send(buffer);
                bytesRemaining -= toRead;
            }
    >
    so the receiving end should receive this in the same order as well, if the receiver accidentally overwrites its send buffer would never be able to send the first byte followed by 511 zeroes.

    I cannot send it as a whole stream because the receiving end is mostly on a mobile network (2/3g) and some of these networks break the connection for unknown reasons. Breaking the whole stream in blocks of 512 bytes solved this problem and made the whole connection stable, therefore the redundancy in the code. For the sockets this does not matter, before sending the data I send a command which specifies the size of the data which will be sent so the receiving end always knows how much data to read before the transfer is complete.

    To complete the picture:

    Sender <----> Relay <----> Receiver

    The reason for the relay is because the receiver is often behind a firewall with a private ip range and the need of authentication.

    The Base64 encoding just makes it easy to debug, to send and receive, to convert to a character data etc, it works fine and is impossible to change and most certainly is not causing the problem.
    I can envisage one possible weakness with this but for the moment it would just cloud the main issue.

    >
    Data is generated through a program, put in a zip archive, encoded and send to the receiver. When the receiver sends similar data to the sender it is received correctly ( the receivers are still java 1.6)

    I will have a very good look later to see whether your advice about the encoded data is indeed wrong, i tested it earlier and that seemed to be fine.
    Edited by: sabre150 on Sep 7, 2012 4:34 PM
  • 13. Re: Jre7 Socket problem
    960486 Newbie
    Currently Being Moderated
    I am not returning anything from a file, it is generated in memory using:
    byte[] data = null;
    ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
    ZipOutputStream zipOutputStream = new ZipOutputStream( byteArrayOutputStream );
    
    while() {
    //add entries
    }
    
    zipOutputStream .close();
    byteArrayOutputStream .close();
    
    data = byteArrayOutputStream .toByteArray();
    
    data =  Base64.encodeBase64( data );
    
    send( data ); //See first code example above
    Everything above send() works perfect on Java 1.6 and Java 1.7.

    In your example, in.read(buffer) does indeed never guarantee to fill the buffer completely BUT it returns the actually read bytes which is guaranteed so you can always reliably read from an input stream. You can see that in my read( nrofbytes) function above, this is to read from a socket. It would indeed be flawed code if i did use the actual byte array length.
  • 14. Re: Jre7 Socket problem
    jtahlborn Expert
    Currently Being Moderated
    957483 wrote:
    Sender <----> Relay <----> Receiver
    do you control the relay code? if so, what does that look like?
1 2 Previous Next

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points