1 2 Previous Next 22 Replies Latest reply: Mar 27, 2014 2:33 AM by EJP RSS

    Breaking the exact message- java network programming

    user10070648

      HI,

      My server (Java socket using multi-thread processing) is receiving the huge traffic continuously and then I'm in a problem while breaking the exact message before processing.

       

      Note : Each message has 2 byes header which labelled the length of the message and the socket connection type is persistence.

      Example receiving buffer (00 0A is length)

       

      00 0A xx xx xx.............00 0A xx xx ...............

       

      In brief,

      After reading the buffer, it's handing over to separate thread for breaking the message by its length. Once it's completed, the broken message will hander over to the processing.

       

       

      (Toggle Plain Text)

       

      READ_BUFF_SIZE = 10000;byte[] buff = new byte[READ_BUFF_SIZE]; in = new DataInputStream(clientSocket.getInputStream()); int readLen = in.read(buff,0,READ_BUFF_SIZE);

       

       

       

       

      The problem is that the next buffer sometimes does not start with message length then the message breaking is totally case me.

       

      The next buffer sometimes as bellow

       

      xx xx xx 00 0A xx xx xx............

       

       

      So let me advice to avoid this situation and how like VISA/MC centeral transactions severs have been handled such huge traffic (Asking only programming technology since it may help me ) ?

       

       

      Hope you understand my problem and waiting for your response.

       

      Thanks

        • 1. Re: Breaking the exact message- java network programming
          EJP

          > The problem is that the next buffer sometimes does not start with message length

           

          No it isn't. The problem is that you aren't reading the entire message in the first place, so there is something left over for next time that you are misinterpreting as a length word. InputStream.read() isn't guaranteed to fill the buffer you provide. See the Javadoc. It isn't guaranteed to transfer more than one byte. The API you need in this situation is DataInputStream.readFully().

          • 2. Re: Breaking the exact message- java network programming
            rp0428

            My server (Java socket using multi-thread processing) is receiving the huge traffic continuously and then I'm in a problem while breaking the exact message before processing.

             

            Note : Each message has 2 byes header which labelled the length of the message and the socket connection type is persistence.

            Example receiving buffer (00 0A is length)

            The proper way to read 'messages' depends on the exact message format you are using and you didn't post the details of that.

             

            EJP told you one possible issue but based on what you posted you are likely using the wrong method altogether. You appear to be trying to use DataInputStream and DataInput to read BINARY data or to read a combination of binary and string data (which it is isn't clear).

             

            Those classes, as the API states, are meant for reading 'Java Primitive Types'.

            http://docs.oracle.com/javase/7/docs/api/java/io/DataInputStream.html

            A data input stream lets an application read primitive Java data types from an underlying input stream in a machine-independent way

            http://docs.oracle.com/javase/7/docs/api/java/io/DataInput.html

            The DataInput interface provides for reading bytes from a binary stream and reconstructing from them data in any of the Java primitive types. There is also a facility for reconstructing a String from data in modified UTF-8 format.
            . . .
            Implementations of the DataInput and DataOutput interfaces represent Unicode strings in a format that is a slight modification of UTF-8. 

            Did you notice that last sentence: 'represent Unicode strings'?

             

            You don't appear to be working with 'Unicode strings'. You appear to be reading raw bytes for the length and then we don't know what you do. Perhaps you are trying to read the rest based on the 'length' you get. But is that length in 'bytes', 'characters' or what? As you can see from the API modified UTF-8 characters can be 1, 2 or 3 bytes in length.

             

            If you are using a binary message format then you need to use I/O methods that work with binary data.

             

            Post the details of the message format you are using, the code you use to 'create' messages in that format, and the code you are using to 'read' messages in that format.

            • 3. Re: Breaking the exact message- java network programming
              EJP

              > You appear to be trying to use DataInputStream and DataInput to read BINARY data

               

              That's what its read() and readFully() methods are for.

               

              > Those classes, as the API states, are meant for reading 'Java Primitive Types'.

               

              And binary data.

               

              > Did you notice that last sentence: 'represent Unicode strings'?

               

              Did you notice that that sentence applies to the writeUTF() method? and not for example to the writeInt() method?

               

              > If you are using a binary message format then you need to use I/O methods that work with binary data

               

              If DataInputStream.readFully() isn't for reading binary data, what is it for exactly?

               

              What on earth are you talking about?

              • 4. Re: Breaking the exact message- java network programming
                rp0428
                What on earth are you talking about?

                What I am talking about is that it all hinges on this:

                The proper way to read 'messages' depends on the exact message format you are using and you didn't post the details of that.

                I am talking about the difference between processing arbitrary binary data and processing 'formatted' binary data.

                A data input stream lets an application read primitive Java data types from an underlying input stream in a machine-independent way

                I consider those 'primitive Java data types' to be 'formatted' binary data. They have a very specific byte order and can comprise multiple bytes depending on the datatype.  That class was branched from its base class specifically for use with Java data types and the String class.

                 

                You can certainly mix raw binary data with 'formatted' binary data and ue the 'DataInputxxx' classes to deal with it but that is NOT the typical way to deal with raw binary data or data that uses a custom format. The reason is to minimize the types of problems that OP is having where they try to process raw bytes when the actual data is 'formatted' as a Java primitive or is part of a string.

                 

                The standard Inputxxx classes are the better choice for dealing with raw binary and custom formats.

                > Did you notice that last sentence: 'represent Unicode strings'?

                 

                Did you notice that that sentence applies to the writeUTF() method? and not for example to the writeInt() method?

                Not necessarily - I took that quote from the DataInput API intro:

                 

                Modified UTF-8

                Implementations of the DataInput and DataOutput interfaces represent Unicode strings in a format that is a slight modification of UTF-8.

                That is why the DataInputStream branch was created: to create a handler for the Java primitive types AND the String class. That combination can handle all of the basic data storage needs in a Java app.

                 

                OP appears to be dealing with length-delimited binary data. For that they need to read the length bytes and then read EXACTLY that many bytes. The interpretation of the bytes reads has to happen AFTER they are read. If that mini bytestream represents a series of Java primitives or a Java String then at that point you can use a DataInputxxx class/method to process that.

                 

                But the success of that still depends on how that actual length was determined and if OP was using the DataOutput classes to create the bytestream they likely are computing the length wrong: the lenght of a modified UTF-8 byte stream that represents a string is different from the length of a 'String' in Java that uses UTF16.

                • 5. Re: Breaking the exact message- java network programming
                  user10070648

                  I'm now using readFully() method but still the issue is there. Indeed, I'm receiving around 75000 incoming requests per second (Normally packets length is different and it's around 520 bytes  ).

                   

                  Further, I just increased the reading buffer size as well (READ_BUFF_SIZE = 200000) then I'm getting huge buffer with readFully()  but the same issue is still happening. 

                   

                  Is there any way to find out the position of length bytes (2 bytes) in next buffer ? then I can omit the starting few bytes.

                  can Java handle such a traffic ? any alternative ?


                  Send() and receive() method

                  [CODE]

                   

                  // socket input from server

                  protected DataInputStream serverIn;

                   

                  private int    READ_BUFF_SIZE       = 200000;

                   

                   

                  public byte[] receive() throws Exception {

                                  byte[] b = new byte[READ_BUFF_SIZE];

                               

                   

                                  try {

                                        

                   

                                          synchronized (serverIn) {

                                                

                                                       getMessage(b);

                                          }

                   

                              

                                  } catch (Exception e) {

                                          throwe e;

                                  }

                                  return b;

                          }

                   

                   

                  protected void getMessage(byte[] b)

                                          throws OException {

                                  serverIn.readFully(b);

                          }

                   

                  [/CODE]


                   

                  Sample raw packet

                   

                  01 33 60 00 00 00 95 08 10 20 38 01 00 0E C0 00 05 01 00 00 00 00 02 11 48 06 03 03 00 95 30 33 30 33 31 31 34 38 30 36 30 31 31 31 34 38 30 36 30 30 34 34 34 34 34 34 34 34 30 30 31 30 30 30 31 36 30 31 30 32 33 36 39 02 26 30 31 30 33 30 32 30 34 30 31 30 30 30 32 30 31 30 30 30 30 33 34 31 30 39 35 37 34 39 30 32 36 31 35 30 30 30 30 30 30 30 31 33 37 33 34 33 39 33 38 33 30 33 32 33 30 33 30 33 30 33 30 33 30 33 30 30 31 33 37 33 36 33 37 33 39 33 31 33 33 34 31 33 36 33 35 33 31 33 35 34 36 33 33 33 34 34 36 34 35 33 33 33 34 33 35 34 35 34 34 34 32 33 38 33 37 33 36 34 34 33 33 33 39 33 37 33 38 33 35 33 36 30 31 33 37 34 36 33 30 33 33 33 36 33 35 30 31 33 36 34 35 33 31 33 33 34 35 33 31 33 33 33 34 33 33 34 35 33 33 33 30 34 32 33 36 33 33 34 32 33 33 33 36 33 30 33 37 34 31 33 37 33 30 33 33 33 38 34 36 33 39 33 33 33 31 33 35 34 35 33 38 30 31 34 36 33 39 34 36 33 32 33 39 33 33 ED D6 72 FC D7 9E 47 86



                  • 6. Re: Breaking the exact message- java network programming
                    EJP

                    > For that they need to read the length bytes and then read EXACTLY that many bytes.

                     

                    And that's exactly what readIntI() and readFully() do, as I posted. I still don't know what you think you're talking about. The whole UTF issue only arises if the message was sent by Java, which there is zero evidence of, just as there is zero evidence that he's got the length wrong. If you mean that readUTF() should be used to read the result of writeUTF(), I agree, but there's no evidence of that either.

                    • 7. Re: Breaking the exact message- java network programming
                      EJP

                      You don't seem to have understood. You get the length of the incoming message with readInt(), and you then call readFully() supplying that length as the 'length' parameter. Supplying a fixed-sized buffer for variable length messages doesn't make sense.

                      • 8. Re: Breaking the exact message- java network programming
                        rp0428
                        And that's exactly what readIntI() and readFully() do, as I posted.

                        We aren't communicating well, are we?

                         

                        OP said the length was two bytes - you can NOT use 'readInt' to read it since a Jave int is four bytes.

                         

                        And the contract for 'readShort' is this:

                        the next two bytes of this input stream, interpreted as a signed 16-bit number.

                        That will ONLY work if the two bytes OP is using is really a signed 16-bit number and uses the proper byte order.

                         

                        I think you may be making a LOT of assumptions about the EXACT format that OP is using. I try not to make such assumptions so I ask OP to post that info.

                         

                        Once again, that is why I said this:

                        The proper way to read 'messages' depends on the exact message format you are using and you didn't post the details of that.

                        Yes, I agree with you: IF the OPs message format properly uses NOTHING but Java primitives and Java Strings produced by the DataOutputxxx classes then they can use the DataInputxxx classes to read back in.

                         

                        But that still requires that the length bytes contain the actual length in bytes of the message payload. And I don't want to assume OP got that part right. Because it seems to me that if OP was really using nothing but Java primitives and strings I don't see what value the 'length' in bytes would have other than to make sure the entire message was received.

                         

                        To parse the message you still need to know 'how many of what primitive/string' are in the message to you can read it properly.

                         

                        I'm pretty I understand the point you are trying to make but I don't think you understand mine.

                        • 9. Re: Breaking the exact message- java network programming
                          rp0428

                          I'm now using readFully() method but still the issue is there. Indeed, I'm receiving around 75000 incoming requests per second (Normally packets length is different and it's around 520 bytes  ).

                           

                          Further, I just increased the reading buffer size as well (READ_BUFF_SIZE = 200000) then I'm getting huge buffer with readFully()  but the same issue is still happening.

                           

                          Is there any way to find out the position of length bytes (2 bytes) in next buffer ? then I can omit the starting few bytes.

                          As EJP already told you the algorithm is REALLY simple:

                           

                          1. read TWO length bytes

                          2. allocate a buffer of that length (or reuse one, fixed-length buffer)

                          3. read 'length' bytes from the stream into the buffer

                          4. process the buffer contents to parse/handle that message

                          5. repeat steps 1 thru 4 until you are done

                           

                          If you are going to use only ONE buffer for all messages then you better make sure it is big enough for the largest message you can receive.

                           

                          And, personally, I'm still not convinced you have created the length bytes properly. That is why I wanted to see the code.

                          protected void getMessage(byte[] b)

                                                  throws OException {

                                          serverIn.readFully(b);

                                  }

                          You can ONLY use that 'readFully' method if your buffer is EXACTLY the right size for the message you are reading.

                          private int    READ_BUFF_SIZE       = 200000;

                          public byte[] receive() throws Exception {

                                          byte[] b = new byte[READ_BUFF_SIZE];

                          That buffer size of 200,000 and your use of 'readFully' means that you are NOT reading messages - you are just reading whatever the next 200000 bytes are in the stream.

                           

                          If you want to read 'messages', one at a time, you need to use that algorithm I show above.

                          • 10. Re: Breaking the exact message- java network programming
                            user10070648

                            serverIn.readInt() returns strange value (Example 303174162) even for telnet. Is thee any thing to be done ?

                             

                            public byte[] receive() throws Exception {

                                           byte[] b = null;

                                         

                             

                                            try {

                                                  

                             

                                                    synchronized (serverIn) {

                                                                 int bSize = serverIn.readInt();

                                                                 byte[] b = new byte[int bSize];

                                                                 getMessage(b,0,bSize);

                                                    }

                             

                                        

                                            } catch (Exception e) {

                                                    throwe e;

                                            }

                                            return b;

                                    }

                             

                             

                            protected void getMessage(byte[] b, int offset, int len)

                                                    throws OException {

                                            serverIn.readFully(b,offset,len);

                                    }

                            • 11. Re: Breaking the exact message- java network programming
                              user10070648

                              Thanks rp0428 and EJP,  I understood the steps you have given but my concern is that if some one sends the wrong request (with Incorrect length ), it's again continuing the wrong selection of packets.

                               

                              Is there any thing to add over here ?

                              • 12. Re: Breaking the exact message- java network programming
                                rp0428
                                I understood the steps you have given

                                No - I don't believe you do. readInt reads FOUR bytes. Your length is TWO bytes. You would use readShort to read those two bytes IF AND ONLY IF those two length bytes were written properly as a short to begin with. Since you STILL haven't posted the actual message format or the code to write and read it we can't help you.

                                if some one sends the wrong request (with Incorrect length ), it's again continuing the wrong selection of packets.

                                And that is why - if you 'understood the steps' you would know what will happen if someone uses the 'Incorrect length'.

                                Is there any thing to add over here ?

                                Not from me there isn't.

                                • 13. Re: Breaking the exact message- java network programming
                                  jschellSomeoneStoleMyAlias

                                  > if some one sends  the wrong request (with incorrect length)

                                   

                                  Basically you are out of luck then.

                                   

                                  However one possible way to handle this.

                                  - Read the message based on the length

                                  - Parse the content

                                  - If the content doesn't match your expectations CLOSE the socket.

                                   

                                  The result of that is that the other end must start over.

                                  • 14. Re: Breaking the exact message- java network programming
                                    user10070648

                                    Since you STILL haven't posted the actual message format or the code to write and read it we can't help you.

                                     

                                    -Already posted

                                     

                                    If the content doesn't match your expectations CLOSE the socket.

                                     

                                    -OK, but is they are any mechanism to clear DataInputStream IN rather than closing it ?

                                    -Can we use new DataInputStream once detected a such failure (In this case, it's not CLOSING the previous one and it's pointed to NULL value ) ?


                                    As per my code as bellow,


                                    clientSocket.setKeepAlive(true);

                                    int relen = 0;

                                    synchronized (in) {

                                       

                                        relen = getMessageLength();

                                        request = new byte[relen];   

                                        in.readFully(request,0,relen);

                                    }

                                     

                                    protected int getMessageLength() throws Exception {

                                    byte[] b = new byte[2];

                                    in.readFully(b, 0, 2);

                                    return (int) (((((int) b[0]) & 0xFF) << 8) | (((int) b[1]) & 0xFF));

                                    }

                                    1 2 Previous Next