1 2 Previous Next 27 Replies Latest reply: Feb 7, 2012 10:29 AM by user10049207 RSS

    java.util.zip.ZipException: invalid stored block lengths

    user10049207
      Only one thread running at the time of test; I have a problem with the following scenario.
      A client reads data from a file, does some processing of incoming data, compresses it and sends it
      to the server. The server at the first instance of the read throws:

      java.util.zip.ZipException: invalid stored block lengths
           at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:164)
           at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:122)
           at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:337)
           at java.io.DataInputStream.readUTF(DataInputStream.java:589)
           at java.io.DataInputStream.readUTF(DataInputStream.java:564)
           

      I highly appreciate any help.
      thanks,
      AR.

      Client Code snippet:

           boolean nowrap = true;
           Deflater deflater = new Deflater(Deflater.DEFAULT_COMPRESSION, nowrap);
           DeflaterOutputStream defOpStrm = new DeflaterOutputStream(new BufferedOutputStream(
                               thrdHlpr.getClntSkt().getOutputStream()), deflater);
           DataOutputStream opStrm = new DataOutputStream(defOpStrm);

      //Inside a loop
           buffer = inFile.readLine();
           if((byteRead + buffer.length()) < BUF_SIZE) { // BUF_SIZE = 512
           rawBuf.append(buffer + System.getProperty("line.separator"));
           }
           else{
      //do some processing of data the size remains the same
      //and transfer the processed rawBuf to outBuf to be sent to server

           opStrm.writeUTF(outBuf.toString());
           deflOpStrm.finish();
           opStrm.flush();
      deflater.reset();
      ............
      =====================================================




      Server code in a loop:
           boolean nowrap = true;
           Inflater inflater = new Inflater(nowrap);
           InflaterInputStream iInput = new InflaterInputStream(clntSokt.getInputStream(), inflater);
           DataInputStream in = new DataInputStream(iInput);

      String result = in.readUTF(); //This line throws "java.util.zip.ZipException: invalid stored block lengths"

           output.seek(output.length());
           output.writeUTF(result);
           inflater.reset();

      Edited by: user10049207 on Feb 3, 2012 8:22 AM

      Edited by: user10049207 on Feb 3, 2012 8:37 AM
        • 1. Re: java.util.zip.ZipException: invalid stored block lengths
          jtahlborn
          what is happening inside the inner loop in the client code? is it messing with the socket output stream?
          • 2. Re: java.util.zip.ZipException: invalid stored block lengths
            user10049207
            No, not at all, I have isolated all of my dealings with the socket and posted it.
            • 3. Re: java.util.zip.ZipException: invalid stored block lengths
              jtahlborn
              so you don't otherwise do anything with the socket streams (before or after the code posted)? (e.g. attempt to send other data, re-use the stream, etc)
              • 4. Re: java.util.zip.ZipException: invalid stored block lengths
                user10049207
                I am adding the entire block blocks from client and server, so that you have a better picture.
                I am just thinking that it might be some sort of synchronization of the client and receiving socket I am missing; Could it be the case?

                Thanks....

                Client code block
                =================
                while(inFile.ready()) {

                buffer = inFile.readLine();
                if((byteRead + buffer.length()) < BUF_SIZE) {
                rawBuf.append(buffer + System.getProperty("line.separator"));               
                }
                else{

                //DO the necessary processing

                opStrm.writeUTF(snitBuf.toString());
                deflOpStrm.finish();
                opStrm.flush();
                deflater.reset();


                offset += buffer.length() + 1;
                byteRead = 0;
                snitBuf.delete(0, snitBuf.length());
                sndBuf.delete(0, sndBuf.length());


                rawBuf.append(buffer + System.getProperty("line.separator"));

                }

                byteRead += buffer.length();
                offset += buffer.length() + 1;     
                buffer = null;                         
                }



                Server code Block:
                =========================
                while(true) {
                          String result = in.readUTF();
                          logger.info("LogFilesWriter Received: " + result );
                          inflater.reset();
                          output.seek(output.length());
                          output.writeUTF(result);
                }
                • 5. Re: java.util.zip.ZipException: invalid stored block lengths
                  jtahlborn
                  it's still unclear from what you've posted, but what i'm trying to understand is whether or not you are trying to write multiple compressed blocks to the same socket stream. if so, i would guess that is where your problem lies.
                  • 6. Re: java.util.zip.ZipException: invalid stored block lengths
                    user10049207
                    In the client's while block, I am obviously trying to write multiple compressed block.

                    You are right that; that is where the problem is. As I have said the write of compressed block is a repeat of what I have posted; My question is what do you think or do you know what is the problem?

                    thanks for the answer.
                    • 7. Re: java.util.zip.ZipException: invalid stored block lengths
                      jtahlborn
                      user10049207 wrote:
                      In the client's while block, I am obviously trying to write multiple compressed block.

                      You are right that; that is where the problem is. As I have said the write of compressed block is a repeat of what I have posted;
                      it wasn't at all clear that you were writing multiple blocks per stream.
                      My question is what do you think or do you know what is the problem?
                      yes, the typical problem in this situation is reading too much information the first time around. most likely, the inflaterinputstream reads blocks of data from the underlying stream. most likely it is reading past the end of the first block and reading part of the next block. even though the inflater will only actually use the bytes from the current block, it has consumed bytes from the next block from the underlying stream. so, after you reset the inflater and try to read the next block, you are now reading mid-compressed block.

                      is there a reason you can't just use one continuous compressed stream?
                      • 8. Re: java.util.zip.ZipException: invalid stored block lengths
                        user10049207
                        is there a reason you can't just use one continuous compressed stream?
                        No, there is not; at the client side the input is dynamic during the process, so the data is massaged, compressed and sent as it comes in.
                        most likely it is reading past the end of the first block and reading part of the next block. even though the inflater will only actually use the bytes from the current block, it has consumed bytes from the next block from the underlying stream. so, after you reset the inflater and try to read the next block, you are now reading mid-compressed block.
                        I agree, this seems to be the issue; I mentioned in my previous post that there seems to be some synchronization problem, because Inside the debugger I did stop the client from sending more data and the processing on the server side went on with no problem.

                        Will you be able to tell me; what is the best way to remedy this race/sync/reading-next-block problem?

                        thanks very much for the answer.
                        • 9. Re: java.util.zip.ZipException: invalid stored block lengths
                          user10049207
                          jtahlborn; My apologies, I did not answer your first question correctly.
                          is there a reason you can't just use one continuous compressed stream?
                          Yes, there is. At the client side the input is continuously coming in, my process has to read the data as it comes in, massage it, compress it and sent to the server.

                          As you can see, I have to compress and send the data over, in several attempts. Will you be able to tell me; what is the best way to remedy this race/sync/reading-next-block problem?


                          thanks.
                          • 10. Re: java.util.zip.ZipException: invalid stored block lengths
                            jtahlborn
                            user10049207 wrote:
                            jtahlborn; My apologies, I did not answer your first question correctly.
                            is there a reason you can't just use one continuous compressed stream?
                            Yes, there is. At the client side the input is continuously coming in, my process has to read the data as it comes in, massage it, compress it and sent to the server.
                            i understand the intent, that still doesn't tell me why you can't compress the data in one continuous stream. if each block is small, then compressing each block individually is going to gain you very little in terms of compression at the expense of extra cpu time. compressing in a continuous stream will give you a much better compression ratio. you can still do stuff "on the fly" using a continuously compressed stream.
                            As you can see, I have to compress and send the data over, in several attempts. Will you be able to tell me; what is the best way to remedy this race/sync/reading-next-block problem?
                            if you want to compress each block individually, you basically need to compress each block to a separate byte[], then send your data as individual "messages" (e.g. send a 4 byte header indicating the length of the next block and then the block itself). on the receiving end, you read the 4 byte length header, then read only that many bytes into a separate byte[] and decompress it. this will ensure that you don't lose any data.
                            • 11. Re: java.util.zip.ZipException: invalid stored block lengths
                              user10049207
                              Thanks very much or the suggestion, i will try it.
                              But is this not a bug that inflater reads the next block .....?
                              • 12. Re: java.util.zip.ZipException: invalid stored block lengths
                                jtahlborn
                                user10049207 wrote:
                                But is this not a bug that inflater reads the next block .....?
                                no, the Inflater can't know that it has read past the end of the data until it actually reads the end of data. and, in order to perform reasonably, it reads data in chunks from the underlying stream (as do most stream implementations). thus, it is inevitable that it will read extra data. in order to perform as you want, it would have to read data byte by byte, which would be extremely inefficient.
                                • 13. Re: java.util.zip.ZipException: invalid stored block lengths
                                  EJP
                                  To return to the scheduled program, I suggest you:

                                  - get rid of the finish() and reset() calls
                                  - use the same DeflaterOututStream and DeflaterInputStream, and ditto the streams on top of them, for the life of the socket, instead of creating new ones per message
                                  - flush() the output after writing every message.
                                  - make sure that your messages can be perceived as such by the receiver, e.g. via a length word prefix

                                  Then you will be deflating a stream and reflating a stream, and most of these difficulties will disappear. However the inflated will still need to read ahead a bit so you won't have transaction-like behaviour on your messages.
                                  • 14. Re: java.util.zip.ZipException: invalid stored block lengths
                                    user10049207
                                    Hi EJP and thnks for the suggestions.
                                    Based on my undrestanding of your advice; where ever necessary i have modifed my code to
                                    follow your suggestions as below:



                                    //Client code snippet
                                    //This block alwayes been happening once only
                                    =========================================
                                    Deflater defltr = new Deflater(Deflater.DEFAULT_COMPRESSION, true); //tried nowrap true or false
                                    DeflaterOutputStream deflOpStrm = new DeflaterOutputStream(new BufferedOutputStream(
                                                             thrdHlpr.getClntSkt().getOutputStream()), defltr, BUF_SIZE); //tried 1024 or 512
                                    DataOutputStream opStrm = new DataOutputStream(deflOpStrm);


                                    //this is the loop's code
                                    while(inFile.ready()) {
                                         buffer = inFile.readLine();
                                         if((byteRead + buffer.length()) < BUF_SIZE) {                    
                                         rawBuf.append(buffer + System.getProperty("line.separator"));
                                         }
                                         else{
                                         //Do my stuff........ and move data to outBuf                     
                                         opStrm.writeUTF(outBuf.toString());
                                         deflOpStrm.flush();
                                         opStrm.flush();
                                              //deflOpStrm.finish();     //Commented out
                                              //defltr.reset(); //Commented out
                                    }//end of while(inFile.ready....



                                    //Server code snippet
                                    //This block alwayes been happening once only
                                    =========================================
                                    Inflater infltr = new Inflater(true); //tried nowrap true or false
                                    InflaterInputStream infInputStrm = new InflaterInputStream(clntSokt.getInputStream(), infltr, ZBUF_SIZE); //tried1024 or 512
                                    DataInputStream dInptStrm = new DataInputStream(infInputStrm);

                                    //this is the loop's code
                                    while(true) {
                                    String result = dInptStrm.readUTF(); // First read throws exception
                                    //logger.info("LogFilesWriter Received: " + result );
                                    //infltr.reset(); //Commented out
                                    output.seek(output.length());
                                    output.writeUTF(result);
                                    }




                                    This is the exception stack and as you have noticed the error message is now changed from
                                    "java.util.zip.ZipException: invalid stored block lengths "
                                    to below, though the stack remain the same ....

                                    java.util.zip.ZipException: invalid code lengths set
                                         at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:164)
                                         at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:122)
                                         at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:337)
                                         at java.io.DataInputStream.readUTF(DataInputStream.java:589)
                                         at java.io.DataInputStream.readUTF(DataInputStream.java:564)
                                         
                                         
                                    If you think i am not doing the right thing; Is there a code snippet i can look at anywhere.....
                                    Thanks very much for the suggestion and answer.

                                    Edited by: user10049207 on Jan 23, 2012 10:07 AM

                                    Edited by: user10049207 on Feb 3, 2012 8:21 AM

                                    Edited by: user10049207 on Feb 3, 2012 8:37 AM
                                    1 2 Previous Next