1 2 Previous Next 27 Replies Latest reply: Feb 7, 2012 10:29 AM by user10049207 Go to original post RSS
      • 15. Re: java.util.zip.ZipException: invalid stored block lengths
        user10049207
        To make sure that I am doing everything right, I have isolated the compression, decompression. Eventhough I am following the documentation line by line, I am still getting "*java.util.zip.DataFormatException: invalid distance too far back"*.


        JAVA: Java(TM) SE Runtime Environment (build 1.7.0_02-b13)
        The compressiion code snippet is:
        ==================================
        private void deflateOutput(StringBuffer snitBuf, StringBuffer sndBuf){
                  
             String tempStr = snitBuf.toString();
             int inLen = tempStr.getBytes().length;
             byte[] outBfr = new byte[inLen];
             Deflater def = new Deflater();
             int outBfrLen = 0;
                  
             try{
             def.setInput(tempStr.getBytes("UTF-8"), 0, inLen);
             def.finish();
             while (!def.finished()) {
                  outBfrLen = def.deflate(outBfr);
             }

             sndBuf.append(new String(outBfr, 0, outBfrLen));           
             }
        catch(Exception ueE){  
             ueE.printStackTrace();
        }

        def.reset();
        def.end();
             
        inflateInputTest(sndBuf.toString().getBytes(), outBfrLen);     
        }



        DecompressCode is:
        ======================================
        private void inflateInputTest(byte[] inBuf, int inBufLen){
        String inStr = new String(inBuf);
        byte[] result = new byte[BUF_SIZE + 512];
        String outputString = null;
        logger.info(this.getName() + " CompressedSize: " + inBufLen);     
                  
        try{
        Inflater infltr = new Inflater();
        infltr.setInput(inStr.getBytes(), 0, inBufLen);
        int resLen = infltr.inflate(result);  //This line throws the exception
        logger.info(this.getName() + " UnCompressedSize: " + resLen);
        infltr.reset();
        infltr.end();

        outputString = new String(result, 0, resLen, "UTF-8");
        } catch(Exception ex) {
        logger.error(this.getName() + " Inflator Error: " , ex);
        }

        logger.info(this.getName() + " Compressed: " + outputString);          
        }



        The Stack is:
        java.util.zip.DataFormatException: invalid distance too far back
             at java.util.zip.Inflater.inflateBytes(Native Method)
             at java.util.zip.Inflater.inflate(Inflater.java:256)
             at java.util.zip.Inflater.inflate(Inflater.java:274)

        Anybody any idea please.....
        thanks.

        Edited by: user10049207 on Feb 3, 2012 8:20 AM

        Edited by: user10049207 on Feb 3, 2012 8:37 AM
        • 16. Re: java.util.zip.ZipException: invalid stored block lengths
          jtahlborn
          why are you trying to use Inflater/Deflater directly instead of using the DeflaterOutputStream and InflaterInputStream? seems a lot safer than trying to use them correctly. then you would avoid bugs like repeatedly calling deflate() and overwriting the data in your byte array.
          • 17. Re: java.util.zip.ZipException: invalid stored block lengths
            user10049207
            jtahlborn wrote:
            why are you trying to use Inflater/Deflater directly instead of using the DeflaterOutputStream and InflaterInputStream? seems a lot safer than trying to use them correctly. then you would avoid bugs like repeatedly calling deflate() and overwriting the data in your byte array.
            Because on your earlier response you wrote;
            if you want to compress each block individually, you basically need to compress each block to a separate byte[], then send your data as individual "messages" (e.g. send a 4 byte header indicating the length of the next block and then the block itself). on the receiving end, you read the 4 byte length header, then read only that many bytes into a separate byte[] and decompress it. this will ensure that you don't lose any data.
            And this is what I undrestood, otherwise I was using the DeflaterOutputStream & InflaterInputStream... and as you know it resulted in the original errors....

            My appologies, It seems that i do not get what you meant. Is it possible for you to correct either set of my codes (DeflaterOutputStream -InflaterInputStream or Deflater-Inflater), so that I can see where is my mistake.

            i highly appreciate your answer.
            thanks.

            Edited by: user10049207 on Jan 30, 2012 2:38 PM
            • 18. Re: java.util.zip.ZipException: invalid stored block lengths
              EJP
              As you have already said there is no reason why you can't use single streams, use single streams, for the lifetime of the socket, one in each direction, the socket input stream wrapped in an InflaterInputStream, and the socket output stream wrapped in a DeflaterOutputStream. Don't reset, don't fiddle with the Inflater/Deflater; just read and write.
              • 19. Re: java.util.zip.ZipException: invalid stored block lengths
                user10049207
                I have already done that and I have posted the result of that process on "Jan 23, 2012 9:11 AM in response to: EJP "

                thnaks.
                • 20. Re: java.util.zip.ZipException: invalid stored block lengths
                  jtahlborn
                  user10049207 wrote:
                  I have already done that and I have posted the result of that process on "Jan 23, 2012 9:11 AM in response to: EJP "
                  i would suggest creating a simple, standalone test for sending compressed data over a socket. if it doesn't work, then post that entire code here (in code tags) and ask for help (because your current snippets are too fragmented to give meaningful help). if it does work, then figure out how it differs from your actual code and then fix it.
                  • 21. Re: java.util.zip.ZipException: invalid stored block lengths
                    user10049207
                    i appreciate the input by various peopl. But despite trying all the different paths that were suggested, i am getting the same error, I started with.

                    I reealy appreciate if anybody can help me with this or let me know if this is known bug.
                    Thanks for your time and input.
                    • 22. Re: java.util.zip.ZipException: invalid stored block lengths
                      jtahlborn
                      user10049207 wrote:
                      i appreciate the input by various peopl. But despite trying all the different paths that were suggested, i am getting the same error, I started with.

                      I reealy appreciate if anybody can help me with this or let me know if this is known bug.
                      Thanks for your time and input.
                      did you follow my last advice? if so, can you post your simple, standalone code here?
                      • 23. Re: java.util.zip.ZipException: invalid stored block lengths
                        user10049207
                        jtahlborn wrote:
                        did you follow my last advice? if so, can you post your simple, standalone code here?
                        As far as i undrestand, this is where i started, so below i am repeating a cleaner version of it.
                        Only one thread running at the time of test;
                        JAVA: Java(TM) SE Runtime Environment (build 1.7.0_02-b13)

                        the scenario is as follow:
                        A client inside a loop reads data from a file, does some processing of incoming data, compresses it and sends it
                        to the server over a socket. The server at the first instance of the read throws: "java.util.zip.ZipException: invalid distance too far back"

                        The Client side compressiion code snippet is:
                        ==================================
                        try{
                        .......

                        Deflater defltr = new Deflater(Deflater.DEFAULT_COMPRESSION, true);
                        DeflaterOutputStream deflOpStrm = new DeflaterOutputStream(new BufferedOutputStream(
                        thrdHlpr.getClntSkt().getOutputStream()), defltr, BUF_SIZE); //BUF_SIZE= 10240
                        DataOutputStream opStrm = new DataOutputStream(deflOpStrm);


                        while(true){
                        while(0 != chBufLen && -1 != chBufLen){
                        char[] chBuff = new char[BUF_SIZE]; //BUF_SIZE= 10240 .....1024/512
                        chBufLen = inFile.read(chBuff, 0, BUF_SIZE);
                        if( 0 < chBufLen){
                        offset += chBufLen;
                        rawBuf.append(chBuff, 0, chBufLen);
                        //Do my things with the data and transfer data to outBuf


                        opStrm.writeUTF(outBuf.toString());
                        deflOpStrm.flush();
                        opStrm.flush();

                        //I have tried with and without calling finish & reset
                        //deflOpStrm.finish();
                        //defltr.reset();
                        chBuff = null;
                        }
                        }//end while(0 != chBufLen &&.....
                        ........

                        ........
                        //Catch block.....



                        Server side UncompressCode is:
                        ======================================
                        try{
                        ...........

                        Inflater infltr = new Inflater(true); //BUF_SIZE= 10240 also used small 512/ 1024
                        InflaterInputStream infInputStrm = new InflaterInputStream(clntSokt.getInputStream(), infltr, BUF_SIZE/2);
                        DataInputStream dInptStrm = new DataInputStream(infInputStrm);

                        while(true) {
                        String result = dInptStrm.readUTF(); //Exception is thrown here
                        output.seek(output.length());
                        output.writeUTF(result);


                        //Have tried with and without reset.
                        //infltr.reset();
                        }//end while(true....
                        .........
                        //catch block


                        The Stak is:
                        =============================
                        java.util.zip.ZipException: invalid distance too far back
                        at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:164)
                        at java.io.DataInputStream.readFully(DataInputStream.java:195)
                        at java.io.DataInputStream.readUTF(DataInputStream.java:609)
                        at java.io.DataInputStream.readUTF(DataInputStream.java:564)


                        Thanks very much for your time....
                        • 24. Re: java.util.zip.ZipException: invalid stored block lengths
                          EJP
                          The following adapted from your code works perfectly for me:
                          import java.io.BufferedOutputStream;
                          import java.io.DataInputStream;
                          import java.io.DataOutputStream;
                          import java.io.EOFException;
                          import java.io.IOException;
                          import java.net.ServerSocket;
                          import java.net.Socket;
                          import java.util.zip.Deflater;
                          import java.util.zip.DeflaterOutputStream;
                          import java.util.zip.Inflater;
                          import java.util.zip.InflaterInputStream;
                          
                          public class InflaterDeflaterTest implements Runnable
                          {
                               static final int BUF_SIZE = 10240;
                               static final int     MSG_SIZE = 512;
                               static final int     COUNT = 1000;
                          
                               int     port;
                          
                               public void run()
                               {
                                    try
                                    {
                                         Server     server = new Server();
                                         Thread     st = new Thread(server);
                                         st.start();
                                         Client     client = new Client();
                                         client.run();
                                    }
                                    catch (IOException exc)
                                    {
                                         exc.printStackTrace();
                                    }
                               }
                          
                               class Server implements Runnable
                               {
                                    ServerSocket     ss;
                          
                                    Server() throws IOException
                                    {
                                         this.ss = new ServerSocket(0);
                                         port = ss.getLocalPort();
                                         System.out.println("Server listening at "+port);
                                    }
                          
                                    public void run()
                                    {
                                         try
                                         {
                                              System.out.println("Server: accepting");
                                              Socket     socket = ss.accept();
                                              System.out.println("Server accepted "+socket);
                                              Thread     t = new Thread(new ConnectionHandler(socket));
                                              t.start();
                                         }
                                         catch (IOException exc)
                                         {
                                              exc.printStackTrace();
                                         }
                                         finally
                                         {
                                              try { ss.close(); } catch (IOException exc){ exc.printStackTrace();}
                                         }
                                    }
                               }
                          
                               class ConnectionHandler implements Runnable
                               {
                                    Socket     socket;
                                    Inflater infltr = new Inflater(true); //BUF_SIZE= 10240 also used small 512/ 1024
                                    DataInputStream     in;
                          
                                    ConnectionHandler(Socket socket) throws IOException
                                    {
                                         this.socket = socket;
                                         InflaterInputStream infInputStrm = new InflaterInputStream(socket.getInputStream(), infltr, BUF_SIZE/2);
                                         DataInputStream dInptStrm = new DataInputStream(infInputStrm);
                                         this.in = dInptStrm;
                                    }
                          
                                    public void     run()
                                    {
                                         byte[]     buffer = new byte[MSG_SIZE];
                                         while (true)
                                         {
                                              try
                                              {
                                                   String result = in.readUTF(); //Exception is thrown here
                                                   System.out.println("Server: read " + result);
                                                   in.readFully(buffer);
                                              }
                                              catch (EOFException exc)
                                              {
                                                   System.out.println("Server read EOS");
                                                   break;
                                              }
                                              catch (IOException exc)
                                              {
                                                   exc.printStackTrace();
                                              }
                                              finally
                                              {
                                                   try
                                                   {
                                                        socket.close();
                                                   }
                                                   catch (IOException exc)
                                                   {
                                                        exc.printStackTrace();
                                                   }
                                              }
                                         }
                                    }
                               }
                          
                               class Client implements Runnable
                               {
                                    Socket socket;
                                    Deflater defltr = new Deflater(Deflater.DEFAULT_COMPRESSION, true);
                                    DataOutputStream     out;
                          
                                    Client() throws IOException
                                    {
                                         this.socket = new Socket("localhost", port);
                                         System.out.println("Client: connected "+socket);
                                         DeflaterOutputStream deflOpStrm = new DeflaterOutputStream(new BufferedOutputStream(
                                              socket.getOutputStream()), defltr, BUF_SIZE); //BUF_SIZE= 10240
                                         DataOutputStream opStrm = new DataOutputStream(deflOpStrm);
                                         this.out = opStrm;
                                    }
                          
                                    public void run()
                                    {
                                         byte[]     buffer = new byte[MSG_SIZE];
                                         try
                                         {
                                              for (int i = 1; i <= COUNT; i++)
                                              {
                                                   out.writeUTF("Hello-"+i);
                                                   out.write(buffer);
                                                   out.flush();
                                                   System.out.println("Client wrote #"+i);
                                              }
                                              out.close();
                                              System.out.println("Client finished");
                                         }
                                         catch (IOException exc)
                                         {
                                              exc.printStackTrace();
                                         }
                                         finally
                                         {
                                              try { socket.close(); } catch (IOException exc){ exc.printStackTrace();}
                                         }
                                    }
                               }
                          
                               public static void main(String[] args) throws Exception
                               {
                                    new InflaterDeflaterTest().run();
                               }
                          }
                          I conclude that you are probably writing something else to the stream, or reading something else from the stream, somewhere or other, not shown in your code, as jtahlborn suggested several weeks ago.

                          However it is notable that the server reads nothing until the client has closed the socket. So basically this isn't going to work for you. Get it working without the deflation/inflation and see if you really have a bandwidth problem.
                          • 25. Re: java.util.zip.ZipException: invalid stored block lengths
                            user10049207
                            EJP and jtahlborn thanks very much, I will try this and let you know.

                            EJP, i played a little with your code sample (had to make a few corrections here and ther), but as you have mentioned, it did not help.

                            In terms of getting my code to work without deflation/Inflation; i have never had any problem with that at all.
                            Do you think this is probably a bug? Any advice?

                            Your time and advice are Heighly appreciated.

                            Edited by: user10049207 on Feb 6, 2012 1:34 PM

                            Edited by: user10049207 on Feb 6, 2012 1:41 PM
                            • 26. Re: java.util.zip.ZipException: invalid stored block lengths
                              EJP
                              Do you think this is probably a bug? Any advice?
                              Do I think what is probably a bug? I have posted code that demonstrates there is no bug. Any bugs are in your code as far as I can see, most probably the bits you haven't posted.
                              • 27. Re: java.util.zip.ZipException: invalid stored block lengths
                                user10049207
                                EJP wrote:
                                However it is notable that the server reads nothing until the client has closed the socket. So basically this isn't going to work for you.
                                Since my code and your try did not work for my situation; don't you consider this a bug?!

                                More importantly and Beside the bug question, my question is any advice on how to handle this issue (In a client-Server setting; countinously send and resend some and more compressed data over a client socket without having to close it). As you might have seen Earlier on this thread i have tried with bare Deflater/Inflatoer combination as well .... and not much success there either......

                                thanks..... and highly appreciate your time and advice.
                                1 2 Previous Next