12 Replies Latest reply: May 31, 2013 3:35 AM by gimbal2 RSS

    download large files

    Hjava
      need to download large amount files, what should I care, Is the following code have problem on download large files
      private static byte[] load(String filename) throws IOException {
              FileInputStream fis = new FileInputStream(filename);
              byte[] data = new byte[fis.available()];
              fis.read(data);
              fis.close();
              return data;
          }
      Edited by: Hjava on May 20, 2013 10:10 PM
        • 1. Re: download large files
          EJP
          That is not a valid use of available(): see the Javadoc.

          You are also assuming that the file size fits into an int; that read() fills the buffer; and that the file fits into available memory. None of these assumptions is tenable.

          Any solution that relies on reading entire files into memory is a problem waiting to happen. It is bound to fail sooner or later when the file gets too large, and when it does fail there is nothing the user can do about it. A very poor failure mode. The idea is to process files a record at a time, or a line at a time.

          Also there is nothing here that constitutes 'downloading' as per your title.
          • 2. Re: download large files
            gimbal2
            For line-by-line I tend to go for BufferedReader since it is so nice as to deal with OS-specific newlines for me. An example:

            http://www.mkyong.com/java/how-to-read-file-from-java-bufferedreader-example/

            But of course this is also making an assumption: that the files being read are actually textual data and not binary. If it is binary the problem becomes more difficult, but it would depend on what binary data it is how to properly process it and what code and classes to use to do so.
            • 3. Re: download large files
              EJP
              If he really is downloading, which is moot, he certainly shouldn't go beyond `InputStream`.
              • 4. Re: download large files
                Hjava
                but the files type is image, can I use line by line in image file?
                if the max size of my image is 1* 1024* 1024, how much I should set on the buffer
                byte[] buffer = new byte[?];

                Edited by: Hjava on May 22, 2013 9:34 PM
                • 5. Re: download large files
                  EJP
                  If the file type is image and you are about to display it you should use javax.imageio.ImageIO.read(URL). If you're just saving it, you don't need it all in memory at once, so you can use a stream copy loop:
                  while ((count = in.read(buffer)) > 0)
                  {
                      out.write(buffer, 0, count);
                  }
                  which works for any kind of file at all.
                  • 6. Re: download large files
                    Kayaman
                    Hjava wrote:
                    but the files type is image, can I use line by line in image file?
                    Of course not, that's binary data. Are you familiar with I/O at all?
                    if the max size of my image is 1* 1024* 1024, how much I should set on the buffer
                    Surprise me!

                    ...or use a 4K, 8K or 16K chunk.
                    • 7. Re: download large files
                      Hjava
                      thanks for all the replies
                      1) it is the very old system, problem encountered when load a lot of images from file, first feeling is the
                       byte[] data = new byte[fis.available()];
                      may have problem.

                      after reading all the replies, it looks like the problems should be did not copy the data to Servlet Output Stream in whole but by piece. did I get the suggestions?
                      there for I should keep the preve load function and change the code in the below
                      byte[] ins = load(filename);
                                         
                                  byte[] outs = ins;
                                  if (ENCRYPT_FILE){
                                       Encrypt encrypt=new Encrypt(keyStr.getBytes());                 
                                       outs= encrypt.decrypt(ins);
                                  }
                                  
                                  
                                  res.getOutputStream().write(outs);
                      1. How do I add the following idea before the outs?
                      while ((count = in.read(buffer)) > 0)
                                  {
                                     out.write(buffer, 0, count);
                                   }
                      2. what is the different between ImageIO.read and FileInputStream.read
                      • 8. Re: download large files
                        gimbal2
                        Hjava wrote:
                        2. what is the different between ImageIO.read and FileInputStream.read
                        One returns a byte or a byte[] array, the other returns a BufferedImage object. Its all right in the javadocs you know.

                        http://docs.oracle.com/javase/6/docs/api/java/io/FileInputStream.html

                        http://docs.oracle.com/javase/6/docs/api/javax/imageio/ImageIO.html

                        I'm not going to comment on your first question.
                        • 9. Re: download large files
                          EJP
                          1) it is the very old system, problem encountered when load a lot of images from file, first feeling is the
                           byte[] data = new byte[fis.available()];
                          may have problem.
                          Considering I told you that four days ago, it is astonishing that you have only now suddenly developed this as a mere 'feeling'. It is wrong. See the Javadoc. I quote: " It is +never+ correct to use the return value of this method to allocate a buffer intended to hold all data in this stream." My emphasis. This is exactly what you are doing. Stop it.
                          after reading all the replies, it looks like the problems should be did not copy the data to Servlet Output Stream in whole but by piece. did I get the suggestions?
                          Yes you did. You not only got 'the suggestions', you got the actual code.
                          1. How do I add the following idea before the outs?
                          I really feel you should be able to figure that out by yourself. If you can't, you need more help than you're going to get here. But I didn't certainly didn't say anything about adding it before anything. When you're told that "you don't need it all in memory at once", you shouldn't be asking how to add code in addition to what you already have. This is really pretty elementary.
                          2. what is the different between ImageIO.read and FileInputStream.read
                          Whatever it says in the Javadoc. Nobody is going to repeat that for you here.
                          • 10. Re: download large files
                            Hjava
                            I still miss understand something, I try the following , but did not get the image ,kindle told me , what I miss?

                             
                            private static byte[] load(String filename) throws IOException {
                                    byte[] buffer = new byte[4096]; 
                                    FileInputStream fis = new FileInputStream(filename);
                                   // byte[] data = new byte[fis.available()];   
                                    int read = 0;
                                    byte[] data= new byte[0];
                                    while ( (read = fis.read(buffer)) != -1 ) {
                                    data = new byte[read];
                                    
                                    fis.read(data);
                                    fis.close();
                                    }
                                    return data;
                                }
                            • 11. Re: download large files
                              EJP
                              I try the following
                              Don't bother. It is complete and utter nonsense, and it bears no resemblance to anything that has been suggested here.
                              What did I miss?
                              You missed everything that's been said in this thread, starting with what I told you ten days ago:
                              You are also assuming that the file size fits into an int; that read() fills the buffer; and that the file fits into available memory. None of these assumptions is tenable.

                              Any solution that relies on reading entire files into memory is a problem waiting to happen. It is bound to fail sooner or later when the file gets too large, and when it does fail there is nothing the user can do about it. A very poor failure mode. The idea is to process files a record at a time, or a line at a time.
                              and what I told you a mere eight days ago:
                              If the file type is image and you are about to display it you should use javax.imageio.ImageIO.read(URL). If you're just saving it, you don't need it all in memory at once, so you can use a stream copy loop
                              As you clearly aren't even reading what you're being told, further responses are clearly a waste of time.
                              • 12. Re: download large files
                                gimbal2
                                At some point you just have to give up. Here is code that reads an entire file into a byte array, in memory where it might just give you out of memory exceptions because it is too much data.
                                import java.io.*;
                                
                                public class Test {
                                
                                    public byte[] readEntireFile(File fp) throws IOException{
                                      
                                      FileInputStream fs = null;
                                
                                      try{
                                        fs = new FileInputStream(fp);
                                        ByteArrayOutputStream baos = new ByteArrayOutputStream();
                                
                                        byte[] buffer = new byte[4096];
                                        int read = 0;
                                
                                        while((read = fs.read(buffer)) != -1){
                                          baos.write(buffer, 0, read);
                                        }
                                
                                        return baos.toByteArray();
                                      
                                      } finally {
                                         if(fs != null){
                                           try { fs.close(); } catch(Throwable t){}
                                         }
                                      }
                                      
                                    }
                                
                                    public static void main(String[] args){
                                    
                                       String fileToTest = "c:\\path\\to\\somefile.ext";
                                       
                                       try{
                                           Test test = new Test();
                                           byte[] data = test.readEntireFile(new File(fileToTest));
                                           
                                           System.out.println("Read " + data.length + " bytes");
                                       } catch(IOException ioe){
                                         System.out.println("The code blew up while reading the file! Does the file " + fileToTest + " even exist?");
                                         ioe.printStackTrace();
                                       }
                                    }
                                }