This discussion is archived
12 Replies Latest reply: May 31, 2013 1:35 AM by gimbal2 RSS

download large files

Hjava Newbie
Currently Being Moderated
need to download large amount files, what should I care, Is the following code have problem on download large files
private static byte[] load(String filename) throws IOException {
        FileInputStream fis = new FileInputStream(filename);
        byte[] data = new byte[fis.available()];
        fis.read(data);
        fis.close();
        return data;
    }
Edited by: Hjava on May 20, 2013 10:10 PM
  • 1. Re: download large files
    EJP Guru
    Currently Being Moderated
    That is not a valid use of available(): see the Javadoc.

    You are also assuming that the file size fits into an int; that read() fills the buffer; and that the file fits into available memory. None of these assumptions is tenable.

    Any solution that relies on reading entire files into memory is a problem waiting to happen. It is bound to fail sooner or later when the file gets too large, and when it does fail there is nothing the user can do about it. A very poor failure mode. The idea is to process files a record at a time, or a line at a time.

    Also there is nothing here that constitutes 'downloading' as per your title.
  • 2. Re: download large files
    gimbal2 Guru
    Currently Being Moderated
    For line-by-line I tend to go for BufferedReader since it is so nice as to deal with OS-specific newlines for me. An example:

    http://www.mkyong.com/java/how-to-read-file-from-java-bufferedreader-example/

    But of course this is also making an assumption: that the files being read are actually textual data and not binary. If it is binary the problem becomes more difficult, but it would depend on what binary data it is how to properly process it and what code and classes to use to do so.
  • 3. Re: download large files
    EJP Guru
    Currently Being Moderated
    If he really is downloading, which is moot, he certainly shouldn't go beyond `InputStream`.
  • 4. Re: download large files
    Hjava Newbie
    Currently Being Moderated
    but the files type is image, can I use line by line in image file?
    if the max size of my image is 1* 1024* 1024, how much I should set on the buffer
    byte[] buffer = new byte[?];

    Edited by: Hjava on May 22, 2013 9:34 PM
  • 5. Re: download large files
    EJP Guru
    Currently Being Moderated
    If the file type is image and you are about to display it you should use javax.imageio.ImageIO.read(URL). If you're just saving it, you don't need it all in memory at once, so you can use a stream copy loop:
    while ((count = in.read(buffer)) > 0)
    {
        out.write(buffer, 0, count);
    }
    which works for any kind of file at all.
  • 6. Re: download large files
    Kayaman Guru
    Currently Being Moderated
    Hjava wrote:
    but the files type is image, can I use line by line in image file?
    Of course not, that's binary data. Are you familiar with I/O at all?
    if the max size of my image is 1* 1024* 1024, how much I should set on the buffer
    Surprise me!

    ...or use a 4K, 8K or 16K chunk.
  • 7. Re: download large files
    Hjava Newbie
    Currently Being Moderated
    thanks for all the replies
    1) it is the very old system, problem encountered when load a lot of images from file, first feeling is the
     byte[] data = new byte[fis.available()];
    may have problem.

    after reading all the replies, it looks like the problems should be did not copy the data to Servlet Output Stream in whole but by piece. did I get the suggestions?
    there for I should keep the preve load function and change the code in the below
    byte[] ins = load(filename);
                       
                byte[] outs = ins;
                if (ENCRYPT_FILE){
                     Encrypt encrypt=new Encrypt(keyStr.getBytes());                 
                     outs= encrypt.decrypt(ins);
                }
                
                
                res.getOutputStream().write(outs);
    1. How do I add the following idea before the outs?
    while ((count = in.read(buffer)) > 0)
                {
                   out.write(buffer, 0, count);
                 }
    2. what is the different between ImageIO.read and FileInputStream.read
  • 8. Re: download large files
    gimbal2 Guru
    Currently Being Moderated
    Hjava wrote:
    2. what is the different between ImageIO.read and FileInputStream.read
    One returns a byte or a byte[] array, the other returns a BufferedImage object. Its all right in the javadocs you know.

    http://docs.oracle.com/javase/6/docs/api/java/io/FileInputStream.html

    http://docs.oracle.com/javase/6/docs/api/javax/imageio/ImageIO.html

    I'm not going to comment on your first question.
  • 9. Re: download large files
    EJP Guru
    Currently Being Moderated
    1) it is the very old system, problem encountered when load a lot of images from file, first feeling is the
     byte[] data = new byte[fis.available()];
    may have problem.
    Considering I told you that four days ago, it is astonishing that you have only now suddenly developed this as a mere 'feeling'. It is wrong. See the Javadoc. I quote: " It is +never+ correct to use the return value of this method to allocate a buffer intended to hold all data in this stream." My emphasis. This is exactly what you are doing. Stop it.
    after reading all the replies, it looks like the problems should be did not copy the data to Servlet Output Stream in whole but by piece. did I get the suggestions?
    Yes you did. You not only got 'the suggestions', you got the actual code.
    1. How do I add the following idea before the outs?
    I really feel you should be able to figure that out by yourself. If you can't, you need more help than you're going to get here. But I didn't certainly didn't say anything about adding it before anything. When you're told that "you don't need it all in memory at once", you shouldn't be asking how to add code in addition to what you already have. This is really pretty elementary.
    2. what is the different between ImageIO.read and FileInputStream.read
    Whatever it says in the Javadoc. Nobody is going to repeat that for you here.
  • 10. Re: download large files
    Hjava Newbie
    Currently Being Moderated
    I still miss understand something, I try the following , but did not get the image ,kindle told me , what I miss?

     
    private static byte[] load(String filename) throws IOException {
            byte[] buffer = new byte[4096]; 
            FileInputStream fis = new FileInputStream(filename);
           // byte[] data = new byte[fis.available()];   
            int read = 0;
            byte[] data= new byte[0];
            while ( (read = fis.read(buffer)) != -1 ) {
            data = new byte[read];
            
            fis.read(data);
            fis.close();
            }
            return data;
        }
  • 11. Re: download large files
    EJP Guru
    Currently Being Moderated
    I try the following
    Don't bother. It is complete and utter nonsense, and it bears no resemblance to anything that has been suggested here.
    What did I miss?
    You missed everything that's been said in this thread, starting with what I told you ten days ago:
    You are also assuming that the file size fits into an int; that read() fills the buffer; and that the file fits into available memory. None of these assumptions is tenable.

    Any solution that relies on reading entire files into memory is a problem waiting to happen. It is bound to fail sooner or later when the file gets too large, and when it does fail there is nothing the user can do about it. A very poor failure mode. The idea is to process files a record at a time, or a line at a time.
    and what I told you a mere eight days ago:
    If the file type is image and you are about to display it you should use javax.imageio.ImageIO.read(URL). If you're just saving it, you don't need it all in memory at once, so you can use a stream copy loop
    As you clearly aren't even reading what you're being told, further responses are clearly a waste of time.
  • 12. Re: download large files
    gimbal2 Guru
    Currently Being Moderated
    At some point you just have to give up. Here is code that reads an entire file into a byte array, in memory where it might just give you out of memory exceptions because it is too much data.
    import java.io.*;
    
    public class Test {
    
        public byte[] readEntireFile(File fp) throws IOException{
          
          FileInputStream fs = null;
    
          try{
            fs = new FileInputStream(fp);
            ByteArrayOutputStream baos = new ByteArrayOutputStream();
    
            byte[] buffer = new byte[4096];
            int read = 0;
    
            while((read = fs.read(buffer)) != -1){
              baos.write(buffer, 0, read);
            }
    
            return baos.toByteArray();
          
          } finally {
             if(fs != null){
               try { fs.close(); } catch(Throwable t){}
             }
          }
          
        }
    
        public static void main(String[] args){
        
           String fileToTest = "c:\\path\\to\\somefile.ext";
           
           try{
               Test test = new Test();
               byte[] data = test.readEntireFile(new File(fileToTest));
               
               System.out.println("Read " + data.length + " bytes");
           } catch(IOException ioe){
             System.out.println("The code blew up while reading the file! Does the file " + fileToTest + " even exist?");
             ioe.printStackTrace();
           }
        }
    }

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points