12 Replies Latest reply: May 31, 2013 3:35 AM by gimbal2

private static byte[] load(String filename) throws IOException {
FileInputStream fis = new FileInputStream(filename);
byte[] data = new byte[fis.available()];
fis.close();
return data;
}
Edited by: Hjava on May 20, 2013 10:10 PM
That is not a valid use of available(): see the Javadoc.

You are also assuming that the file size fits into an int; that read() fills the buffer; and that the file fits into available memory. None of these assumptions is tenable.

Any solution that relies on reading entire files into memory is a problem waiting to happen. It is bound to fail sooner or later when the file gets too large, and when it does fail there is nothing the user can do about it. A very poor failure mode. The idea is to process files a record at a time, or a line at a time.

For line-by-line I tend to go for BufferedReader since it is so nice as to deal with OS-specific newlines for me. An example:

But of course this is also making an assumption: that the files being read are actually textual data and not binary. If it is binary the problem becomes more difficult, but it would depend on what binary data it is how to properly process it and what code and classes to use to do so.
If he really is downloading, which is moot, he certainly shouldn't go beyond InputStream.
but the files type is image, can I use line by line in image file?
if the max size of my image is 1* 1024* 1024, how much I should set on the buffer
byte[] buffer = new byte[?];

Edited by: Hjava on May 22, 2013 9:34 PM
If the file type is image and you are about to display it you should use javax.imageio.ImageIO.read(URL). If you're just saving it, you don't need it all in memory at once, so you can use a stream copy loop:
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
which works for any kind of file at all.
Hjava wrote:
but the files type is image, can I use line by line in image file?
Of course not, that's binary data. Are you familiar with I/O at all?
if the max size of my image is 1* 1024* 1024, how much I should set on the buffer
Surprise me!

...or use a 4K, 8K or 16K chunk.
thanks for all the replies
1) it is the very old system, problem encountered when load a lot of images from file, first feeling is the
 byte[] data = new byte[fis.available()];
may have problem.

after reading all the replies, it looks like the problems should be did not copy the data to Servlet Output Stream in whole but by piece. did I get the suggestions?
there for I should keep the preve load function and change the code in the below
byte[] ins = load(filename);

byte[] outs = ins;
if (ENCRYPT_FILE){
Encrypt encrypt=new Encrypt(keyStr.getBytes());
outs= encrypt.decrypt(ins);
}

res.getOutputStream().write(outs);
1. How do I add the following idea before the outs?
while ((count = in.read(buffer)) > 0)
{
out.write(buffer, 0, count);
}
Hjava wrote:
One returns a byte or a byte[] array, the other returns a BufferedImage object. Its all right in the javadocs you know.

http://docs.oracle.com/javase/6/docs/api/java/io/FileInputStream.html

http://docs.oracle.com/javase/6/docs/api/javax/imageio/ImageIO.html

I'm not going to comment on your first question.
1) it is the very old system, problem encountered when load a lot of images from file, first feeling is the
 byte[] data = new byte[fis.available()];
may have problem.
Considering I told you that four days ago, it is astonishing that you have only now suddenly developed this as a mere 'feeling'. It is wrong. See the Javadoc. I quote: " It is +never+ correct to use the return value of this method to allocate a buffer intended to hold all data in this stream." My emphasis. This is exactly what you are doing. Stop it.
after reading all the replies, it looks like the problems should be did not copy the data to Servlet Output Stream in whole but by piece. did I get the suggestions?
Yes you did. You not only got 'the suggestions', you got the actual code.
1. How do I add the following idea before the outs?
I really feel you should be able to figure that out by yourself. If you can't, you need more help than you're going to get here. But I didn't certainly didn't say anything about adding it before anything. When you're told that "you don't need it all in memory at once", you shouldn't be asking how to add code in addition to what you already have. This is really pretty elementary.
Whatever it says in the Javadoc. Nobody is going to repeat that for you here.
I still miss understand something, I try the following , but did not get the image ,kindle told me , what I miss?


private static byte[] load(String filename) throws IOException {
byte[] buffer = new byte[4096];
FileInputStream fis = new FileInputStream(filename);
// byte[] data = new byte[fis.available()];
byte[] data= new byte[0];

fis.close();
}
return data;
}
I try the following
Don't bother. It is complete and utter nonsense, and it bears no resemblance to anything that has been suggested here.
What did I miss?
You missed everything that's been said in this thread, starting with what I told you ten days ago:
You are also assuming that the file size fits into an int; that read() fills the buffer; and that the file fits into available memory. None of these assumptions is tenable.

Any solution that relies on reading entire files into memory is a problem waiting to happen. It is bound to fail sooner or later when the file gets too large, and when it does fail there is nothing the user can do about it. A very poor failure mode. The idea is to process files a record at a time, or a line at a time.
and what I told you a mere eight days ago:
If the file type is image and you are about to display it you should use javax.imageio.ImageIO.read(URL). If you're just saving it, you don't need it all in memory at once, so you can use a stream copy loop
As you clearly aren't even reading what you're being told, further responses are clearly a waste of time.
At some point you just have to give up. Here is code that reads an entire file into a byte array, in memory where it might just give you out of memory exceptions because it is too much data.
import java.io.*;

public class Test {

public byte[] readEntireFile(File fp) throws IOException{

FileInputStream fs = null;

try{
fs = new FileInputStream(fp);
ByteArrayOutputStream baos = new ByteArrayOutputStream();

byte[] buffer = new byte[4096];

}

return baos.toByteArray();

} finally {
if(fs != null){
try { fs.close(); } catch(Throwable t){}
}
}

}

public static void main(String[] args){

String fileToTest = "c:\\path\\to\\somefile.ext";

try{
Test test = new Test();
}