This discussion is archived
11 Replies Latest reply: Apr 30, 2013 1:15 PM by aksarben RSS

Slicing a file

1006106 Newbie
Currently Being Moderated
Hello Guys,
first I am new here on this forum with hope to clear my frustation against JAVA. Basicly, I like JAVA... Due an internal company project, I have to develop an application in JAVA, instead of in C++ (this my first experience with java after severval years of programing). So I have extended programing knowledge.
Now there is a situation and I cant deal with that.

Basicly, I want to slice a file from offset 5-50 and encoding this with base64 and return the String. For this, I created a Method. The problem is: My Result is always "AAAAAAAAAAAAAA=" (depending on length)... I just reached my limit.

The code is simple:
     private String sliceFile(File fHandler, int start, int length) {
        String out = new String();
        
        int allocate = 0;
        int remains = (int)fHandler.length() - start;
        if(remains < length) allocate = remains;
        else remains = length;
        
        byte bytes[] = new byte[allocate];
            
        try {
            RandomAccessFile fAccess = new RandomAccessFile(fHandler, "r");
            fAccess.read(bytes, start, length);
            fAccess.close();
        } catch(IOException e) {
            System.err.println(e);
        } finally {
            BASE64Encoder encoder = new BASE64Encoder();
            out = encoder.encodeBuffer(bytes);
            return out;
        }
    }
Its very simple implementation. But it wont work. I tried almost everything.
I replaced
            out = encoder.encodeBuffer(bytes);
with
            out = encoder.encodeBuffer("Hello".getBytes());
To ensure, that the base64 method is not "failing".
It seems to be that
            fAccess.read(bytes, start, length);
Is not reading properly.

Any suggestions about this?

Thank you,

- Ercan
  • 1. Re: Slicing a file
    jtahlborn Expert
    Currently Being Moderated
    you don't seem to be handling the return value from the read() method. my guess is that you actually want to use the readFully() method, though.

    more importantly, you should read the javadoc on the read/readFully methods. the "start" parameter is for the given array, not the file. you need to first seek the correct position in the file.
  • 2. Re: Slicing a file
    gimbal2 Guru
    Currently Being Moderated
    int allocate = 0;
            int remains = (int)fHandler.length() - start;
            if(remains < length) {
              allocate = remains;
            }
            else { 
              remains = length;
            }
    
            byte bytes[] = new byte[allocate];
    Sometimes allocate can be 0 according to this code, now that I cleaned it up I hope you can see that too.

    Regardless, where exactly did you read that the 'start' parameter of read() will start to read from that offset in the file? I would double check the documentation if I were you.

    http://docs.oracle.com/javase/7/docs/api/java/io/RandomAccessFile.html

    On top of that I wouldn't use RandomAccessFile at all, I would use regular IO streaming which is a far more standard way of doing any kind of I/O in Java. You can find hundreds of examples through Google.

    http://docs.oracle.com/javase/tutorial/essential/io/

    EDIT: ninja'd!
  • 3. Re: Slicing a file
    jtahlborn Expert
    Currently Being Moderated
    gimbal2 wrote:
    On top of that I wouldn't use RandomAccessFile at all, I would use regular IO streaming which is a far more standard way of doing any kind of I/O in Java. You can find hundreds of examples through Google.

    http://docs.oracle.com/javase/tutorial/essential/io/
    not sure why you'd recommend that? for reading a specific byte range of a file, RAF is by far the most logical choice.
  • 4. Re: Slicing a file
    1006106 Newbie
    Currently Being Moderated
    Of course RAF is the best method for my "problem"...

    I had a simple, but big mistake in my code. Ive noticed couple of minutes ago, that my "allocate" variable is sometime 0, because of using the wrong variable...

    I solved like this:
        private String sliceFile(File fHandler, RandomAccessFile fAccess, int start, int length) {
            String tmpOut = new String();
            
            int allocate = 0;
            int remains = (int)fHandler.length() - start;
            if(remains < length) allocate = remains;
            else allocate = length;
            
            byte bytes[] = new byte[allocate];
                
            try {
                fAccess.seek(start);
                fAccess.read(bytes, 0, allocate);
            } catch(IOException e) {
                System.err.println(e);
            } finally {
                tmpOut = encoder.encodeBuffer(bytes);
                return tmpOut;
            }
        }
    Everything work fine...

    One more question.... Ive caused an heap leak. I dont know, how to program my application, that the garbage collector frees the reserved ram.

    The method "sliceFile" is called up to 1000 times, with an output data size of approx. 1 MB.

    The call function (I am spliting the files into 1mb pieces and send them to my server):
        private void uploadNextPart() {
    
            ..............
    
            try {
                // Construct data
                tmpData = URLEncoder.encode("cUpload[UniqueName]", "UTF-8") + "=" + URLEncoder.encode(uOptions.UniqueName, "UTF-8");
                tmpData += "&" + URLEncoder.encode("cUpload[Data]", "UTF-8") + "=" + URLEncoder.encode(
                     sliceFile(uFile.getFileHandler(), uFile.getRandomAccessFile(), CurrentPart * uOptions.PartSize, uOptions.PartSize)
                , "UTF-8");
    
                // Send data
                URL url = new URL(uOptions.RequestURL);
                URLConnection conn = url.openConnection();
                conn.setDoOutput(true);
                OutputStreamWriter wr = new OutputStreamWriter(conn.getOutputStream());
                wr.write(tmpData);
                wr.flush();
    
                // Get the response
                BufferedReader rd = new BufferedReader(new InputStreamReader(conn.getInputStream()));
                String line;
                while ((line = rd.readLine()) != null) {}
                
                JavaScriptCommunication.onFileUploadedPart(uFile.getFileUniqueName(), uOptions.PartSize);
    
                uploadNextPart();
            
                wr.close();
                rd.close();
                
                System.gc();
            } catch (Exception e) {
            }
    
            ..............
    
        }
    After 102 packages (approx. 104 mbs of datas already sent to server there), my Application crashes, because of a Heap leak...
    Frustration...

    In C++, I can deallocate my object everywhere and how I want...

    Is it possible, that the garbage collector have no "breath" to free space, because of this?
                while ((line = rd.readLine()) != null) {} // could this cause an heap leak, because GC have no "breath" due the while loop?
                
                JavaScriptCommunication.onFileUploadedPart(uFile.getFileUniqueName(), uOptions.PartSize);
    
                uploadNextPart();
    Edited by: 1003103 on 29.04.2013 07:58

    Edited by: 1003103 on 29.04.2013 07:59
  • 5. Re: Slicing a file
    gimbal2 Guru
    Currently Being Moderated
    jtahlborn wrote:
    gimbal2 wrote:
    On top of that I wouldn't use RandomAccessFile at all, I would use regular IO streaming which is a far more standard way of doing any kind of I/O in Java. You can find hundreds of examples through Google.

    http://docs.oracle.com/javase/tutorial/essential/io/
    not sure why you'd recommend that? for reading a specific byte range of a file, RAF is by far the most logical choice.
    Not really (IMO), RandomAccessFile is nice if you want exactly that - random access to the file. Reading a range from offset X to offset Y is still a sequential action, although RAF does have that convenient seek() method of course.

    My main concern where people that are not accustomed to the API are concerned is that of documentation. I/O streaming is far better covered in my experience.
  • 6. Re: Slicing a file
    Tolls Journeyer
    Currently Being Moderated
    private void uploadNextPart() {
    
    ..............
    
    try {
    ....... removed other stuff that didn't seem to have a way out of this
    uploadNextPart(); <-- Call this method again.
    
    wr.close();
    rd.close();
    
    System.gc();
    } catch (Exception e) {
    }
    
    ..............
    
    }
    Aren't you just recursively calling uploadNextPart()?
  • 7. Re: Slicing a file
    jtahlborn Expert
    Currently Being Moderated
    well, i'd imagine that one problem is that you never close the RandomAccessFile. file resources tend to be expensive things that you want to close in finally blocks.

    as for diagnosing OOME's, use the "heap dump on OOME" option and open the generated heap dump in a memory profiler. 9 times out of 10 the solution is pretty obvious.
  • 8. Re: Slicing a file
    1006106 Newbie
    Currently Being Moderated
    Yes, I am recursively calling that method, because I have to upload all parts... But have to waiting till one request is through. So I have a while loop. I think while loop is very bad... I have to look for an event listener maybe, something like "urlconnectionfinished".

    I miss something like delete, like in C++... I have to figure out, why I get an Heap Leak...

    By the way, I am closeing the RandomAccessFile after a file is completly through.
  • 9. Re: Slicing a file
    Tolls Journeyer
    Currently Being Moderated
    Where in your code is uFile declared?
    What does getRandomAccessFile do?
    And you don't close the BufferedReader for the response until after all the recursion...so each time you go round this loop you are adding a reader.
    Same goes for the output to the URL. A new one is created each time, but is not closed off.

    None of these things get a chance to go out of scope, so you will end up with 100+ readers and 100+ output streams, 100+ URLs and 100+ connections.

    The call to gc() at the end is pointless.

    As said, though, since this is an OOME your first port of call is the heap dump, which should tell you exactly what is filling the memory.

    Out of curiosity, how much memory have you given the app?
  • 10. Re: Slicing a file
    1006106 Newbie
    Currently Being Moderated
    After I analysed the Heap Dump, I found the problem. Ive solved it...

    I will post the solution tomorrow for people with similar problems :)
  • 11. Re: Slicing a file
    aksarben Journeyer
    Currently Being Moderated
    Glad the forum was able to help out. On a personal note, I spent 5 years in C++ in a previous lifetime, and became pretty proficient with it. Due to a job change, I switched to Java, which I've been doing now for about 10 years. To be honest, I can say I enjoy Java much more than C++. It has it quirks & learning curve, like any language. But once you get familiar with it, I found it much easier to use.

    As a wise man once said, "You can shoot yourself in the foot in any language, but in C++ you blow your whole darn leg off!"

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points