13 Replies Latest reply on Nov 29, 2006 12:04 PM by JosAH

    RandomAccessFile's getFilePointer() does it take time?

    807607
      I've realized that the seek method takes several ms and i am using getFilePointer() method in order to check if i need to seek. Will it be better if i save the current pointer (position in file) myself instead?
      Is this method problematic?

      Thanks,
      Snayit
        • 1. Re: RandomAccessFile's getFilePointer() does it take time?
          JosAH
          A physical seek operation has almost nothing to do with a logical seek
          operation due to disk fragmentation. Don't bother about keeping track
          of a "current" file pointer yourself. Simply try to minimize the number of
          logical seek operations. On top of that: the OS keeps a couple of LRU
          (Least Recently Used) and LA (Look Ahead) buffers itself, so you could
          be lucky sometimes. In short: don't try to outsmart the OS's filing system.

          kind regards,

          Jos
          • 2. Re: RandomAccessFile's getFilePointer() does it take time?
            807607
            Hi,
            What did you mean by "to minimize the number of logical seek operations" please?

            I thought outsmarting the OS was the right way to go, since i am having much trouble reading from disk - it seems like it takes too much time and i don't know what is to blame (besides the disk itself): Java IO?! The JVM?

            I did strace for the IO operations and it looks alot different then when writing it in C++ (which i really don't want to do).
            • 3. Re: RandomAccessFile's getFilePointer() does it take time?
              791266
              Hi,
              What did you mean by "to minimize the number of
              logical seek operations" please?
              Do only call seek when you need to. Call getFilePointer as often as you want.
              I thought outsmarting the OS was the right way to go,
              since i am having much trouble reading from disk - it
              seems like it takes too much time and i don't know
              what is to blame (besides the disk itself): Java IO?!
              The JVM?
              What you doing? Reading or writing? You can use a memory mapped file if you only want to read (you can use if for writes as well, but it gets complicated if you want the file to dynamically grow)

              Kaj
              • 4. Re: RandomAccessFile's getFilePointer() does it take time?
                JosAH
                I thought outsmarting the OS was the right way to go, since i am
                having much trouble reading from disk - it seems like it takes too
                much time and i don't know what is to blame (besides the disk itself):
                Java IO?! The JVM?
                I'm afraid you have to blame your own program; the disk filing system
                does its best to optimize real physical disk reads (and writes). If you're
                disturbing the execution of its algorithms by seeking your file all over
                the place you're effectively slowing down your program.

                Only seek when its really necessary, try to store logically consecutive
                objects physically consecutive etc. etc.

                kind regards,

                Jos
                • 5. Re: RandomAccessFile's getFilePointer() does it take time?
                  807607
                  I am mostly reading (but also writing some) - i have a server that gets requests for files parts and i send them the part they requested.

                  I have many files and they ask for different parts. Sometimes i receive file parts myself and i write.

                  I am using NIO and my communication seems fine but my bottleneck seems to be reading the file parts from disk - i see that most of the time i am waiting for the disk. I am getting better results with multiple threads (nio threads, non-blocking for comm. but blocked IO) - also tried to use transferTo but with worse results.

                  I am using the simple read method in file channel - how can i use the memory mapped file? What is the difference between that and "regular" read?
                  • 6. Re: RandomAccessFile's getFilePointer() does it take time?
                    JosAH
                    I am mostly reading (but also writing some) - i have a server that gets
                    requests for files parts and i send them the part they requested.
                    Those file parts, are those, say, fixed sized page bounded parts? e.g.
                    four kilobytes blocks or similar?

                    kind regards,

                    Jos
                    • 7. Re: RandomAccessFile's getFilePointer() does it take time?
                      807607
                      Yes - but much much bigger - i am dividing it to 180K but i can't control which file and the position and it varies so a lot is read from the disk.
                      • 8. Re: RandomAccessFile's getFilePointer() does it take time?
                        791266
                        @Op. How many simultaneous read/writes do you have? A disk is really bad when it comes to "simultaneous" accesses.

                        Example of usage of a memory mapped file:
                             private RandomAccessFile randomAccessfile;
                             private FileChannel channel;
                             private MappedByteBuffer data
                         
                             :
                        
                             randomAccessfile = new RandomAccessFile(name, "rw");
                             channel = randomAccessfile.getChannel();
                             data = channel.map(MapMode.READ_WRITE, 0, filesize);
                        Read the javadoc for MappedByteBuffer or google on it.

                        Kaj
                        • 9. Re: RandomAccessFile's getFilePointer() does it take time?
                          807607
                          Sadly, i have few hundreds simultaneous reads (lot fewer writes).

                          I can't seem to use all disk speed (if the spec says 170Mbit - i get only 50-70Mbit at most and not usually). Since the calls are mui random i can't know in advanced which will be the next call.

                          I tried not re-open the file-descriptor each time but that did not improve much, i did do not seek if no need but i think the problem is the random calls (and the disk speed itself).
                          • 10. Re: RandomAccessFile's getFilePointer() does it take time?
                            791266
                            Sadly, i have few hundreds simultaneous reads (lot
                            fewer writes).

                            I can't seem to use all disk speed (if the spec says
                            170Mbit - i get only 50-70Mbit at most and not
                            usually). Since the calls are mui random i
                            can't know in advanced which will be the next call.

                            I tried not re-open the file-descriptor each time but
                            that did not improve much, i did do not seek if no
                            need but i think the problem is the random calls (and
                            the disk speed itself).
                            You probably want to read data into memory or distribute the data over more disks. Did you try to use memory mapped files?

                            Kaj
                            • 11. Re: RandomAccessFile's getFilePointer() does it take time?
                              807607
                              Hardware solution is not possible yet (till i give more restults).

                              Another related question: i wrote a small program that reads from 1 file in a random location 180K each time and someone else wrote the same in C++ and he got better results by far. How is this possible?
                              I need to prove that Java is the right way to go here (lucky me, huh?!) but the results really sucks. Did you encountered such thing?
                              • 12. Re: RandomAccessFile's getFilePointer() does it take time?
                                791266
                                He might be caching data or doing less seeks.

                                Kaj
                                • 13. Re: RandomAccessFile's getFilePointer() does it take time?
                                  JosAH
                                  Yes - but much much bigger - i am dividing it to 180K but i can't control
                                  which file and the position and it varies so a lot is read from the disk.
                                  You could cache those 180KB pages in a LRU (Least Recently Used)
                                  list. If a page has been read before you don't have to get it from the disk
                                  again if the page is still present in the LRU list.

                                  In a low priority thread you remove the last pages from that list if needed.

                                  kind regards,

                                  Jos