7 Replies Latest reply: Jul 16, 2012 1:13 PM by EJP RSS

    Sending a big file

    user10878887
      I must send a big file: 130MB to a postgresql server and I receive the message Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: Java heap space
      I increased the project memory with thise options: -Xms600m -Xmx1024m -XX:MaxHeapSize=1024m but it is not enough.
      If the file is 130MB and the java memory is 1024MB why isn't the memory enough?
      Is java memory measured in bites and I need 130*8=1040 Mbites to send this file?
      Can I do this with a lot less memory?
      Is there a "standard" solution to this problem?
      try {
         gdb.exec("INSERT INTO testxml VALUES(1, '"+
                 new String(FileUtils.readFileToByteArray(new File("Nom.xml")), "UTF-8")
                 +"')");
      } catch (IOException ex) {
          Logger.getLogger(main_class.class.getName()).log(Level.SEVERE, null, ex);
      }
        • 1. Re: Sending a big file
          sabre150
          For some reason you read the whole of the file into memory which, as you have found out, is almost never a good idea. If you use a BLOB then you don't have to load the whole file into memory. I don't know about a standard solution but when loading files into a database I split a file into segments of size the smaller of the maximum BLOB size for the database I am using or 1 MByte.

          P.S. I hate your exception handling. Surely you can't carry on if you get an IOException yet all you do is log it and continue.
          • 2. Re: Sending a big file
            rp0428
            >
            Is there a "standard" solution to this problem?
            >
            To troubleshoot this type of problem start with a simpler example with a small number of bytes. If I were you I would use exactly the same file but would test this
            FileUtils.readFileToByteArray
            to make sure it isn't the problem.

            So instead of doing an insert just try to create a string
            String myString = new String(FileUtils.readFileToByteArray(new File("Nom.xml")), "UTF-8");
            But modify the readFileToByteArray function to just read 10 bytes and see if that works. If it works then try reading 32k (32767) bytes.

            Then try reading the whole 130MB. If that works for creating the string itself then repeat the tests using the INSERT. But start again with a 10 byte string to make sure all of the pieces work. Then increase the length of the string until it fails.

            A process like the above will tell you where it is failing.
            • 3. Re: Sending a big file
              EJP
              Reading the entire file into memory just wastes time and space.
              byte[] buffer = new byte[8192]; // or more
              int count;
              while ((count = in.read(buffer)) > 0)
              {
                out.write(buffer, 0, count);
              }
              • 4. Re: Sending a big file
                rp0428
                >
                Reading the entire file into memory just wastes time and space.
                >
                Yep! But he is not writing it to a file. For some reason he is trying to insert the whole thing into postgresql as a string
                gdb.exec("INSERT INTO testxml VALUES(1, '"+
                           new String(FileUtils.readFileToByteArray(new File("Nom.xml")), "UTF-8")
                           +"')");
                • 5. Re: Sending a big file
                  EJP
                  Then he should be using a blob, or CLOB. To which you should write via stream as above.
                  • 6. Re: Sending a big file
                    user10878887
                    I wiill try to feed the stream to postgres with BlobInputStream
                    http://jdbc.postgresql.org/documentation/publicapi/org/postgresql/largeobject/BlobInputStream.html
                    • 7. Re: Sending a big file
                      EJP
                      BlobOutputStream, you mean.