This discussion is archived
1 2 Previous Next 24 Replies Latest reply: Aug 12, 2010 10:16 AM by 800330 RSS

XMLEncoder runs out of Memory when writing pdf files, even with base64 enc.

843790 Newbie
Currently Being Moderated
Hi,

I'm trying to write pdf-files to the object-Stream of java.beans.XMLEncoder. With small files it works fine and the base64 encoding is used. But when I use bigger files, especially scanned files (only images), it consumes a lot of memory and finally I ran out of it. Even setting it to 1.5 GB (with -Xmx; which is nearly the max for 32 Bit OS) it still happens.

The pdf-file is about 3.5 MB. Optimized for Acrobat 8.0 or later, just below 1 M.

I tried to save only this file, but the memory consumption is still the same.

How can I solve this?

Thanks a lot.

Greetings Michael
    /** Base64 Encoder */
    private static final BASE64Encoder base64enc = new sun.misc.BASE64Encoder();

    /** Base64 Decoder */
    private static final BASE64Decoder base64dec = new sun.misc.BASE64Decoder();

  /**
     * @return the pdf in base64 encoding
     */
    public String getPdf() {
     if (this.pdf != null) {
         return base64enc.encodeBuffer(this.pdf);
     }
     return null;
    }


    /**
     * @param pdf
     *            the pdf to set in base64 encoding
     */
    public void setPdf(final String byteStream) {
     try {
         if (byteStream != null) {
          this.pdf = base64dec.decodeBuffer(byteStream);
         }
     }
     catch (final IOException ioe) {
         /* can't do anything. */
         LOG.error(ioe.getMessage(), ioe);
     }
    }
  • 1. Re: XMLEncoder runs out of Memory when writing pdf files, even with base64 enc.
    800330 Explorer
    Currently Being Moderated
    Why do you include the Base64 encoding? It's not compressing (as if PDF is not compressed already) and will make the amount of data larger...

    Could I suggest to break your problem in smaller chunks to identify the cause? Is it the XMLEncoder that runs out of memory or the Base64 encoding? Try to find out by constructing some test classes that verify the two things separately.

    I am looking forward to see your progress in this topic. One of my applications is able to write the main data structure into an XML file using hand-crafted XML generation (such that it may be in a non Java XML context) on one end and uses serialization to transfer the data over RMI. I am considering to RMI-transfer the XML as a string to never encounter an UnmarshallException again. The one thing that the XML yet doesn't do is inline the contents of referenced PDF file...
  • 2. Re: XMLEncoder runs out of Memory when writing pdf files, even with base64 enc.
    843790 Newbie
    Currently Being Moderated
    Hi, thanks for your reply.

    The problem is that the XMLEncoder treats my byte-Array as a normal array.

    so the encoder encodes every single byte with code like this
    <void index="0">
    <byte>111</byte>
    </void>
    which consumes a lot of space. Base64-Encoding is used for E-Mails and even if the resulting String is most likely 4 times as big as before, that is better than the above solution. So even if I could avoid this overhead, it would fail if the source file gets bigger. Also it should consume max about 20 Megs (3.5 Megs is the size of original file) of extra heap space, but not 1,5 GB. I think the problem is that XML-Encoder generates to much meta-data about the resulting string.

    I've already tested that the base encoder is no problem. Also I tried to split up the String into an String[]-Array or an List<String>, which hadn't helped either. (Unless I would store each line manually by hand. A Persistence-Delegate which does something like that would be nice.) Also Stringbuffer isn't reducing the heap space consumption.

    XMLEncoder works nice, but heap space consumption is always a problem, especially with large object-graphs, but not into this extremes. Also it doesn't likes EnumMaps and Collectionclasses from Hibernate. It is quite unlikely that the default heap-size of 64 Megs would be enough.

    Greetings Michael
  • 3. Re: XMLEncoder runs out of Memory when writing pdf files, even with base64 enc.
    800330 Explorer
    Currently Being Moderated
    Okay, now I wonder: how do you set the PDF in the first place, when creating an instance directly and not by de-serialization? The setPDF method assumes that the incoming String is base64 encoded. Naively, you could have read just the (binary) bytes from the file and passed them into the setter, that happily base64 decodes them...

    (The trouble with posting a code fragment and not all of it, is that I am now just guessing)
  • 4. Re: XMLEncoder runs out of Memory when writing pdf files, even with base64 enc.
    jtahlborn Expert
    Currently Being Moderated
    the problem with diagnosing memory problems, is that anything which does not include using a memory profiler is just guessing. i can't imagine what you could possibly be doing with a 3.5MB file which would blow out the memory on a jvm set with a 1.5GB memory max. even blowing that file up to 100 times it's size shouldn't be a problem. so, the problem is probably something completely different. i'd suggest enabling the "heap dump on OOM" jvm option and then evaluating the resulting heap dump in a memory profiler. there are plenty of decent free ones and some excellent non-free ones. until you do that, it's all guessing.
  • 5. Re: XMLEncoder runs out of Memory when writing pdf files, even with base64 enc.
    800330 Explorer
    Currently Being Moderated
    By the way, do you know whether Base64Encoder.encodeBuffer is thread-safe? The way you've set things up now relies on (1) the class being provided by the JVM (which is not guaranteed) and (2) being thread-safe. The same encoder instance is used for all instances and possibly concurrently by all Threads XmlEncoding an instance.

    You see this approach in DateFormatter, Cyphers and the like and come haunt you once the code is deployed and running under load... (See [ wonky dating|http://www.javaspecialists.eu/archive/Issue172.html] )
  • 6. Re: XMLEncoder runs out of Memory when writing pdf files, even with base64 enc.
    843790 Newbie
    Currently Being Moderated
    Hi,

    first of all, I just want to mention, that base64 normally increases the size only about 20%, cause 40 Bits (5 Bytes) are mapped to 48 Bits (6 Bytes).

    The property is directly accessed by hibernate (database). I also have get/setPlainPdf methods to do file-upload within the program.
    By a subclass of DefaultPersitenceDelegates I redirect the XML-Encoder to the appropriate constructor, so the get/set-methods are just for the XML-Encoder to store base64 and read base 64. Which works fine for other pdf's. I was quite happy to move the base64-encoding from the storage-routine to the pdf-file-class. (Was quite ugly before, cause I had to remove them from my object-graph, tracking the relations and restoring the relations afterwards. Which was also needed to special treatment during loading.)

    I don't really need a heap profile or a heap dump, cause if I run the program in the debugger, it increases exactly then, when I pass the object to the XMLEncoder (Single-Line-statement), so I'm quite sure that that's the cause. So no loop or circle, causes the problem.

    Also the "Thread-Safe" issue, doesn't seems to be a problem for me, cause the storing is running in one single thread (only to ensure repainting of gui) and if XMLEncoder would use threads (which I'm quite sure it doesn't), it has to care for it. Most of the Collection-Classes aren't thread safe either.

    So I would need help from someone, who has deep Knowledge of the XMLEncoder.

    Thanks or all your posts.

    Greetings Michael
  • 7. Re: XMLEncoder runs out of Memory when writing pdf files, even with base64 enc.
    800330 Explorer
    Currently Being Moderated
    As I am obviously not an XMLEncoder expert, I browsed the forum's serialization section to learn some more and came across [this tip to always attach an ExceptionListener when chasing problems|http://forums.sun.com/thread.jspa?messageID=11018669#11018669].

    I wonder whether you could come up with an SSCCE to demontrate the problem you're seeing.
  • 8. Re: XMLEncoder runs out of Memory when writing pdf files, even with base64 enc.
    843790 Newbie
    Currently Being Moderated
    Hi,

    SSCCE -- see below.


    I use this PersistenceDelegate
    enc.setPersistenceDelegate(PdfFile.class, new DefaultPersistenceDelegate(new String[] { "filename",
              "pdf", "comment" }));
    and I'm setting this Exception listener.
      enc.setExceptionListener(new ExceptionListener() {
                   /     Avoid multiple popups of error warnings */*
    *               private boolean messageShown = false;*
    
    *               public void exceptionThrown(final Exception e) {*
    *                   e.printStackTrace();*
    *                   XML_WriteOperations.LOG.error(e.getMessage(), e);*
    *                   if (!this.messageShown) {*
    *                    JOptionPane.showMessageDialog(MainFrame.INSTANCE.getFrame(),*
    *                         "File may be corrupt!\r\n"  +e.getClass().getSimpleName()< </em>" "*
    *                              + e.getMessage(), "Problem occured!",*
    *                         JOptionPane.WARNING_MESSAGE);*
    *                    this.messageShown = true;*
    *                   }*
    *               }*
    *              });
    *

    This isn't really helpful, cause it doesn't provides information on which type of data it was working on, unless the exception itself gives you a clue.
  • 9. Re: XMLEncoder runs out of Memory when writing pdf files, even with base64 enc.
    843790 Newbie
    Currently Being Moderated
    So putting it all together for a test program:

    *
    *
    *import java.beans.*;
    import java.io.*;*
    
    *import javax.swing.JOptionPane;*
    
    *import org.apache.log4j.Logger;*
    
    *import sun.misc.BASE64Decoder;*
    *import sun.misc.BASE64Encoder;*
    
    */**
     *@author Michael Gr&uuml;newald*
     
    */*
    *public class TestPDF_withEncoder {*
    
    *    /* Put your pdf-filenames here. */*
    *    private static final String[] pdfFilenames = { ".\\TestPdfs\\1.pdf", ".\\TestPdfs\\2.pdf",*
    *         ".\\TestPdfs\\3.pdf", ".\\TestPdfs\\4.pdf", ".\\TestPdfs\\5.pdf" };*
    
    *    /* Target filename */*
    *    private static final String targetFilename = ".\\TestPdfs\\enc_out.xml";*
    
    *    private static Logger LOG = Logger.getLogger(TestPDF_withEncoder.class);*
    
    *    /** Base64 Encoder */*
    *    private static final BASE64Encoder base64enc = new sun.misc.BASE64Encoder();*
    
    *    /** Base64 Decoder */*
    *    private static final BASE64Decoder base64dec = new sun.misc.BASE64Decoder();*
    
    *    public static void main(final String args[]) {*
    *     openXMLEncoder(new File(targetFilename), pdfFilenames);*
    *     System.exit(0);*
    *    }*
    
    *    private static void openXMLEncoder(final File file, final String[] pdfNames) {*
    *     FileOutputStream fos = null;*
    *     BufferedOutputStream out = null;*
    *     XMLEncoder enc = null;*
    
    *     try {*
    *         if (file != null) {*
    *          System.out.println(file.getAbsolutePath());*
    *          if (file.exists()) {*
    *              file.delete();*
    *          }*
    *          file.createNewFile();*
    *          if (file.canWrite()) {*
    
    *              fos = new FileOutputStream(file);*
    *              out = new BufferedOutputStream(fos);*
    *              enc = new XMLEncoder(new BufferedOutputStream(fos));*
    
    *              enc.setPersistenceDelegate(TestPDF_withEncoder.class, new DefaultPersistenceDelegate(*
    *                   new String[] { "filename", "pdf", "comment" }));*
    
    *              enc.setExceptionListener(new ExceptionListener() {*
    *               /*                     *Avoid multiple popups of error warnings* /
                   public void exceptionThrown(final Exception e) {
                       e.printStackTrace();
                   }
                  });
                  for (final String name : pdfNames) {
                   final TestPDF_withEncoder pdf = new TestPDF_withEncoder(new File(name));
                   System.out.println(name);
                   enc.writeObject(pdf);
                   /
     Isn't possible for every pdf in my realcode, cause
     *they are part of a hughe object-graph. So you may*
     comment this out.
    */*
    *               // enc.flush();*
    *              }*
    
    *              enc.close();*
    *          }*
    
    *         }*
    *     }*
    *     catch (final Throwable e) {*
    *         e.printStackTrace();*
    *     }*
    *     finally {*
    *         if (enc != null) {*
    *          enc.close();*
    *         }*
    *         if (out != null) {*
    *          try {*
    *              out.close();*
    *          }*
    *          catch (final IOException ioe) {*
    *              LOG.warn(ioe.getMessage(), ioe);*
    *              ioe.printStackTrace();*
    *          }*
    *         }*
    *         if (fos != null) {*
    *          try {*
    *              fos.close();*
    *          }*
    *          catch (final IOException ioe) {*
    *              LOG.warn(ioe.getMessage(), ioe);*
    *              ioe.printStackTrace();*
    *          }*
    *         }*
    
    *     }*
    *    }*
    
    *    private Long id;*
    
    *    private String filename;*
    
    *    private String comment;*
    
    *    private byte[] pdf;*
    
    *    public TestPDF_withEncoder(final File file) {*
    *     this.filename = file.getName();*
    *     this.pdf = readFileIntoByteArray(file);*
    *    }*
    
    *    public TestPDF_withEncoder(final String filename) {*
    *     this.filename = filename;*
    *     final File file = new File(this.filename);*
    *     this.pdf = readFileIntoByteArray(file);*
    *    }*
    
    *    public TestPDF_withEncoder(final String filename, final byte[] plainPdf) {*
    *     this.filename = filename;*
    *     this.pdf = plainPdf;*
    *    }*
    
    *    public TestPDF_withEncoder(final String filename, final byte[] plainPdf, final String comment) {*
    *     this(filename, plainPdf);*
    *     this.comment = comment;*
    *    }*
    
    *    public TestPDF_withEncoder(final String filename, final String comment) {*
    *     this(filename);*
    *     this.comment = comment;*
    *    }*
    
    *    public TestPDF_withEncoder(final String filename, final String pdf, final String comment) {*
    *     this.filename = filename;*
    *     this.setPdf(pdf);*
    *     this.comment = comment;*
    *    }*
    
    *    private TestPDF_withEncoder(final TestPDF_withEncoder source, final boolean copyIDs) {*
    *     this.pdf = source.pdf.clone();*
    *     this.comment = source.comment;*
    *     this.filename = source.filename;*
    *     if (copyIDs) {*
    *         this.id = source.id;*
    *     }*
    *    }*
    
    *    /**
     *@return the comment*
    /
        public String getComment() {
         return this.comment;
        }
    
        /
     *@return the filename*
    /
        public String getFilename() {
         return this.filename;
        }
    
        /
     *@return the id*
    /
        public Long getId() {
         return this.id;
        }
    
        /
     *@return the pdf in base64 encoding*
    /
        public String getPdf() {
         if (this.pdf != null) {
             return base64enc.encodeBuffer(this.pdf);
         }
         return null;
        }
    
        /
     *@return the pdf (normal not in base64 encoding)*
    /
        public byte[] getPlainPdf() {
         return this.pdf;
        }
    
        private byte[] readFileIntoByteArray(final File file) {
         if (file.length() > Integer.MAX_VALUE) {
             JOptionPane.showMessageDialog(null, file.getAbsolutePath(), "File is too big!",
                  JOptionPane.ERROR_MESSAGE);
             return new byte[0];
         }
         final byte[] array = new byte[(int) file.length()];
    
         FileInputStream in = null;
         try {
             in = new FileInputStream(file);
             final int numsOBytes = in.read(array);
             if (numsOBytes < file.length()) {
              JOptionPane.showMessageDialog(null, file.getAbsolutePath(), "File wasn't read completly",
                   JOptionPane.ERROR_MESSAGE);
             }
             return array;
         }
         catch (final IOException ioe) {
             ioe.printStackTrace();
             return new byte[0];
         }
         finally {
             try {
              if (in != null) {
                  in.close();
              }
             }
             catch (final IOException ioe) {
              ioe.printStackTrace();
             }
         }
        }
    
        /
     *@param comment*
                the comment to set
    */*
    *    public void setComment(final String comment) {*
    *     this.comment = comment;*
    *    }*
    
    *    /**
     *@param filename*
                the filename to set
    */*
    *    public void setFilename(final String filename) {*
    *     this.filename = filename;*
    *    }*
    
    *    /**
     *@param id*
                the id to set
    */*
    *    public void setId(final Long id) {*
    *     this.id = id;*
    *    }*
    
    *    /**
     *@param pdf*
                the pdf to set in base64 encoding
    */*
    *    public void setPdf(final String byteStream) {*
    *     try {*
    *         if (byteStream != null) {*
    *          this.pdf = base64dec.decodeBuffer(byteStream);*
    *         }*
    *     }*
    *     catch (final IOException ioe) {*
    *         /* can't do anything. */*
    *         LOG.error(ioe.getMessage(), ioe);*
    *     }*
    *    }*
    
    *    /**
     *@param pdf*
                the pdf to set (normal not in base64 encoding)
    */*
    *    public void setPlainPdf(final byte[] pdf) {*
    *     this.pdf = pdf;*
    *    }*
    
    *    /*
     *(non-Javadoc)*
     
     *@see java.lang.Object#toString()*
    /
        @Override
        public String toString() {
         return getFilename();
        }
    }
  • 10. Re: XMLEncoder runs out of Memory when writing pdf files, even with base64 enc.
    843790 Newbie
    Currently Being Moderated
    Snipset of result:
    ?xml version="1.0" encoding="UTF-8"?> 
    <java version="1.6.0_20" class="java.beans.XMLDecoder"> 
     <object class="TestPDF_withEncoder"> 
      <string>1.pdf</string> 
      <string>JVBERi0xLjUNJeLjz9MNCjI3IDAgb2JqDTw8L0xpbmVhcml6ZWQgMS9MIDY4NjUwL08gMjkvRSAz
    
    MjYzNi9OIDUvVCA2ODI5Ny9IIFsgNTAxIDIxMV0+Pg1lbmRvYmoNICAgICAgICAgICAgICAgICAg
    
    DQo1MCAwIG9iag08PC9EZWNvZGVQYXJtczw8L0NvbHVtbnMgNC9QcmVkaWN0b3IgMTI+Pi9GaWx0
    With my special file and without flush it reaches 1.5 GB and produces an "java.lang.OutOfMemoryError: Java heap space" with -Xms256m -Xmx1536m as options. Even with flush or as single file this happens.

    So anything goes wrong, but how can I fix it? The above sample output is produced with other pdf-files, which only consumes about 200-300 MB.

    I have uploaded the [problematic file|http://www.file-upload.net/download-2736843/5.pdf.html] .
    I hope you can reproduce the problem now.

    Thanks.

    Greetings Michael

    PS: Had to split up my post, cause of max. char limitations. The stars was added by the forum tool, Sorry for that.
  • 11. Re: XMLEncoder runs out of Memory when writing pdf files, even with base64 enc.
    843790 Newbie
    Currently Being Moderated
    Cause the above code is missed up a little bit, here is it again.
    import java.beans.*;
    import java.io.*;
    
    import javax.swing.JOptionPane;
    
    import org.apache.log4j.Logger;
    
    import sun.misc.BASE64Decoder;
    import sun.misc.BASE64Encoder;
    
    /**
     * @author Michael Grünewald
     * 
     */
    public class TestPDF_withEncoder {
    
        /* Put your pdf-filenames here. */
        private static final String[] pdfFilenames = { /*
                                      * ".\\TestPdfs\\1.pdf",
                                      * ".\\TestPdfs\\2.pdf",
                                      * ".\\TestPdfs\\3.pdf",
                                      * ".\\TestPdfs\\4.pdf",
                                      */".\\TestPdfs\\5.pdf" };
    
        /* Target filename */
        private static final String targetFilename = ".\\TestPdfs\\enc_out.xml";
    
        private static Logger LOG = Logger.getLogger(TestPDF_withEncoder.class);
    
        /** Base64 Encoder */
        private static final BASE64Encoder base64enc = new sun.misc.BASE64Encoder();
    
        /** Base64 Decoder */
        private static final BASE64Decoder base64dec = new sun.misc.BASE64Decoder();
    
        public static void main(final String args[]) {
         openXMLEncoder(new File(targetFilename), pdfFilenames);
         System.exit(0);
        }
    
        private static void openXMLEncoder(final File file, final String[] pdfNames) {
         FileOutputStream fos = null;
         BufferedOutputStream out = null;
         XMLEncoder enc = null;
    
         try {
             if (file != null) {
              System.out.println(file.getAbsolutePath());
              if (file.exists()) {
                  file.delete();
              }
              file.createNewFile();
              if (file.canWrite()) {
    
                  fos = new FileOutputStream(file);
                  out = new BufferedOutputStream(fos);
                  enc = new XMLEncoder(new BufferedOutputStream(fos));
    
                  enc.setPersistenceDelegate(TestPDF_withEncoder.class, new DefaultPersistenceDelegate(
                       new String[] { "filename", "pdf", "comment" }));
    
                  enc.setExceptionListener(new ExceptionListener() {
                   /*                      * Avoid multiple popups of error warnings */
                   public void exceptionThrown(final Exception e) {
                       e.printStackTrace();
                   }
                  });
                  for (final String name : pdfNames) {
                   final TestPDF_withEncoder pdf = new TestPDF_withEncoder(new File(name));
                   System.out.println(name);
                   enc.writeObject(pdf);
                   /*
                    * Isn't possible for every pdf in my realcode, cause
                    * they are part of a hughe object-graph. So you may
                    * comment this out.
                    */
                   enc.flush();
                  }
    
                  enc.close();
              }
    
             }
         }
         catch (final Throwable e) {
             e.printStackTrace();
         }
         finally {
             if (enc != null) {
              enc.close();
             }
             if (out != null) {
              try {
                  out.close();
              }
              catch (final IOException ioe) {
                  LOG.warn(ioe.getMessage(), ioe);
                  ioe.printStackTrace();
              }
             }
             if (fos != null) {
              try {
                  fos.close();
              }
              catch (final IOException ioe) {
                  LOG.warn(ioe.getMessage(), ioe);
                  ioe.printStackTrace();
              }
             }
    
         }
        }
    
        private Long id;
    
        private String filename;
    
        private String comment;
    
        private byte[] pdf;
    
        public TestPDF_withEncoder(final File file) {
         this.filename = file.getName();
         this.pdf = readFileIntoByteArray(file);
        }
    
        public TestPDF_withEncoder(final String filename) {
         this.filename = filename;
         final File file = new File(this.filename);
         this.pdf = readFileIntoByteArray(file);
        }
    
        public TestPDF_withEncoder(final String filename, final byte[] plainPdf) {
         this.filename = filename;
         this.pdf = plainPdf;
        }
    
        public TestPDF_withEncoder(final String filename, final byte[] plainPdf, final String comment) {
         this(filename, plainPdf);
         this.comment = comment;
        }
    
        public TestPDF_withEncoder(final String filename, final String comment) {
         this(filename);
         this.comment = comment;
        }
    
        public TestPDF_withEncoder(final String filename, final String pdf, final String comment) {
         this.filename = filename;
         this.setPdf(pdf);
         this.comment = comment;
        }
    
        private TestPDF_withEncoder(final TestPDF_withEncoder source, final boolean copyIDs) {
         this.pdf = source.pdf.clone();
         this.comment = source.comment;
         this.filename = source.filename;
         if (copyIDs) {
             this.id = source.id;
         }
        }
    
        /**
         * @return the comment
         */
        public String getComment() {
         return this.comment;
        }
    
        /**
         * @return the filename
         */
        public String getFilename() {
         return this.filename;
        }
    
        /**
         * @return the id
         */
        public Long getId() {
         return this.id;
        }
    
        /**
         * @return the pdf in base64 encoding
         */
        public String getPdf() {
         if (this.pdf != null) {
             return base64enc.encodeBuffer(this.pdf);
         }
         return null;
        }
    
        /**
         * @return the pdf (normal not in base64 encoding)
         */
        public byte[] getPlainPdf() {
         return this.pdf;
        }
    
        private byte[] readFileIntoByteArray(final File file) {
         if (file.length() > Integer.MAX_VALUE) {
             JOptionPane.showMessageDialog(null, file.getAbsolutePath(), "File is too big!",
                  JOptionPane.ERROR_MESSAGE);
             return new byte[0];
         }
         final byte[] array = new byte[(int) file.length()];
    
         FileInputStream in = null;
         try {
             in = new FileInputStream(file);
             final int numsOBytes = in.read(array);
             if (numsOBytes < file.length()) {
              JOptionPane.showMessageDialog(null, file.getAbsolutePath(), "File wasn't read completly",
                   JOptionPane.ERROR_MESSAGE);
             }
             return array;
         }
         catch (final IOException ioe) {
             ioe.printStackTrace();
             return new byte[0];
         }
         finally {
             try {
              if (in != null) {
                  in.close();
              }
             }
             catch (final IOException ioe) {
              ioe.printStackTrace();
             }
         }
        }
    
        /**
         * @param comment
         *            the comment to set
         */
        public void setComment(final String comment) {
         this.comment = comment;
        }
    
        /**
         * @param filename
         *            the filename to set
         */
        public void setFilename(final String filename) {
         this.filename = filename;
        }
    
        /**
         * @param id
         *            the id to set
         */
        public void setId(final Long id) {
         this.id = id;
        }
    
        /**
         * @param pdf
         *            the pdf to set in base64 encoding
         */
        public void setPdf(final String byteStream) {
         try {
             if (byteStream != null) {
              this.pdf = base64dec.decodeBuffer(byteStream);
             }
         }
         catch (final IOException ioe) {
             /* can't do anything. */
             LOG.error(ioe.getMessage(), ioe);
         }
        }
    
        /**
         * @param pdf
         *            the pdf to set (normal not in base64 encoding)
         */
        public void setPlainPdf(final byte[] pdf) {
         this.pdf = pdf;
        }
    
        /*
         * (non-Javadoc)
         * 
         * @see java.lang.Object#toString()
         */
        @Override
        public String toString() {
         return getFilename();
        }
    }
  • 12. Re: XMLEncoder runs out of Memory when writing pdf files, even with base64 enc.
    800330 Explorer
    Currently Being Moderated
    Are you having fun? What a beast that XMlEncoder! I cannot wrap my head around its happy flow-mode of operation entirely. I've looked for online resources with some more in-depth but tutorial-style explanations but the API and the [Using XMlEncoder article|http://java.sun.com/products/jfc/tsc/articles/persistence4/] seem to remain the best I've found so far.

    I took your code (edited a bit so it worked for me, I believe one of the S's in SSCCE is for small or at least simple...) and ran it. Just As you claim: it works for regular PDFs and fails on a larger one (I tried a 5M one).

    No solution yet, but: if you put a breakpoint in getPdf() and run with a smaller PDF you'll notice that you pass getPdf() three times before successful completion! I am suspecting that your approach of a DefaultPersistenceDelegate based on the 3 argument constructor (3 too, I suspect not a coincidence) based on properties, while convenient coding is taking too much of a short-cut. The XMLEncoder internal bindings member builds up to contain at least 3 copies of your bean. (this involves the mutatesTo() and then some subsequent writeObject() calls, which in the end I could not follow (mentally) anymore - see 1st paragraph).

    Be that as it may, for a 1Gb heap you could have 200 5Mb copies of your bean before it gets exhausted - but that perhaps is way too naive too...
  • 13. Re: XMLEncoder runs out of Memory when writing pdf files, even with base64 enc.
    jtahlborn Expert
    Currently Being Moderated
    when i ran this in a profiler, it became obvious how the encoder was being confused. (the problem is not multiple copies of the pdf data).

    Edited by: jtahlborn on Aug 11, 2010 10:45 AM
  • 14. Re: XMLEncoder runs out of Memory when writing pdf files, even with base64 enc.
    843790 Newbie
    Currently Being Moderated
    Hi, the three times calling getPDF is normal, cause XMLEncoder tries to store only changed attributes to save space.
    So it constructs a second instance and tries to rebuild it. This operations are coded to the XML-Encoder.
    For Comparing the two instances it needs two calls and one to write it out as far as I know.

    Sure I cloud have removed some more code from my example, but that are only some simple pieces, so I thought that wouldn't matter.I should have removed the Log4j-lib to keep it more simple, but haven't recognized it.

    So here is the reduce one and surprisingly it runs very smooth and fast. Below 120 MB.
    So something must be wrong with my code. But which piece?
    I will try adding the plainPdf-methods, cause they are the only reasons I can think about it why it doesn't works.
    Will edit or reply, when I have checked this.
    import java.beans.*;
    import java.io.*;
    
    import javax.swing.JOptionPane;
    
    import sun.misc.BASE64Decoder;
    import sun.misc.BASE64Encoder;
    
    /**
     * @author Michael Grünewald
     * 
     */
    public class TestPDF_withEncoder {
    
        /* Put your pdf-filenames here. */
        private static final String[] pdfFilenames = { ".\\TestPdfs\\1.pdf", ".\\TestPdfs\\2.pdf",
             ".\\TestPdfs\\3.pdf", ".\\TestPdfs\\4.pdf", ".\\TestPdfs\\5.pdf" };
    
        /* Target filename */
        private static final String targetFilename = ".\\TestPdfs\\enc_out.xml";
    
        /** Base64 Encoder */
        private static final BASE64Encoder base64enc = new sun.misc.BASE64Encoder();
    
        /** Base64 Decoder */
        private static final BASE64Decoder base64dec = new sun.misc.BASE64Decoder();
    
        public static void main(final String args[]) {
         openXMLEncoder(new File(targetFilename), pdfFilenames);
         System.exit(0);
        }
    
        private static void openXMLEncoder(final File file, final String[] pdfNames) {
         FileOutputStream fos = null;
         BufferedOutputStream out = null;
         XMLEncoder enc = null;
    
         try {
             if (file != null) {
              System.out.println(file.getAbsolutePath());
              if (file.exists()) {
                  file.delete();
              }
              file.createNewFile();
              if (file.canWrite()) {
    
                  fos = new FileOutputStream(file);
                  out = new BufferedOutputStream(fos);
                  enc = new XMLEncoder(new BufferedOutputStream(fos));
    
                  enc.setPersistenceDelegate(TestPDF_withEncoder.class, new DefaultPersistenceDelegate(
                       new String[] { "filename", "pdf" }));
    
                  enc.setExceptionListener(new ExceptionListener() {
                   /*                              * Avoid multiple popups of error warnings */
                   public void exceptionThrown(final Exception e) {
                       e.printStackTrace();
                   }
                  });
                  for (final String name : pdfNames) {
                   final TestPDF_withEncoder pdf = new TestPDF_withEncoder(new File(name));
                   System.out.println(name);
                   enc.writeObject(pdf);
                   /*
                    * Isn't possible for every pdf in my realcode, cause
                    * they are part of a hughe object-graph. So you may
                    * comment this out.
                    */
                   enc.flush();
                  }
    
                  enc.close();
              }
    
             }
         }
         catch (final Throwable e) {
             e.printStackTrace();
         }
         finally {
             if (enc != null) {
              enc.close();
             }
             if (out != null) {
              try {
                  out.close();
              }
              catch (final IOException ioe) {
                  ioe.printStackTrace();
              }
             }
             if (fos != null) {
              try {
                  fos.close();
              }
              catch (final IOException ioe) {
                  ioe.printStackTrace();
              }
             }
    
         }
        }
    
        private String filename;
    
        private byte[] pdf;
    
        public TestPDF_withEncoder(final File file) {
         this.filename = file.getName();
         this.pdf = readFileIntoByteArray(file);
        }
    
        public TestPDF_withEncoder(final String filename, final String pdf) {
         this.filename = filename;
         this.setPdf(pdf);
        }
    
        /**
         * @return the filename
         */
        public String getFilename() {
         return this.filename;
        }
    
        /**
         * @return the pdf in base64 encoding
         */
        public String getPdf() {
         if (this.pdf != null) {
             return base64enc.encodeBuffer(this.pdf);
         }
         return null;
        }
    
        private byte[] readFileIntoByteArray(final File file) {
         if (file.length() > Integer.MAX_VALUE) {
             JOptionPane.showMessageDialog(null, file.getAbsolutePath(), "File is too big!",
                  JOptionPane.ERROR_MESSAGE);
             return new byte[0];
         }
         final byte[] array = new byte[(int) file.length()];
    
         FileInputStream in = null;
         try {
             in = new FileInputStream(file);
             final int numsOBytes = in.read(array);
             if (numsOBytes < file.length()) {
              JOptionPane.showMessageDialog(null, file.getAbsolutePath(), "File wasn't read completly",
                   JOptionPane.ERROR_MESSAGE);
             }
             return array;
         }
         catch (final IOException ioe) {
             ioe.printStackTrace();
             return new byte[0];
         }
         finally {
             try {
              if (in != null) {
                  in.close();
              }
             }
             catch (final IOException ioe) {
              ioe.printStackTrace();
             }
         }
        }
    
        /**
         * @param filename
         *            the filename to set
         */
        public void setFilename(final String filename) {
         this.filename = filename;
        }
    
        /**
         * @param pdf
         *            the pdf to set in base64 encoding
         */
        public void setPdf(final String byteStream) {
         try {
             if (byteStream != null) {
              this.pdf = base64dec.decodeBuffer(byteStream);
             }
         }
         catch (final IOException ioe) {
             /* can't do anything. */
             ioe.printStackTrace();
         }
        }
    }
    Thanks for your efforts.

    Greetings Michael

    PPS: Found the problem quicker than I thought.

    The error seem to be really caused by this method:
     /**
         * @param pdf
         *            the pdf to set (normal not in base64 encoding)
         */
        public void setPlainPdf(final byte[] pdf) {
         this.pdf = pdf;
        }
    Even I can't see any access to it, in the debugger. Renaming it to storePlainPdf solves the problem, but why?

    Only reason I could imagine would be, that XML-Encoder thinks this method is the setter of the Property, even if it violates the naming constraints. So it tries to compare the encoded version with the plain. But than it should crash wit every pdf. So I'm still clueless, what the reason is. Also providing a setPDF(byte[]), doesn't made a difference.

    I will try to change it for my complex program and hope the fix will work there too.

    Thanks.

    Greetings Michael

    Edited by: Urmech on Aug 11, 2010 8:33 AM
1 2 Previous Next