This discussion is archived
8 Replies Latest reply: Aug 14, 2012 1:50 PM by sabre150 RSS

Is CertificateFactory.generateCRL horribly inefficient?

955540 Newbie
Currently Being Moderated
This refers back to http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6670894, which appears never to have been addressed, and to have gotten worse in Java versions from 1.6_21 on: In 1.6.0_17, about 95 MB of CRLs takes 1.33GB ( GB! ) of memory. Post 1.6.0_21, the same siize in CRLs takes roughly 1.45GB!

Here is sample code demonstrating how they're being loaded:

<pre>
import java.io.*;
import java.security.cert.*;
import javax.security.auth.x500.*;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.Map;
import java.lang.Thread;

public class CRLLoader
{
private static Map<X500Principal, X509CRL> crlList = new HashMap<X500Principal, X509CRL>();

public static void main( String[] args )
{
String dirName = "path/to/crl_files";
ArrayList<File> newFiles = new ArrayList<File>();
File dir = new File( dirName );
String[] fileNames = dir.list();

for( String name : fileNames )
{
if( name.endsWith( "crl" ) )
{
newFiles.add( new File( dirName + name ) ) ;
}
}

for( File newFile: newFiles )
{
try
{
CertificateFactory cf = CertificateFactory.getInstance("X509");
FileInputStream crlFileStream = null;
X509CRL crl = null;

//Read the File
System.out.println("Adding CRL: " + newFile.toString());
crlFileStream = new FileInputStream(newFile);
crl = (X509CRL)cf.generateCRL(crlFileStream);
crlFileStream.close();

//and add to map
crlList.put(crl.getIssuerX500Principal(), crl);
}
catch( Exception e )
{
System.out.println( e.toString() );
e.printStackTrace();
System.out.println( e.getMessage() );
}
}

try
{
Thread.sleep( 10000 );// to capture mem size
}
catch( InterruptedException ie )
{
System.out.println( ie.getMessage() );
}
}
}
</pre>

Is CertificateFactory.generateCRL just horribly inefficient, or is there a far better way to code this?

Thanks in advance for any tips/info/guidance provided.
  • 1. Re: Is CertificateFactory.generateCRL horribly inefficient?
    EJP Guru
    Currently Being Moderated
    Slightly off topic but why the ArrayList of File? You don't need it. Just process each file as you encounter it in the first loop. If memory is a problem you shouldn't be double-storing. But that's a lot of data. Maybe you should be using an OCSP server.
  • 2. Re: Is CertificateFactory.generateCRL horribly inefficient?
    sabre150 Expert
    Currently Being Moderated
    952537 wrote:
    about 95 MB of CRLs takes 1.33GB ( GB! ) of memory. Post 1.6.0_21, the same siize in CRLs takes roughly 1.45GB!
    How are you measuring that? I have downloaded 9.8 MByte of CRL from Verisign and using JDK1.7.0_05 on Linux processed them using essentially your code but, though difficult to measure with any accuracy, see nothing like that sort of inflation. Using the Netbeans profiler the 9.8 MBytes seems to consume about 45 MByte of memory estimated from runs with and without storing the CRL in the map. Still large but only 1/3 of the value you experience.

    Edited by: sabre150 on Aug 14, 2012 10:11 AM
  • 3. Re: Is CertificateFactory.generateCRL horribly inefficient?
    955540 Newbie
    Currently Being Moderated
    I should have mentioned a couple of baseline details about how I'm testing this:

    Windows 2008 Server
    x64 with 32 bit Java
    Intel Core 2 Duo E8400 @ 3Ghz, 4GB RAM

    I'm measuring memory use using Windows Task Manager, watching the Memory gauge on the Performance tab. I have not yet used a MAT tool to do a closer analysis, and this code is extracted from a web-app that, not surprisingly, runs out of memory when I try to test it in place.

    Also, in this case, an OCSP server is not an option, but I know that would off-load this work. Also, there seems to be no difference between using the file names array from list() directly or using the ArrayList, again using the Task Manager gauge.

    My point is that high resource use has been noted for these classes in a prior Java version, and that it has not been addressed( if it can be ), and has gotten worse in newer versions.

    So, given that this is a known memory hog, is there a better way to handle caching CRLs?
  • 4. Re: Is CertificateFactory.generateCRL horribly inefficient?
    sabre150 Expert
    Currently Being Moderated
    952537 wrote:
    I should have mentioned a couple of baseline details about how I'm testing this:
    I'm measuring memory use using Windows Task Manager, watching the Memory gauge on the Performance tab.
    Not a good indicator since it represents the memory allocated to the JVM and not the memory currently being used to hold the CRLs.
    I have not yet used a MAT tool to do a closer analysis, and this code is extracted from a web-app that, not surprisingly, runs out of memory when I try to test it in place.
    I would not store the whole CRL set in memory especially for web-apps.

    >
    Also, in this case, an OCSP server is not an option, but I know that would off-load this work.
    I can't comment on this.
    Also, there seems to be no difference between using the file names array from list() directly or using the ArrayList, again using the Task Manager gauge.
    EJP didn't say there would be ! He was just noting that there seemed to be a redundancy in having both an array of file names and an ArrayList holding the filtered names. I would take a different approach and use a FileFilter -
         File[] crlFiles = dirFile.listFiles(new FileFilter()
            {
                @Override
                public boolean accept(File child)
                {
                    return child.isFile() && child.canRead() && child.getName().endsWith(".crl");
                }
    
            });
    
           CertificateFactory cf = CertificateFactory.getInstance("X509"); // Can be re-used so does not need to be generated within the loop
            for (File crlFile : crlFiles)
            {
                  // read the CRLs
            }
    This won't be any faster than your approach and it won't use any less memory but I think it is much cleaner.

    >
    My point is that high resource use has been noted for these classes in a prior Java version, and that it has not been addressed( if it can be ), and has gotten worse in newer versions.
    I cannot comment on the high resource usage of the classes except that I doubt if one can reduce the memory usage. As to whether or not it has gotten worse - it may be worse but I don't think you have proved it is worse since I think your measurement technique is flawed.

    >
    So, given that this is a known memory hog, is there a better way to handle caching CRLs?
    Depending on exactly how it is to be used I would consider using a relational DB ( loaded offline) with a small volatile transient cache so that any cache memory used is released when the application is running short.
  • 5. Re: Is CertificateFactory.generateCRL horribly inefficient?
    955540 Newbie
    Currently Being Moderated
    Thanks for your suggestions. If there are others, this is still an open question.

    I ran the code through the eclipse Java Monitor, and the memory use is around 1.15GB for ~95MB worth of crl file data. Most of the memory is used by byte[]. These are memory stats using 1.6.0_33:
    -----
    <pre>
    Used heap memory          1,154,336 kbytes
    Max heap memory          1,520,448 kbytes
    Committed heap memory     1,520,448 kbytes
    Used non-heap memory     7,642 kbytes
    Max non-heap memory          98,304 kbytes
    Committed non-heap memory     13,792 kbytes
    </pre>
    -----
    This is the class memory break down:

    <pre>Class                    Size(bytes)          Count
    byte[]                    344300824          4096841
    java.util.TreeMap$Entry          131041344          4095042
    java.math.BigInteger               130933024          4091657
    sun.security.x509.X509CRLEntryImpl     130929344          4091542
    java.util.Date               98199240          4091635
    java.util.LinkedList$Entry          98198208          4091592
    sun.security.x509.X509CRLImpl$X509IssuerSerial     98197008     4091542
    int[]                    73912720          4095304
    sun.security.x509.SerialNumber          65464672          4091542
    char[]                    488176          8576
    java.lang.String               213552          8898
    java.util.TreeMap               164448          3426
    java.lang.Class               135552          1412
    short[]                    111432          1849
    java.util.Collections$SynchronizedMap     107712          3366
    java.lang.reflect.Method          101280          1266
    int[][]                    95104          1997
    sun.security.x509.CRLExtensions          53760          3360
    sun.security.util.ObjectIdentifier          52656          2194
    sun.security.x509.Extension          48144          2006
    java.util.HashMap.Entry[]          40632          356
    java.lang.reflect.Field               39600          550
    sun.security.x509.CRLReasonCodeExtension     31488          1312
    java.lang.Object[]               29008          709
    java.util.LinkedHashMap$Entry          26848          839
    java.lang.Class[]               24664          1357
    java.util.HashMap$Entry          17424          726
    java.lang.reflect.Constructor          14208          222
    java.lang.String[]               13784          459
    java.util.HashMap               13120          328
    java.lang.ref.SoftReference          12224          382
    java.lang.reflect.Method[]          7664          144
    java.util.concurrent.ConcurrentHashMap$Segment     6144     192
    java.util.Hashtable.Entry[]          6144          84
    sun.security.util.DerInputBuffer          5544          231
    sun.security.util.DerValue          5544          231
    sun.security.x509.RDN               5544          231
    java.util.Hashtable$Entry          5520          230
    java.lang.Long               5168          323
    java.util.concurrent.locks.ReentrantLock$NonfairSync     4824     201
    java.util.WeakHashMap$Entry          4480          112
    javax.management.ImmutableDescriptor     4416          184
    long[]                    4056          55
    java.io.ObjectStreamClass          4032          42
    java.io.ObjectStreamField          3904          122
    java.util.concurrent.ConcurrentHashMap.HashEntry[]     3816     192
    sun.security.x509.AVA               3696          231
    sun.security.x509.AVA[]          3696          231
    sun.security.util.DerInputStream          3696          231
    sun.security.x509.X509CRLImpl          3680          46</pre>
  • 6. Re: Is CertificateFactory.generateCRL horribly inefficient?
    sabre150 Expert
    Currently Being Moderated
    952537 wrote:
    I ran the code through the eclipse Java Monitor, and the memory use is around 1.15GB for ~95MB worth of crl file data.
    Are you basing this 1.15 GB on
    Used heap memory          1,154,336 kbytes
    If so then I'm not sure this is valid. I'm not an Eclipse person but I thought this was the memory used while processing and not that used after the data is loaded. If I am right the the GC will have recovered some/much of that 1.15 GB.
  • 7. Re: Is CertificateFactory.generateCRL horribly inefficient?
    955540 Newbie
    Currently Being Moderated
    Yes, that's what I'm basing it on. There's an option of running gc in the tool's UI, but using it doesn't change the results, or the class/memory mappings. Neither does making the attempt in the code( System.gc() ).

    If you have any suggestions for getting valid memory use for the code, let me know.
  • 8. Re: Is CertificateFactory.generateCRL horribly inefficient?
    sabre150 Expert
    Currently Being Moderated
    MNI92 wrote:
    Yes, that's what I'm basing it on. There's an option of running gc in the tool's UI, but using it doesn't change the results, or the class/memory mappings. Neither does making the attempt in the code( System.gc() ).
    OK.

    >
    If you have any suggestions for getting valid memory use for the code, let me know.
    Sounds like the 1.15 GB is valid but I suspect that even if you can halve that you are not going to be happy so you need to look at the alternatives. Other than the DB approach I have already mentioned I have no suggestions.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points