This discussion is archived
13 Replies Latest reply: Dec 13, 2012 6:35 AM by 923493 RSS

Suppressing the generation of DelegatingClassLoader

923493 Newbie
Currently Being Moderated
We have question regarding DelegatingClassLoader which is apparently causing the OutOfMemoryError: PermGen space with our system.

While analyzing a heap dump using Yourkit we have observed many instances of DelegatingClassLoader created through reflection by frameworks used for our web application.
It is allegedly used for some sort of reflection optimization mechanism according to the url at http://www-01.ibm.com/support/docview.wss?uid=swg21566549.
Becuase this is what seems to be eating up our memory allocated for PermGen, as the number of the DelegatingClassLoader is gradually increased and JVM crushes with OutOfMemoryError,
we have tried setting sun.reflect.inflationThreshold = 0 as suggested thereby. However it does not seem to be working as we are still seeing DelegatingClassLoaders getting generated.
We also scrutinized the following source code corresponds to where this property is set and its value is retrieved.

http://javasourcecode.org/html/open-source/jdk/jdk-6u23/sun/reflect/ReflectionFactory.java.html

and

http://javasourcecode.org/html/open-source/jdk/jdk-6u23/sun/reflect/NativeMethodAccessorImpl.java.html

Looks to me that specifying 0 will not have any effect on suppressing the generation of the DelegatingClassLoaders. The value apparently needs to something very big and it has to be

something like Integer.MAX_VALUE in order to turn that feature off forever.

My question is, why there are a lot of mentioning about setting this property value to 0 and whether setting a large number such as Integer.MAX_VALUE is safe, if that is what needs to be done.

Thanks in advance.
  • 1. Re: Suppressing the generation of DelegatingClassLoader
    Kayaman Guru
    Currently Being Moderated
    920490 wrote:
    Looks to me that specifying 0 will not have any effect on suppressing the generation of the DelegatingClassLoaders. The value apparently needs to something very big and it has to be

    something like Integer.MAX_VALUE in order to turn that feature off forever.

    My question is, why there are a lot of mentioning about setting this property value to 0 and whether setting a large number such as Integer.MAX_VALUE is safe, if that is what needs to be done.
    Either a mistake in documentation, or it has been changed and documentation hasn't been updated.

    If you look at the ReflectionFactory source you posted, there's a "sun.reflect.noInflation" which when set to true will do what you want.

    --
    Also, I wasn't aware of this optimization mechanism. It's still nice to learn new things on the forum, even after all these years.
  • 2. Re: Suppressing the generation of DelegatingClassLoader
    923493 Newbie
    Currently Being Moderated
    Hi, Kayaman,

    Thanks for your reply. By setting "noInflation" to true, I believe it will always use the Bytecode accessor regardless of the number of reflective method calls made(or count)

    With so many Java frameworks reflectively invoking methods on dynamically generated instances, I wonder why this hasn't been a big topic of discussion as such,
    I may be wrong with my guess about PermGen occupied with the generated DelegatingClassLoaders.

    Anyway thanks for your attention, I really appreciated it.
  • 3. Re: Suppressing the generation of DelegatingClassLoader
    Kayaman Guru
    Currently Being Moderated
    920490 wrote:
    Thanks for your reply. By setting "noInflation" to true, I believe it will always use the Bytecode accessor regardless of the number of reflective method calls made(or count)
    Indeed, so you want to keep using the native accessor? You're right, the threshold would need to be at Integer.MAX_VALUE. Of course that would just be postponing the inevitable.
    With so many Java frameworks reflectively invoking methods on dynamically generated instances, I wonder why this hasn't been a big topic of discussion as such,
    I may be wrong with my guess about PermGen occupied with the generated DelegatingClassLoaders.
    Well, it does come up every now and then. What size is your PermGen at the moment?
  • 4. Re: Suppressing the generation of DelegatingClassLoader
    923493 Newbie
    Currently Being Moderated
    Kayaman wrote:
    Well, it does come up every now and then. What size is your PermGen at the moment?
    Currently it is 512MB. The number of DelegatingClassLoader resides in the heap is around 1647 at the beginning and it gradually increases up to 7210.
    Application is running in JBoss environment and the dominator of the heap memory space is spring, hibernate, and JAX-WS.
  • 5. Re: Suppressing the generation of DelegatingClassLoader
    Kayaman Guru
    Currently Being Moderated
    920490 wrote:
    Kayaman wrote:
    Well, it does come up every now and then. What size is your PermGen at the moment?
    Currently it is 512MB.
    Well, that's quite significant, the application must be very large?
    The number of DelegatingClassLoader resides in the heap is around 1647 at the beginning and it gradually increases up to 7210.
    Application is running in JBoss environment and the dominator of the heap memory space is spring, hibernate, and JAX-WS.
    Well, I can imagine Spring using plenty of reflection, but still it sounds a bit too excessive to run out of permgen just with these.
  • 6. Re: Suppressing the generation of DelegatingClassLoader
    923493 Newbie
    Currently Being Moderated
    >
    Well, that's quite significant, the application must be very large?
    >
    It is presumably very large. I say presumably, because it is a project we recently took over and not something we know thoroughly.
    >
    Well, I can imagine Spring using plenty of reflection, but still it sounds a bit too excessive to run out of permgen just with these.
    >
    I wonder if DelegationClassLoader is ever garbage collected...
  • 7. Re: Suppressing the generation of DelegatingClassLoader
    Kayaman Guru
    Currently Being Moderated
    920490 wrote:
    It is presumably very large. I say presumably, because it is a project we recently took over and not something we know thoroughly.
    The default value was at least at one point 64MB, so running out of half a gig of permgen would suggest that there's a design flaw causing this.
    I wonder if DelegationClassLoader is ever garbage collected...
    In what situation are you running out of permgen space? Are you hot redeploying the app and that causes it to run out, or is it long running and after N days of usage it crashes?
  • 8. Re: Suppressing the generation of DelegatingClassLoader
    923493 Newbie
    Currently Being Moderated
    >
    In what situation are you running out of permgen space? Are you hot redeploying the app and that causes it to run out, or is it long running and after N days of usage it crashes?
    >

    We are not hot redeploying the app. We always shutdown and restart the J2EE container. The allotted permgen space is filled up within 5 to 6 hours, after people start using the web app.
    It is definitely design flaw causing this, but there is little we can do right now.

    Nonetheless, all what we know now is that those DelegationClassLoaders, each with 1 or 2 instantiated classes, are gradually increasing in number and they are not created explicitly.

    Edited by: 920490 on 2012/12/11 5:58
  • 9. Re: Suppressing the generation of DelegatingClassLoader
    923493 Newbie
    Currently Being Moderated
    >
    "Any reference from outside the application to an object in the application of which the class is loaded by the application's classloader will cause a classloader leak"
    ---
    http://frankkieviet.blogspot.jp/2006/10/classloader-leaks-dreaded-permgen-space.html
    >

    and I think it is what is happening with the DelegatingClassLoaders. It is going to be very hard to spot the ClassLoader leak though.
  • 10. Re: Suppressing the generation of DelegatingClassLoader
    Kayaman Guru
    Currently Being Moderated
    920490 wrote:
    and I think it is what is happening with the DelegatingClassLoaders. It is going to be very hard to spot the ClassLoader leak though.
    But the classloader leak only applies to when undeploying the application, when parts of the previous deployment can't be unloaded.
    Since you aren't redeploying, it would mean that your permgen size just isn't big enough for normal operation. Have you tried upping the permgen space some more, just to test whether it will eventually run out with 768M or even 1G?

    If you have tests made for the software (yea yea, but just in case you did), you could churn out some load testing to see whether it'll happen or not.
  • 11. Re: Suppressing the generation of DelegatingClassLoader
    923493 Newbie
    Currently Being Moderated
    >
    But the classloader leak only applies to when undeploying the application, when parts of the previous deployment can't be unloaded.
    Since you aren't redeploying, it would mean that your permgen size just isn't big enough for normal operation. Have you tried upping the permgen space some more, just to test whether it will eventually run out with 768M or even 1G?

    If you have tests made for the software (yea yea, but just in case you did), you could churn out some load testing to see whether it'll happen or not.
    Thanks for your response and sorry for this belated notice.
    I thinks what's using up our PermGen may be the ognl.OgnlRuntime, as depicted in "path to the cloass loader" hierarchical view provided by YourKit heap dump snapshot, which is the following:
    sun.reflect.DelegatingClassLoader 
    |_<loader> of sun.reflect.GeneratedMethodAccessor14121 
       |_<class> of sun.reflect.GeneratedMethodAccessor14121 
          |_delegate of sun.reflect.DelegationMethodAccessorImpl 
             |_methodAccessor of java.lang.reflect.Method 
                |_key of java.util.HashMap$Entry 
                   |_[49269] of java.util.HashMap$Entry[16384] 
                      |_table of java.util.HashMap 
                         |_ _methodParameterTypesCache of ognl.OgnlRuntime 
                            |_[7690] of java.util.Object[20480] 
                              |_ elementData of java.util.Vector 
                                 |_classes of org.jboss.classloader.spi.base.BaseClassLoader[Stack Local] 
                                   |_<local variable> of java.lang.Thread[Stack local, Thread] "ajp-xxx.xxx.xxx.xxx-8009-3" 
    We see thousands of these, each with different GeneratedMethodAccessorXXXX, where XXXX represents some sequential number as the one in the hierarchy above. The element at the very bottom is Thread which is a GC root and it is not garbage collectable. It references a BaseClassLoader which holds the ognl.OgnlRuntime indirectly in its Vector. The ognl.OgnlRuntime retains a reference to java.lang.reflect.Method who has reference to sun.reflect.GeneratedMethodAccessor14121 that is loaded by sun.reflect.DelegatingClassLoader. Could this be a ClassLoader Leak?

    I also created the following issue with Apache. Unfortunately, I have not got any helpful answer yet though.
    https://issues.apache.org/jira/browse/OGNL-228

    What is your opinion? May be that system really needs more memory and 512 is not enough at all.

    By the way, I really appreciate you have been giving me advices. Thanks!
  • 12. Re: Suppressing the generation of DelegatingClassLoader
    Kayaman Guru
    Currently Being Moderated
    920490 wrote:
    Thanks for your response and sorry for this belated notice.
    No worries, I'm always here.
    I thinks what's using up our PermGen may be the ognl.OgnlRuntime
    Sounds like a very likely culprit.

    I noticed the methodParameterTypesCache there, and googled to see whether there was any info about it, like whether it's possible to flush it.
    Found this http://grokbase.com/t/struts/user/0948kcdd77/struts-and-permgen-errors-with-ognl-method-parameters-cache so you're not alone with this problem, and the cache seems to be involved.
    Also another googling trip provided with "CMSPermGenSweepingEnabled" and "CMSClassUnloadingEnabled" runtime parameters, you might want to look into those as well.
    What is your opinion? May be that system really needs more memory and 512 is not enough at all.
    You might want to test it out, if you can.
    By the way, I really appreciate you have been giving me advices. Thanks!
    It's been an interesting question. I'm not sure if we have a "resident" expert on these things on the forums, at least he hasn't participated (also he might not be following this particular forum).
  • 13. Re: Suppressing the generation of DelegatingClassLoader
    923493 Newbie
    Currently Being Moderated
    I thinks what's using up our PermGen may be the ognl.OgnlRuntime
    Sounds like a very likely culprit.

    I noticed the methodParameterTypesCache there, and googled to see whether there was any info about it, like whether it's possible to flush it.
    Found this http://grokbase.com/t/struts/user/0948kcdd77/struts-and-permgen-errors-with-ognl-method-parameters-cache so you're not alone with this problem, and the cache seems to be involved.
    Also another googling trip provided with "CMSPermGenSweepingEnabled" and "CMSClassUnloadingEnabled" runtime parameters, you might want to look into those as well.
    Thank you for this info. I think it is culprit of the memory leak. There are indeed other DelegatingClassLoaders also observed in the heap dump, as other frame works also uses reflective method calls. However, there is difference. I see no "other classloader" between the Thread(GC root) and the DelegatingClassLoader. For instance, DelegatingClassLoaders which is created upon reflective call made by an object returned by DefaultListableBeanFactory has the GC root of a Thread which probably terminates only when JBoss is shutdown.
    In the ognl.OgnlRuntime case, if its GC root, which probably is a request Thread, would have called a method reflectively in more "direct" manner, the DelegatingClassLoader holding sun.reflect.GeneratedMethodAccessorXXX could be expelled eventually. However, as this is done though ognl.OgnlRuntime who is loaded by org.jboss.classloader.spi.base.BaseClassLoader, which has a different GC root, sun.reflect.GeneratedMethodAccessorXXX remains, because the ognl.OgnlRuntime holds them in its HashMap.

    I will look into "CMSPermGenSweepingEnabled" and "CMSClassUnloadingEnabled". Hope those helps.

    Thanks

    Edited by: 920490 on 2012/12/13 6:34

    Edited by: 920490 on 2012/12/13 6:35

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points