6 Replies Latest reply: Jul 14, 2011 8:17 PM by 875683 RSS

    Is-it normal to face situation where the "Reference Handler" is blocked?

    623766
      Hi all,

      I would like to know if it can be a normal behavior to have the "Reference Handler" thread from the class java.lang.ref.Reference in BLOCKED state in my thread dump (generated from visualvm) as you can see below.
      "http-0.0.0.0-8080-56" - Thread t@578
         java.lang.Thread.State: RUNNABLE
      at org.apache.lucene.index.TermBuffer.toTerm(TermBuffer.java:122)
      at org.apache.lucene.index.SegmentTermEnum.term(SegmentTermEnum.java:167)
      at org.apache.lucene.index.SegmentMergeInfo.next(SegmentMergeInfo.java:66)
      at org.apache.lucene.index.MultiSegmentReader$MultiTermEnum.next(MultiSegmentReader.java:494)
      at org.apache.lucene.index.SegmentMergeInfo.next(SegmentMergeInfo.java:65)
      at org.apache.lucene.index.MultiSegmentReader$MultiTermEnum.next(MultiSegmentReader.java:494)
      at org.apache.lucene.search.FilteredTermEnum.next(FilteredTermEnum.java:67)
      at org.apache.lucene.search.FilteredTermEnum.setEnum(FilteredTermEnum.java:49)
      at org.apache.lucene.search.FuzzyTermEnum.<init>(FuzzyTermEnum.java:127)
      at org.apache.lucene.search.FuzzyQuery.getEnum(FuzzyQuery.java:100)
      at org.apache.lucene.search.FuzzyQuery.rewrite(FuzzyQuery.java:104)
      at org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:383)
      at org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:368)
      ..
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.exoplatform.services.security.web.SetCurrentIdentityFilter.doFilter(SetCurrentIdentityFilter.java:90)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.exoplatform.portal.application.localization.LocalizationFilter.doFilter(LocalizationFilter.java:179)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.exoplatform.web.login.ClusteredSSOFilter.doFilter(ClusteredSSOFilter.java:62)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.exoplatform.container.web.PortalContainerFilter.doFilter(PortalContainerFilter.java:69)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
      at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:183)
      at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525)
      at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:95)
      at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)
      at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
      at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
      at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:402)
      at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:567)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)
      at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:829)
      at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
      at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:451)
      at java.lang.Thread.run(Thread.java:662)
         JNI locked monitors:
      - locked <5258ed3e> (a java.lang.ref.Reference$Lock)
      
         Locked ownable synchronizers:
      - None
      ...
      "Reference Handler" - Thread t@2
         java.lang.Thread.State: BLOCKED
      at java.lang.Object.wait(Native Method)
      - waiting on <5258ed3e> (a java.lang.ref.Reference$Lock)
      at java.lang.Object.wait(Object.java:485)
      at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
      
         Locked ownable synchronizers:
      - None
      As you can see in my thread dump the object "java.lang.ref.Reference$Lock" is locked by a common thread which has for consequences to annoy a lot the GC and to freeze the whole application. Regarding the code (http://kickjava.com/src/java/lang/ref/Reference.java.htm) since the class java.lang.ref.Reference$Lock is private and the field "lock" is also (static) private, I was wondering if it is normal that this field can be locked by a common thread and if so can you tell me why and how (by reflection I assume like my example below)? Can it be a JVM Bug?

      This issue makes me realized that any Java application without a Security Manager installed can be easily frozen thanks to this simple code:
            // The thread that will steal the lock to the "Reference Handler" to block the GC and then the whole application
            new Thread("Reference$Lock Holder Thread")
            {
               public void run()
               {
                  try
                  {
                     // Get the lock field by reflection
                     Field fieldLock = Reference.class.getDeclaredField("lock");
                     // Make it accessible
                     fieldLock.setAccessible(true);
                     // Get the static lock object
                     Object o = fieldLock.get(null);
                     Object mutex = new Object();
                     // Lock the lock object
                     synchronized (o)
                     {
                        // Wait forever
                        synchronized (mutex)
                        {
                           mutex.wait();
                        }
                     }
                  }
                  catch (Exception e)
                  {
                     e.printStackTrace();
                  }
               }
            }.start();
            // This is a simple code used to simulate normal object creation until the GC call
            for (int i = 1;;i++)
            {
               new Object();
               if ((i % 1000) == 0)
               {
                  System.out.println(i);
               }
            }
      NB: This issue has been occured on RHEL 5.3 et JVM Sun 1.6 u26 (64 bit)

      Thanks for helping,
      BR,
      Nicolas
        • 1. Re: Is-it normal to face situation where the "Reference Handler" is blocked?
          jtahlborn
          yes, that is very normal. as you can see from the javadoc on that field, the "Reference Handler" could be BLOCKED on that lock anytime that the garbage collector is running.

          and yes, without a SecurityManager, there are any of a thousand ways that your jvm can be locked up. even with a SecurityManager you can't, for example, limit the amount of cpu that a thread can consume, so you can potentially severely degrade the jvm performance even with a simple, tight, infinite loop.

          Edited by: jtahlborn on Jul 12, 2011 4:42 PM
          • 2. Re: Is-it normal to face situation where the "Reference Handler" is blocked?
            623766
            Hi jtahlborn ,

            Thank you for your answer,

            I did not understand that from the javadoc by "javadoc on that field" you mean this :
            /* Object used to synchronize with the garbage collector. The collector
            * must acquire this lock at the beginning of each collection cycle. It is
            * therefore critical that any code holding this lock complete as quickly
            * as possible, allocate no new objects, and avoid calling user code.
            */
            ?

            So for you it is normal that the normal thread "http-0.0.0.0-8080-56" holds the lock expected by the "Reference Handler" and 20 other normal threads? If so do you have any idea why it works like this? Is-it how the GC works internally?

            Thank you in advance,
            BR,
            Nicolas
            • 3. Re: Is-it normal to face situation where the "Reference Handler" is blocked?
              jtahlborn
              essobedo wrote:
              I did not understand that from the javadoc by "javadoc on that field" you mean this :
              /* Object used to synchronize with the garbage collector. The collector
              * must acquire this lock at the beginning of each collection cycle. It is
              * therefore critical that any code holding this lock complete as quickly
              * as possible, allocate no new objects, and avoid calling user code.
              */
              ?

              So for you it is normal that the normal thread "http-0.0.0.0-8080-56" holds the lock expected by the "Reference Handler" and 20 other normal threads? If so do you have any idea why it works like this? Is-it how the GC works internally?
              if those threads are doing garbage collection work, then yes, they will be holding that lock (which is exactly what the javadoc says).
              • 4. Re: Is-it normal to face situation where the "Reference Handler" is blocked?
                623766
                As I already said this thread is a normal thread that has nothing to do with the garbage collection work (unless someone can prove me the opposite) if you check the stack trace in the thread dump you will see that it is here a simple Lucene query nothing more and that the thread is a tomcat thread coming from its thread pool otherwise believe me, I would not post any question in the forum if it was that easy.
                • 5. Re: Is-it normal to face situation where the "Reference Handler" is blocked?
                  jtahlborn
                  are you using non-standard GC settings?
                  • 6. Re: Is-it normal to face situation where the "Reference Handler" is blocked?
                    875683
                    Here is part of the settings we used in such case :

                    -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled