6 Replies Latest reply on Jul 15, 2011 1:17 AM by henri_gomez

    Is-it normal to face situation where the "Reference Handler" is blocked?

    623766
      Hi all,

      I would like to know if it can be a normal behavior to have the "Reference Handler" thread from the class java.lang.ref.Reference in BLOCKED state in my thread dump (generated from visualvm) as you can see below.
      "http-0.0.0.0-8080-56" - Thread t@578
         java.lang.Thread.State: RUNNABLE
      at org.apache.lucene.index.TermBuffer.toTerm(TermBuffer.java:122)
      at org.apache.lucene.index.SegmentTermEnum.term(SegmentTermEnum.java:167)
      at org.apache.lucene.index.SegmentMergeInfo.next(SegmentMergeInfo.java:66)
      at org.apache.lucene.index.MultiSegmentReader$MultiTermEnum.next(MultiSegmentReader.java:494)
      at org.apache.lucene.index.SegmentMergeInfo.next(SegmentMergeInfo.java:65)
      at org.apache.lucene.index.MultiSegmentReader$MultiTermEnum.next(MultiSegmentReader.java:494)
      at org.apache.lucene.search.FilteredTermEnum.next(FilteredTermEnum.java:67)
      at org.apache.lucene.search.FilteredTermEnum.setEnum(FilteredTermEnum.java:49)
      at org.apache.lucene.search.FuzzyTermEnum.<init>(FuzzyTermEnum.java:127)
      at org.apache.lucene.search.FuzzyQuery.getEnum(FuzzyQuery.java:100)
      at org.apache.lucene.search.FuzzyQuery.rewrite(FuzzyQuery.java:104)
      at org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:383)
      at org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:368)
      ..
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.exoplatform.services.security.web.SetCurrentIdentityFilter.doFilter(SetCurrentIdentityFilter.java:90)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.exoplatform.portal.application.localization.LocalizationFilter.doFilter(LocalizationFilter.java:179)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.exoplatform.web.login.ClusteredSSOFilter.doFilter(ClusteredSSOFilter.java:62)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.exoplatform.container.web.PortalContainerFilter.doFilter(PortalContainerFilter.java:69)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
      at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:183)
      at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525)
      at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:95)
      at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)
      at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
      at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
      at org.apache.catalina.authenticator.SingleSignOn.invoke(SingleSignOn.java:402)
      at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:567)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)
      at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:829)
      at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
      at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:451)
      at java.lang.Thread.run(Thread.java:662)
         JNI locked monitors:
      - locked <5258ed3e> (a java.lang.ref.Reference$Lock)
      
         Locked ownable synchronizers:
      - None
      ...
      "Reference Handler" - Thread t@2
         java.lang.Thread.State: BLOCKED
      at java.lang.Object.wait(Native Method)
      - waiting on <5258ed3e> (a java.lang.ref.Reference$Lock)
      at java.lang.Object.wait(Object.java:485)
      at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
      
         Locked ownable synchronizers:
      - None
      As you can see in my thread dump the object "java.lang.ref.Reference$Lock" is locked by a common thread which has for consequences to annoy a lot the GC and to freeze the whole application. Regarding the code (http://kickjava.com/src/java/lang/ref/Reference.java.htm) since the class java.lang.ref.Reference$Lock is private and the field "lock" is also (static) private, I was wondering if it is normal that this field can be locked by a common thread and if so can you tell me why and how (by reflection I assume like my example below)? Can it be a JVM Bug?

      This issue makes me realized that any Java application without a Security Manager installed can be easily frozen thanks to this simple code:
            // The thread that will steal the lock to the "Reference Handler" to block the GC and then the whole application
            new Thread("Reference$Lock Holder Thread")
            {
               public void run()
               {
                  try
                  {
                     // Get the lock field by reflection
                     Field fieldLock = Reference.class.getDeclaredField("lock");
                     // Make it accessible
                     fieldLock.setAccessible(true);
                     // Get the static lock object
                     Object o = fieldLock.get(null);
                     Object mutex = new Object();
                     // Lock the lock object
                     synchronized (o)
                     {
                        // Wait forever
                        synchronized (mutex)
                        {
                           mutex.wait();
                        }
                     }
                  }
                  catch (Exception e)
                  {
                     e.printStackTrace();
                  }
               }
            }.start();
            // This is a simple code used to simulate normal object creation until the GC call
            for (int i = 1;;i++)
            {
               new Object();
               if ((i % 1000) == 0)
               {
                  System.out.println(i);
               }
            }
      NB: This issue has been occured on RHEL 5.3 et JVM Sun 1.6 u26 (64 bit)

      Thanks for helping,
      BR,
      Nicolas