4 Replies Latest reply: Dec 29, 2010 4:56 AM by Dhaval Parikh-Oracle RSS

    Threads blocked - Apparently in toplink

    681739
    681739 Jan 26, 2009 3:59 PM
      Hi,

      We are using glassfish in a recent release project to production, with the increase in the load and usage, the glassfish started to demonstrate an abnormal behaviour.

      It locks in transactions. From our analysis is not always related with special loads, for example it can be without problem for one week and them it block’s twice in one day. The only solution is to do a restart and lost whatever was executing…

      When these locks occur most of the time there isn’t any Database lock.

      The impact now is huge, and without solution at sight…

      We are using EJB’s 3.0 with JPA and toplink. And Glassfish with versions:
      - Sun Java System Application Server 9.1_02 (build b04-fcs) and Sun Java System Application Server 9.1 (build b58g-fcs)

      We have tried almost every thing, simplify some of our code, reduce transactions, install different Glassfish versions and some other tips out the in these posts, without success.

      From two thread dumps we notice the following pattern:


      at java.lang.Thread.sleep(Native Method)
      at oracle.toplink.essentials.internal.helper.ConcurrencyManager.releaseDeferredLock(ConcurrencyManager.java:429)
      at oracle.toplink.essentials.internal.identitymaps.CacheKey.releaseDeferredLock(CacheKey.java:373)
      at oracle.toplink.essentials.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:614)
      at oracle.toplink.essentials.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneNormally(ObjectBuilder.java:451)
      at oracle.toplink.essentials.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:421)
      at oracle.toplink.essentials.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:387)
      at oracle.toplink.essentials.queryframework.ReportQueryResult.processItem(ReportQueryResult.java:220)
      at oracle.toplink.essentials.queryframework.ReportQueryResult.buildResult(ReportQueryResult.java:182)
      at oracle.toplink.essentials.queryframework.ReportQueryResult.(ReportQueryResult.java:98)
      at oracle.toplink.essentials.queryframework.ReportQuery.buildObject(ReportQuery.java:594)
      at oracle.toplink.essentials.queryframework.ReportQuery.buildObjects(ReportQuery.java:643)
      at oracle.toplink.essentials.queryframework.ReportQuery.executeDatabaseQuery(ReportQuery.java:804)
      at oracle.toplink.essentials.queryframework.DatabaseQuery.execute(DatabaseQuery.java:628)
      at oracle.toplink.essentials.queryframework.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:692)
      at oracle.toplink.essentials.queryframework.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:746)
      at oracle.toplink.essentials.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2233)
      at oracle.toplink.essentials.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:952)
      at oracle.toplink.essentials.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:924)
      at oracle.toplink.essentials.internal.ejb.cmp3.base.EJBQueryImpl.executeReadQuery(EJBQueryImpl.java:367)
      at oracle.toplink.essentials.internal.ejb.cmp3.base.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:508)
      at com.sun.enterprise.util.QueryWrapper.getSingleResult(QueryWrapper.java:212)



      Can anybody help on this ?
        • 1. Re: Threads blocked - Apparently in toplink
          JamesSutherland
          JamesSutherland Jan 27, 2009 8:05 AM (in response to 681739)
          This is very odd.

          The stack has a deferred lock on it, which means two threads tried to build the same set of objects from the database at the same time, and one is waiting for the other to be complete, it should not be a deadlock as the other should complete. Please include the full thread stack dump from all blocked threads, its takes at least two threads to have a deadlock. Are the threads deadlocking, or just taking a long time?

          If the issue is with deferred locks you can avoid these by ensuring you use lazy relationships. Make sure all of your relationships are marked as lazy. Usage of join fetching can also require deferred locks, so check if you are using join fetching, perhaps switch to batch reading.

          There is also a back door to disable deferred locks that can be used to determine the cause.
          DeferredLockManager.SHOULD_USE_DEFERRED_LOCKS = false;

          If the deadlock is occurring in the cache, as with deferred locks, then a possible solution is to disable caching, using the persistence.xml property "toplink.cache.shared.default"="false".

          You can also configure the TopLink cache isolation using, DatabaseLogin.setCacheTransactionIsolation(), try CONCURRENT_READ_WRITE.

          I would also recommend trying to upgrade to EclipseLink or TopLink 11.

          ---
          James : http://www.eclipselink.org
          • 2. Re: Threads blocked - Apparently in toplink
            504035
            504035 Jan 31, 2009 11:59 AM (in response to 681739)
            Many related fixes will be present in next released version of EclipseLink, which is 1.1 (TopLink 11).
            • 3. Re: Threads blocked - Apparently in toplink
              newToplinkUser
              newToplinkUser Feb 10, 2009 5:47 PM (in response to 504035)
              We ran into a similar situation in toplink 10g, we had a call back into toplink for one of our custom fields and in the custom field we were trying to do a toplink query befor a uow.commit, we removed the
              query from the callabck and everything worked, also we had the cache usage to readcacheonly which was very bad
              • 4. Re: Threads blocked - Apparently in toplink
                Dhaval Parikh-Oracle
                Dhaval Parikh-Oracle Dec 29, 2010 4:56 AM (in response to 681739)
                This seems to be a bug in EclipseLink Bug 312462 - ConcurrencyManager.acquire() hangs forever

                https://bugs.eclipse.org/bugs/show_bug.cgi?id=312462