This discussion is archived
4 Replies Latest reply: Dec 29, 2010 2:56 AM by dhaval.parikh RSS

Threads blocked - Apparently in toplink

681739 Newbie
681739 Jan 26, 2009 1:59 PM
Currently Being Moderated
Hi,

We are using glassfish in a recent release project to production, with the increase in the load and usage, the glassfish started to demonstrate an abnormal behaviour.

It locks in transactions. From our analysis is not always related with special loads, for example it can be without problem for one week and them it block’s twice in one day. The only solution is to do a restart and lost whatever was executing…

When these locks occur most of the time there isn’t any Database lock.

The impact now is huge, and without solution at sight…

We are using EJB’s 3.0 with JPA and toplink. And Glassfish with versions:
- Sun Java System Application Server 9.1_02 (build b04-fcs) and Sun Java System Application Server 9.1 (build b58g-fcs)

We have tried almost every thing, simplify some of our code, reduce transactions, install different Glassfish versions and some other tips out the in these posts, without success.

From two thread dumps we notice the following pattern:


at java.lang.Thread.sleep(Native Method)
at oracle.toplink.essentials.internal.helper.ConcurrencyManager.releaseDeferredLock(ConcurrencyManager.java:429)
at oracle.toplink.essentials.internal.identitymaps.CacheKey.releaseDeferredLock(CacheKey.java:373)
at oracle.toplink.essentials.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:614)
at oracle.toplink.essentials.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneNormally(ObjectBuilder.java:451)
at oracle.toplink.essentials.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:421)
at oracle.toplink.essentials.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:387)
at oracle.toplink.essentials.queryframework.ReportQueryResult.processItem(ReportQueryResult.java:220)
at oracle.toplink.essentials.queryframework.ReportQueryResult.buildResult(ReportQueryResult.java:182)
at oracle.toplink.essentials.queryframework.ReportQueryResult.(ReportQueryResult.java:98)
at oracle.toplink.essentials.queryframework.ReportQuery.buildObject(ReportQuery.java:594)
at oracle.toplink.essentials.queryframework.ReportQuery.buildObjects(ReportQuery.java:643)
at oracle.toplink.essentials.queryframework.ReportQuery.executeDatabaseQuery(ReportQuery.java:804)
at oracle.toplink.essentials.queryframework.DatabaseQuery.execute(DatabaseQuery.java:628)
at oracle.toplink.essentials.queryframework.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:692)
at oracle.toplink.essentials.queryframework.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:746)
at oracle.toplink.essentials.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2233)
at oracle.toplink.essentials.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:952)
at oracle.toplink.essentials.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:924)
at oracle.toplink.essentials.internal.ejb.cmp3.base.EJBQueryImpl.executeReadQuery(EJBQueryImpl.java:367)
at oracle.toplink.essentials.internal.ejb.cmp3.base.EJBQueryImpl.getSingleResult(EJBQueryImpl.java:508)
at com.sun.enterprise.util.QueryWrapper.getSingleResult(QueryWrapper.java:212)



Can anybody help on this ?
  • 1. Re: Threads blocked - Apparently in toplink
    JamesSutherland Pro
    JamesSutherland Jan 27, 2009 6:05 AM (in response to 681739)
    Currently Being Moderated
    This is very odd.

    The stack has a deferred lock on it, which means two threads tried to build the same set of objects from the database at the same time, and one is waiting for the other to be complete, it should not be a deadlock as the other should complete. Please include the full thread stack dump from all blocked threads, its takes at least two threads to have a deadlock. Are the threads deadlocking, or just taking a long time?

    If the issue is with deferred locks you can avoid these by ensuring you use lazy relationships. Make sure all of your relationships are marked as lazy. Usage of join fetching can also require deferred locks, so check if you are using join fetching, perhaps switch to batch reading.

    There is also a back door to disable deferred locks that can be used to determine the cause.
    DeferredLockManager.SHOULD_USE_DEFERRED_LOCKS = false;

    If the deadlock is occurring in the cache, as with deferred locks, then a possible solution is to disable caching, using the persistence.xml property "toplink.cache.shared.default"="false".

    You can also configure the TopLink cache isolation using, DatabaseLogin.setCacheTransactionIsolation(), try CONCURRENT_READ_WRITE.

    I would also recommend trying to upgrade to EclipseLink or TopLink 11.

    ---
    James : http://www.eclipselink.org
  • 2. Re: Threads blocked - Apparently in toplink
    504035 Explorer
    504035 Jan 31, 2009 9:59 AM (in response to 681739)
    Currently Being Moderated
    Many related fixes will be present in next released version of EclipseLink, which is 1.1 (TopLink 11).
  • 3. Re: Threads blocked - Apparently in toplink
    newToplinkUser Newbie
    newToplinkUser Feb 10, 2009 3:47 PM (in response to 504035)
    Currently Being Moderated
    We ran into a similar situation in toplink 10g, we had a call back into toplink for one of our custom fields and in the custom field we were trying to do a toplink query befor a uow.commit, we removed the
    query from the callabck and everything worked, also we had the cache usage to readcacheonly which was very bad
  • 4. Re: Threads blocked - Apparently in toplink
    dhaval.parikh Newbie
    dhaval.parikh Dec 29, 2010 2:56 AM (in response to 681739)
    Currently Being Moderated
    This seems to be a bug in EclipseLink Bug 312462 - ConcurrencyManager.acquire() hangs forever

    https://bugs.eclipse.org/bugs/show_bug.cgi?id=312462

Actions

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points