This discussion is archived
1 2 Previous Next 18 Replies Latest reply: Jan 3, 2008 9:09 AM by cdelahun RSS

Hung Threads (Toplink 10.1.3)

zb Newbie
Currently Being Moderated
We are experiencing hung threads (waiting in TopLink code) in our app server, each one consuming a db connection so we eventually run out of db connections and die.

We've got 9 threads waiting over 10 minutes on ConcurrencyManager.releaseDeferredLock(ConcurrencyManager.java:389)
And 1 thread waiting over 10 minutes on WriteLockManager.acquireLocksForClone(WriteLockManager.java:66)

Thread dump Java HotSpot(TM) Client VM (1.4.2_05-b04 mixed mode):

"Servlet.Engine.Transports : 12" daemon prio=5 tid=0x02976988 nid=0x5a waiting on condition [5e57c000..5e57fc30]
at java.lang.Thread.sleep(Native Method)
at oracle.toplink.internal.helper.ConcurrencyManager.releaseDeferredLock(ConcurrencyManager.java:389)
at oracle.toplink.internal.identitymaps.CacheKey.releaseDeferredLock(CacheKey.java:311)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:619)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:378)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObjectsInto(ObjectBuilder.java:667)
at oracle.toplink.internal.queryframework.DatabaseQueryMechanism.buildObjectsFromRows(DatabaseQueryMechanism.java:141)
at oracle.toplink.queryframework.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:468)
at oracle.toplink.queryframework.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:800)
at oracle.toplink.queryframework.DatabaseQuery.execute(DatabaseQuery.java:603)
at oracle.toplink.queryframework.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:768)
at oracle.toplink.queryframework.ReadAllQuery.execute(ReadAllQuery.java:436)
at oracle.toplink.publicinterface.Session.internalExecuteQuery(Session.java:2062)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:981)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:938)
at com.saic.ct.sys.persistence.common.BaseDAO.executeQueryUsingSession(BaseDAO.java:1249)
at com.saic.ct.sys.persistence.common.BaseDAO.findMany(BaseDAO.java:1432)
at com.saic.ct.sys.persistence.common.BaseDAO.findMany(BaseDAO.java:1407)
at com.saic.ct.sys.persistence.common.CommonDAO.findClassifiers(CommonDAO.java:1609)
...

"Servlet.Engine.Transports : 11" daemon prio=5 tid=0x028181a0 nid=0x59 in Object.wait() [5e67c000..5e67fc30]
at java.lang.Object.wait(Native Method)
- waiting on <0x9e5bc300> (a oracle.toplink.internal.helper.ConcurrencyManager)
at java.lang.Object.wait(Object.java:429)
at oracle.toplink.internal.helper.WriteLockManager.acquireLocksForClone(WriteLockManager.java:66)
- locked <0x9e5bc300> (a oracle.toplink.internal.helper.ConcurrencyManager)
at oracle.toplink.publicinterface.UnitOfWork.cloneAndRegisterObject(UnitOfWork.java:715)
at oracle.toplink.internal.sessions.UnitOfWorkIdentityMapAccessor.getAndCloneCacheKeyFromParent(UnitOfWorkIdentityMapAccessor.java:154)
at oracle.toplink.internal.sessions.UnitOfWorkIdentityMapAccessor.getFromIdentityMap(UnitOfWorkIdentityMapAccessor.java:99)
at oracle.toplink.internal.sessions.IdentityMapAccessor.getFromIdentityMap(IdentityMapAccessor.java:257)
at oracle.toplink.publicinterface.UnitOfWork.registerExistingObject(UnitOfWork.java:3465)
- locked <0x9e97ac68> (a oracle.toplink.internal.sessions.UnitOfWorkImpl)
at oracle.toplink.publicinterface.UnitOfWork.registerExistingObject(UnitOfWork.java:3427)
- locked <0x9e97ac68> (a oracle.toplink.internal.sessions.UnitOfWorkImpl)
at oracle.toplink.queryframework.ObjectLevelReadQuery.registerIndividualResult(ObjectLevelReadQuery.java:1717)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneNormally(ObjectBuilder.java:449)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:413)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:376)
at oracle.toplink.queryframework.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:451)
at oracle.toplink.queryframework.ObjectLevelReadQuery.registerIndividualResult(ObjectLevelReadQuery.java:1701)
at oracle.toplink.queryframework.ReadAllQuery.registerResultInUnitOfWork(ReadAllQuery.java:686)
at oracle.toplink.queryframework.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:466)
at oracle.toplink.queryframework.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:800)
at oracle.toplink.queryframework.DatabaseQuery.execute(DatabaseQuery.java:603)
at oracle.toplink.queryframework.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:768)
at oracle.toplink.queryframework.ReadAllQuery.execute(ReadAllQuery.java:436)
at oracle.toplink.queryframework.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:825)
at oracle.toplink.publicinterface.UnitOfWork.internalExecuteQuery(UnitOfWork.java:2532)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:981)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:938)
at com.saic.ct.sys.persistence.common.BaseDAO.executeQueryUsingUowOld(BaseDAO.java:256)
at com.saic.ct.sys.persistence.common.BaseDAO.findVOsForUpdate(BaseDAO.java:485)
at com.saic.ct.sys.persistence.common.BaseDAO.findVOByPK(BaseDAO.java:291)
at com.saic.ct.sys.persistence.schedule.ScheduleDAO.findWorkGroupByPK(ScheduleDAO.java:1248)
...

"Servlet.Engine.Transports : 10" daemon prio=5 tid=0x029b0168 nid=0x58 waiting on condition [5e77c000..5e77fc30]
at java.lang.Thread.sleep(Native Method)
at oracle.toplink.internal.helper.ConcurrencyManager.releaseDeferredLock(ConcurrencyManager.java:389)
at oracle.toplink.internal.identitymaps.CacheKey.releaseDeferredLock(CacheKey.java:311)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:619)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:378)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObjectsInto(ObjectBuilder.java:667)
at oracle.toplink.internal.queryframework.DatabaseQueryMechanism.buildObjectsFromRows(DatabaseQueryMechanism.java:141)
at oracle.toplink.queryframework.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:468)
at oracle.toplink.queryframework.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:800)
at oracle.toplink.queryframework.DatabaseQuery.execute(DatabaseQuery.java:603)
at oracle.toplink.queryframework.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:768)
at oracle.toplink.queryframework.ReadAllQuery.execute(ReadAllQuery.java:436)
at oracle.toplink.publicinterface.Session.internalExecuteQuery(Session.java:2062)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:981)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:938)
at com.saic.ct.sys.persistence.common.BaseDAO.executeQueryUsingSession(BaseDAO.java:1249)
at com.saic.ct.sys.persistence.common.BaseDAO.findMany(BaseDAO.java:1432)
at com.saic.ct.sys.persistence.common.BaseDAO.findMany(BaseDAO.java:1407)
at com.saic.ct.sys.persistence.common.CommonDAO.findClassifiers(CommonDAO.java:1609)
...

"Servlet.Engine.Transports : 9" daemon prio=5 tid=0x02c880c8 nid=0x57 waiting on condition [5e87c000..5e87fc30]
at java.lang.Thread.sleep(Native Method)
at oracle.toplink.internal.helper.ConcurrencyManager.releaseDeferredLock(ConcurrencyManager.java:389)
at oracle.toplink.internal.identitymaps.CacheKey.releaseDeferredLock(CacheKey.java:311)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:619)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:378)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObjectsInto(ObjectBuilder.java:667)
at oracle.toplink.internal.queryframework.DatabaseQueryMechanism.buildObjectsFromRows(DatabaseQueryMechanism.java:141)
at oracle.toplink.queryframework.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:468)
at oracle.toplink.queryframework.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:800)
at oracle.toplink.queryframework.DatabaseQuery.execute(DatabaseQuery.java:603)
at oracle.toplink.queryframework.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:768)
at oracle.toplink.queryframework.ReadAllQuery.execute(ReadAllQuery.java:436)
at oracle.toplink.publicinterface.Session.internalExecuteQuery(Session.java:2062)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:981)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:938)
at com.saic.ct.sys.persistence.common.BaseDAO.executeQueryUsingSession(BaseDAO.java:1249)
at com.saic.ct.sys.persistence.common.BaseDAO.findMany(BaseDAO.java:1432)
at com.saic.ct.sys.persistence.common.BaseDAO.findMany(BaseDAO.java:1407)
at com.saic.ct.sys.persistence.common.CommonDAO.findClassifiers(CommonDAO.java:1609)
...

"Servlet.Engine.Transports : 8" daemon prio=5 tid=0x0257e900 nid=0x56 waiting on condition [5ec7c000..5ec7fc30]
at java.lang.Thread.sleep(Native Method)
at oracle.toplink.internal.helper.ConcurrencyManager.releaseDeferredLock(ConcurrencyManager.java:389)
at oracle.toplink.internal.identitymaps.CacheKey.releaseDeferredLock(CacheKey.java:311)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:619)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:378)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObjectsInto(ObjectBuilder.java:667)
at oracle.toplink.internal.queryframework.DatabaseQueryMechanism.buildObjectsFromRows(DatabaseQueryMechanism.java:141)
at oracle.toplink.queryframework.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:468)
at oracle.toplink.queryframework.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:800)
at oracle.toplink.queryframework.DatabaseQuery.execute(DatabaseQuery.java:603)
at oracle.toplink.queryframework.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:768)
at oracle.toplink.queryframework.ReadAllQuery.execute(ReadAllQuery.java:436)
at oracle.toplink.publicinterface.Session.internalExecuteQuery(Session.java:2062)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:981)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:938)
at com.saic.ct.sys.persistence.common.BaseDAO.executeQueryUsingSession(BaseDAO.java:1249)
at com.saic.ct.sys.persistence.common.BaseDAO.findMany(BaseDAO.java:1432)
at com.saic.ct.sys.persistence.common.BaseDAO.findMany(BaseDAO.java:1407)
at com.saic.ct.sys.persistence.common.CommonDAO.findClassifiers(CommonDAO.java:1609)
...

"Servlet.Engine.Transports : 5" daemon prio=5 tid=0x02d99b40 nid=0x53 waiting on condition [5e97c000..5e97fc30]
at java.lang.Thread.sleep(Native Method)
at oracle.toplink.internal.helper.ConcurrencyManager.releaseDeferredLock(ConcurrencyManager.java:389)
at oracle.toplink.internal.identitymaps.CacheKey.releaseDeferredLock(CacheKey.java:311)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:619)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:378)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObjectsInto(ObjectBuilder.java:667)
at oracle.toplink.internal.queryframework.DatabaseQueryMechanism.buildObjectsFromRows(DatabaseQueryMechanism.java:141)
at oracle.toplink.queryframework.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:468)
at oracle.toplink.queryframework.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:800)
at oracle.toplink.queryframework.DatabaseQuery.execute(DatabaseQuery.java:603)
at oracle.toplink.queryframework.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:768)
at oracle.toplink.queryframework.ReadAllQuery.execute(ReadAllQuery.java:436)
at oracle.toplink.publicinterface.Session.internalExecuteQuery(Session.java:2062)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:981)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:938)
at com.saic.ct.sys.persistence.common.BaseDAO.executeQueryUsingSession(BaseDAO.java:1249)
at com.saic.ct.sys.persistence.common.BaseDAO.findMany(BaseDAO.java:1432)
at com.saic.ct.sys.persistence.common.BaseDAO.findMany(BaseDAO.java:1407)
at com.saic.ct.sys.persistence.common.CommonDAO.findClassifiers(CommonDAO.java:1609)
...

"Servlet.Engine.Transports : 3" daemon prio=5 tid=0x02407580 nid=0x4f waiting on condition [5f17c000..5f17fc30]
at java.lang.Thread.sleep(Native Method)
at oracle.toplink.internal.helper.ConcurrencyManager.releaseDeferredLock(ConcurrencyManager.java:389)
at oracle.toplink.internal.identitymaps.CacheKey.releaseDeferredLock(CacheKey.java:311)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:619)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:378)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObjectsInto(ObjectBuilder.java:667)
at oracle.toplink.internal.queryframework.DatabaseQueryMechanism.buildObjectsFromRows(DatabaseQueryMechanism.java:141)
at oracle.toplink.queryframework.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:468)
at oracle.toplink.queryframework.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:800)
at oracle.toplink.queryframework.DatabaseQuery.execute(DatabaseQuery.java:603)
at oracle.toplink.queryframework.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:768)
at oracle.toplink.queryframework.ReadAllQuery.execute(ReadAllQuery.java:436)
at oracle.toplink.publicinterface.Session.internalExecuteQuery(Session.java:2062)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:981)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:938)
at com.saic.ct.sys.persistence.common.BaseDAO.executeQueryUsingSession(BaseDAO.java:1249)
at com.saic.ct.sys.persistence.common.BaseDAO.findMany(BaseDAO.java:1432)
at com.saic.ct.sys.persistence.common.BaseDAO.findMany(BaseDAO.java:1407)
at com.saic.ct.sys.persistence.common.CommonDAO.findClassifiers(CommonDAO.java:1609)
...

"Servlet.Engine.Transports : 2" daemon prio=5 tid=0x027d1bf0 nid=0x49 waiting on condition [5ef7c000..5ef7fc30]
at java.lang.Thread.sleep(Native Method)
at oracle.toplink.internal.helper.ConcurrencyManager.releaseDeferredLock(ConcurrencyManager.java:389)
at oracle.toplink.internal.identitymaps.CacheKey.releaseDeferredLock(CacheKey.java:311)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:619)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneNormally(ObjectBuilder.java:443)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:413)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:376)
at oracle.toplink.queryframework.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:451)
at oracle.toplink.queryframework.ObjectLevelReadQuery.registerIndividualResult(ObjectLevelReadQuery.java:1701)
at oracle.toplink.queryframework.ReadAllQuery.registerResultInUnitOfWork(ReadAllQuery.java:686)
at oracle.toplink.queryframework.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:466)
at oracle.toplink.queryframework.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:800)
at oracle.toplink.queryframework.DatabaseQuery.execute(DatabaseQuery.java:603)
at oracle.toplink.queryframework.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:768)
at oracle.toplink.queryframework.ReadAllQuery.execute(ReadAllQuery.java:436)
at oracle.toplink.queryframework.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:825)
at oracle.toplink.publicinterface.UnitOfWork.internalExecuteQuery(UnitOfWork.java:2532)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:981)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:938)
at com.saic.ct.sys.persistence.common.BaseDAO.executeQueryUsingUowOld(BaseDAO.java:256)
at com.saic.ct.sys.persistence.common.BaseDAO.findVOsForUpdate(BaseDAO.java:485)
at com.saic.ct.sys.persistence.common.BaseDAO.findVOByPK(BaseDAO.java:291)
at com.saic.ct.sys.persistence.schedule.ScheduleDAO.findWorkGroupByPK(ScheduleDAO.java:1248)
...

"Servlet.Engine.Transports : 1" daemon prio=5 tid=0x029773e0 nid=0x46 waiting on condition [664fc000..664ffc30]
at java.lang.Thread.sleep(Native Method)
at oracle.toplink.internal.helper.ConcurrencyManager.releaseDeferredLock(ConcurrencyManager.java:389)
at oracle.toplink.internal.identitymaps.CacheKey.releaseDeferredLock(CacheKey.java:311)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:619)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:378)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObjectsInto(ObjectBuilder.java:667)
at oracle.toplink.internal.queryframework.DatabaseQueryMechanism.buildObjectsFromRows(DatabaseQueryMechanism.java:141)
at oracle.toplink.queryframework.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:468)
at oracle.toplink.queryframework.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:800)
at oracle.toplink.queryframework.DatabaseQuery.execute(DatabaseQuery.java:603)
at oracle.toplink.queryframework.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:768)
at oracle.toplink.queryframework.ReadAllQuery.execute(ReadAllQuery.java:436)
at oracle.toplink.publicinterface.Session.internalExecuteQuery(Session.java:2062)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:981)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:938)
at com.saic.ct.sys.persistence.common.BaseDAO.executeQueryUsingSession(BaseDAO.java:1249)
at com.saic.ct.sys.persistence.common.BaseDAO.findMany(BaseDAO.java:1432)
at com.saic.ct.sys.persistence.common.BaseDAO.findMany(BaseDAO.java:1407)
at com.saic.ct.sys.persistence.common.CommonDAO.findClassifiers(CommonDAO.java:1609)
...

"Servlet.Engine.Transports : 0" daemon prio=5 tid=0x0241ce38 nid=0x44 waiting on condition [5fe7c000..5fe7fc30]
at java.lang.Thread.sleep(Native Method)
at oracle.toplink.internal.helper.ConcurrencyManager.releaseDeferredLock(ConcurrencyManager.java:389)
at oracle.toplink.internal.identitymaps.CacheKey.releaseDeferredLock(CacheKey.java:311)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:619)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:378)
at oracle.toplink.internal.descriptors.ObjectBuilder.buildObjectsInto(ObjectBuilder.java:667)
at oracle.toplink.internal.queryframework.DatabaseQueryMechanism.buildObjectsFromRows(DatabaseQueryMechanism.java:141)
at oracle.toplink.queryframework.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:468)
at oracle.toplink.queryframework.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:800)
at oracle.toplink.queryframework.DatabaseQuery.execute(DatabaseQuery.java:603)
at oracle.toplink.queryframework.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:768)
at oracle.toplink.queryframework.ReadAllQuery.execute(ReadAllQuery.java:436)
at oracle.toplink.publicinterface.Session.internalExecuteQuery(Session.java:2062)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:981)
at oracle.toplink.publicinterface.Session.executeQuery(Session.java:938)
at com.saic.ct.sys.persistence.common.BaseDAO.executeQueryUsingSession(BaseDAO.java:1249)
at com.saic.ct.sys.persistence.common.BaseDAO.findMany(BaseDAO.java:1432)
at com.saic.ct.sys.persistence.common.BaseDAO.findMany(BaseDAO.java:1407)
at com.saic.ct.sys.persistence.common.CommonDAO.findClassifiers(CommonDAO.java:1609)
...
  • 1. Re: Hung Threads (Toplink 10.1.3)
    cdelahun Pro
    Currently Being Moderated
    Hello Zev,

    I can't tell much from the threads. The ConcurrencyManager basically synchronizes objects in the cache. From the looks of it, these 10 threads are all accessing the same object, waiting for another thread to release its lock on it

    If you can get the problem again, the following can be used to print off which threads have the lock that is causing the problem:
    session.getIdentityMapAccessor().printIdentityMapLocks();

    With this, and the TopLink log, it maybe possible to figure out what situation is leading to this problem.

    Best Regards,
    Chris
  • 2. Re: Hung Threads (Toplink 10.1.3)
    490647 Newbie
    Currently Being Moderated
    zb,
    I am experiencing the same problem. I am running Toplink 10.1.3 in a WebLogic 8.1 cluster.

    The situation is that every thread that tries to update a Toplink persisted object hangs in ConcurrencyManager.acquire() or ConcurrencyManager.acquireDeferredLock(). It occurs under very light load. I haven't been able to determine what is triggering it.

    The problem appears to be associated with asynchronous JMS cache synchronization. We were running synchronous RMI and never saw the problem, switched to asynchronous JMS and the problem occured a number of times in a row. We switched back to synchronous RMI and haven't had the problem since.

    Are you also running in a cluster with cache synchronization? If so how is it configured? Have you made any progress in resolving the issue?

    Thanks,
    Thomas Buchan
  • 3. Re: Hung Threads (Toplink 10.1.3)
    zb Newbie
    Currently Being Moderated
    The issue is currently unresolved. We continue to try and diagnose this issue. We are using a cluster, but we have set "refresh always" on certain objects. We tried using a JMS queue with changesets but we ran into some issues. We will soon try again with invalidation. I dont think the clustering is the trigger for us. The issue for us remains intermittent and we do not have a single test case that can reproduce it. As this is extremely important to us, please post here if you do have some method or process for reproducing this issue.
    Thanks,
    Zev.
  • 4. Re: Hung Threads (Toplink 10.1.3)
    504035 Explorer
    Currently Being Moderated
    You have access to the source code of a light version of TopLink:
    http://fisheye5.cenqua.com/viewrep/~raw,r=1.3/glassfish/entity-persistence/src/java/oracle/toplink/essentials/internal/helper/WriteLockManager.java
    http://fisheye5.cenqua.com/viewrep/~raw,r=1.1/glassfish/entity-persistence/src/java/oracle/toplink/essentials/internal/helper/ConcurrencyManager.java

    WriteLockManager.java have recently received some fixes, one of them named: "deferred lock bug". You may want to check if your version of TopLink has those fixes:

    1.3 annotated / raw | Diffs: previous, other | Lines: 399 ( +32, -20 )

    Created: 2006-04-10 16:40:11 -0500 (23 days ago) | Author: tware | Changeset: MAIN:tware:20060410213856

    Issue number: entity-persistence update
    Obtained from: TopLink
    Submitted by: Tom Ware
    Reviewed by: TopLink Team

    - Added oracle.toplink.essentials.PersistenceProvider as potential persistence provider name
    - Glassfish issue #545
    - Fixes concurrency performance issues, fixes deferred lock bug and fixes WriteLock concurrency issue.
    - Glassfish issue # 486
    - Fixed binding problem. Returned back to binding literals, unless the platform is DB2 or Derby.
    - Glassfish issue #530
    - Glassfish issue #494

    Tested with QuickLook, entity-persistence-tests and Oracle-internal tests
  • 5. Re: Hung Threads (Toplink 10.1.3)
    JamesSutherland Pro
    Currently Being Moderated
    This is a bug in 10.1.3. Please contact Oracle technical support with this issue.
  • 6. Re: Hung Threads (Toplink 10.1.3)
    490647 Newbie
    Currently Being Moderated
    Zev,
    I contacted Oracle technical support and they gave me a patch for the deadlock problem. Unfortunetaly after about a week of testing we encountered the same problem again, it seems the patch doesn't fix the problem.

    I am still working with tech support, but right now things don't look good.

    Thomas Buchan
  • 7. Re: Hung Threads (Toplink 10.1.3)
    zb Newbie
    Currently Being Moderated
    I worked with Oracle support and was able to generate a test case (on our system) that reproduced the hung thread issue 100% of the time. Using WSAD in debug mode (to slow it down), I opened 50 groups of 5 threads each. Each thread acted in a different way according to my idea of how the issue occured. After talking to Toplink support, a slight modification brought about the hung thread issue. We applied the patch and it worked in the test case properly. No more hung threads were seen. This was a test case and not the production system but the issue disappeared in the testcase. Are you sure that you deployed the patch properly? Which patch are you using? It might be different. I would need more information to confirm it is the same issue since our objects are obviously different.
    Send me a log of your server starting and logging into Toplink.
    Zev.
  • 8. Re: Hung Threads (Toplink 10.1.3)
    490647 Newbie
    Currently Being Moderated
    Zev,
    The patch I applied was distributed in the file p5206620_101300_GENERIC.zip and contains the ObjectBuilder and Version classes. I'm pretty sure it is appplied correctly, I will verify again.

    This are the Toplink startup logs:

    [TopLink Info]: 2006.05.18 02:29:32.737--TopLink, version: Oracle TopLink - 10g Release 3 (10.1.3.0.0) (Build Patch for Bugs 5145690 and 5156075)
    [TopLink Info]: 2006.05.18 02:29:32.775--Server: WebLogic Server 8.1 SP4 Mon Nov 29 16:21:29 PST 2004 471647
    [TopLink Config]: 2006.05.18 02:29:32.807--connecting(DatabaseLogin(
         platform=>Oracle9Platform
         user name=> "epharm_owner"
         connector=>JNDIConnector datasource name=>com/pfizer/pgrd/epharm/ds1
    ))
    [TopLink Config]: 2006.05.18 02:29:35.390--Connected: jdbc:oracle:thin:@nlsunk770b:1521:clnval3
         User: EPHARM_OWNER
         Database: Oracle Version: Oracle9i Enterprise Edition Release 9.2.0.5.0 - Production
    With the Partitioning, OLAP and Oracle Data Mining options
    JServer Release 9.2.0.5.0 - Production
         Driver: Oracle JDBC driver Version: 9.2.0.4.0
    [TopLink Config]: 2006.05.18 02:29:35.392--connecting(DatabaseLogin(
         platform=>Oracle9Platform
         user name=> "epharm_owner"
         connector=>JNDIConnector datasource name=>com/pfizer/pgrd/epharm/ds1
    ))
    [TopLink Config]: 2006.05.18 02:29:35.405--Connected: jdbc:oracle:thin:@nlsunk770b:1521:clnval3
         User: EPHARM_OWNER
         Database: Oracle Version: Oracle9i Enterprise Edition Release 9.2.0.5.0 - Production
    With the Partitioning, OLAP and Oracle Data Mining options
    JServer Release 9.2.0.5.0 - Production
         Driver: Oracle JDBC driver Version: 9.2.0.4.0

    Thomas Buchan
  • 9. Re: Hung Threads (Toplink 10.1.3)
    zb Newbie
    Currently Being Moderated
    looks like you applied it correctly. the only other thing to note is we use WAS instead of Weblogic and we upgraded to the latest JDBC driver version 10.2.0.1.0. and use Oracle 9.2.0.7
    Zev.
  • 10. Re: Hung Threads (Toplink 10.1.3)
    445013 Newbie
    Currently Being Moderated
    We have installed the patch also, but the productionserver still hangs several times a day. We create a SR today. I except that toplink patch this bug end of this week. The problem occurs when two threads access an indirectList. One thread is running inside a transaction and the other one outside a transaction.
  • 11. Re: Hung Threads (Toplink 10.1.3)
    452360 Newbie
    Currently Being Moderated
    Has this problem been solved now?
  • 12. Re: Hung Threads (Toplink 10.1.3)
    479104 Newbie
    Currently Being Moderated
    Do you have any one to many mappings that dont have indirection turned on ?

    What we did in these kind of situations is to identify the problem mapping. Unmap that particular attribute and see if you can still reproduce the deadlock. If the problem still happens then this may also be of some help.
    Always use the same path for accessing the object tree. In other words if say A and B are involved in 1 many mapping. Always get a reference to B from A. Do not access A from B. Avoid the circular routes for accessing objects.


    We have installed the patch also, but the
    productionserver still hangs several times a day. We
    create a SR today. I except that toplink patch this
    bug end of this week. The problem occurs when two
    threads access an indirectList. One thread is running
    inside a transaction and the other one outside a
    transaction.
  • 13. Re: Hung Threads (Toplink 10.1.3) - RESOLVED
    DougClarke Employee ACE
    Currently Being Moderated
    There are two issues that were identified and resolved through patches available through Oracle support.

    Please open a support case if you encountered these and they can provide you the patch.

    Our next release will include both of these patches.

    Doug
  • 14. Re: Hung Threads (Toplink 10.1.3)
    445013 Newbie
    Currently Being Moderated
    Yes this patch resolves our problem. The patch number is 5442078
1 2 Previous Next