4 Replies Latest reply: Jul 27, 2012 6:06 AM by 928454 RSS

    My ReentrantReadWriteLock.writeLock's owning thread has gone missing

    928454
      We had a deadlock-like failure of our application recently. I initially reported it on the BDB JE forum (Re: Deadlock found in je-5.0.48 but further analysis of the heap and thread dumps has exposed a problem that looks like a Java bug. I'm hoping someone on this forum can offer advice on whether this is the case.

      We’re using Oracle JVM 1.6.0_25-b06, running on Linux version: 2.6.18-194.32.1.el5.

      Thread t@41101 is blocked indefinitely in myReentrantReadWriteLock.writeLock().lock().

      The output of "jstack -l" should list which thread holds this exclusive lock in the "locked ownable synchronizers" section but does not.

      Our first theory was that the owning thread might have terminated.

      We wrote a simple test program to explore this. We found from heap dump analysis that even if the owning thread terminates, the lock itself still refers to it via the ReentrantReadWriteLock.WriteLock.sync.exclusiveOwnerThread field. Looking in the java.util.concurrent source code, it seems that this field only gets null'ed when the lock is released.

      However, looking in the heap dump taken following our "deadlock", we were surprised to find that the lock in question has a null sync.exclusiveOwnerThread field.

      Surely a write lock should be in one of two states (except possibly for a tiny instant when its state is being non-atomically switched):

      1) The lock is available, and sync.exclusiveOwnerThread is null
      2) The lock is unavailable, and sync.exclusiveOwnerThread is populated

      But our lock was indefinitely in this state:

      3) The lock is unavailable and sync.exclusiveOwnerThread is null


      Can anyone explain this strange behaviour?


      Thanks,
      Phil
        • 1. Re: My ReentrantReadWriteLock.writeLock's owning thread has gone missing
          806469
          philipharvey44 wrote:
          ...

          Thread t@41101 is blocked indefinitely in myReentrantReadWriteLock.writeLock().lock().
          I did not study the sources but just two things I wonder. First I hope this my in myReentrantReadWriteLock is a typo. If you use your own implementation it will be hard to get help.
          philipharvey44 wrote:
          ....
          3) The lock is unavailable and sync.exclusiveOwnerThread is null
          That's what I would expect if the lock is acquired by a reader. Several reader can access at the same time so there is no need to set an exclusiveOwnerThread

          Good luck

          Edited by: domi on 26.07.2012 19:02 Corrected some typo
          • 2. Re: My ReentrantReadWriteLock.writeLock's owning thread has gone missing
            928454
            >
            I did not study the sources but just two things I wonder. First I hope this my in myReentrantReadWriteLock is a typo. If you use your own implementation it will be hard to get help.
            >

            I used the term "myReentrantReadWriteLock" to refer to a particular instance of ReentrantReadWriteLock. We are not using our own lock subclass.
            domi wrote:
            3) The lock is unavailable and sync.exclusiveOwnerThread is null
            That's what I would expect if the lock is acquired by a reader. Several reader can access at the same time so there is no need to set an exclusiveOwnerThread
            >

            We've already ascertained by cross-referencing the stack trace with our source code that no read locks have been taken.

            Sorry I didn't make these points clear in my original question.

            Edited by: philipharvey44 on 27-Jul-2012 01:30
            • 3. Re: My ReentrantReadWriteLock.writeLock's owning thread has gone missing
              EJP
              You must release the lock in a finally {} block so you can be sure it happens, and it specifically says so in the Javadoc. It is your responsibility.
              • 4. Re: My ReentrantReadWriteLock.writeLock's owning thread has gone missing
                928454
                EJP wrote:
                You must release the lock in a finally {} block so you can be sure it happens, and it specifically says so in the Javadoc. It is your responsibility.
                Agreed, although that wasn't the point of my question. My question was really about the strange state that the lock is in. To reiterate:
                philipharvey44 wrote:
                analysis of the heap and thread dumps has exposed a problem that looks like a Java bug. I'm hoping someone on this forum can offer advice on whether this is the case.
                ...
                philipharvey44 wrote:
                ... our lock was indefinitely in this state:
                3) The lock is unavailable and sync.exclusiveOwnerThread is null
                Can anyone explain this strange behaviour?