7 Replies Latest reply on Jan 9, 2009 6:43 PM by 678712

    Memory leak issue with static basicWriteLockInfo

    658136
    658136 Sep 4, 2008 2:52 PM
      Hi,

      I am using JE 3.3.69 and am seeing a memory issue that I do not see with JE 3.2.76. The application loads the database (filling the DB cache in the process), fully closes the environment, re-opens the environment, and then preloads databases until the DB cache of the new environment is full. What seems to be happening is that the the DB cache from the closed environment is never garbage collected and the preload causes an out-of-memory error since the heap is not big enough for two full caches.

      I saw that there is a static field in WriteLockInfo referencing a database (through the abortDb field) and is thereby holding on to the objects that make up the DB cache of the closed environment.

      static final WriteLockInfo basicWriteLockInfo =
           new WriteLockInfo();

      Should this static field belong to the environment instead?
      Is there a work-around I can try?

      Thanks,
      • 4009 Views
      • Tags:
        • 1. Re: Memory leak issue with static basicWriteLockInfo
          Linda Lee-Oracle
          Linda Lee-Oracle Sep 4, 2008 3:41 PM (in response to 658136)
          Andy,

          Can you give us a little more detail on the two loads? Are the loads are done non-transactionally -- that is, with the environments and databases both opened in non-transactional mode?

          Thanks

          Linda
          • 2. Re: Memory leak issue with static basicWriteLockInfo
            658136
            658136 Sep 4, 2008 5:59 PM (in response to Linda Lee-Oracle)
            Linda,

            The first load of the data is non-transactional. The subsequent preload is in a transactional environment.

            I made some JE source changes to move the static basicWriteLockInfo out of WriteLockInfo and make it a non-static field of EnvironmentImpl. Initial testing is encouraging and suggests that this fixes the problem, i.e. the db cache from the first load of the data is garbage collected before the preload. I need to do some more testing though.

            Andy
            • 3. Re: Memory leak issue with static basicWriteLockInfo
              Charles Lamb
              Charles Lamb Sep 4, 2008 7:01 PM (in response to 658136)
              Andy, please drop me an email. I don't seem to have your current email address. Thanks.

              charles.lamb @ o.com
              • 4. Re: Memory leak issue with static basicWriteLockInfo
                Charles Lamb
                Charles Lamb Sep 11, 2008 3:04 AM (in response to Charles Lamb)
                To close this out, we understand the cause of this bug and will put a fix in the next release (anything after 3.3.71 will have the fix). It will only affect users who disable locking.

                Charles Lamb
                • 5. Re: Memory leak issue with static basicWriteLockInfo
                  678712
                  678712 Jan 8, 2009 1:42 AM (in response to Charles Lamb)
                  This bug, or something very similar, still exists for databases using the shared cache.
                  The whole shared cache ends up leaked.

                  I don't know what is causing the leak, and if it is the same or similar internal cause as the previous bug.

                  I can confirm that for databases using their own (non-shared) cache, it no longer occurs but did in the past.

                  Reproduction:
                  Open Environment A (large) with locking disabled, using the shared cache, set the cache to a known percent value.
                  Open Environment B, C, D, etc in the same shared cache

                  Access Environment A until a large amount of RAM is used by the cache. Stop activity. Garbage Collect. Note remaining heap size (JConsole, Jstat, JMX, various other ways).

                  Open Environment A2 (same thing, different dir -- in our case we are swapping out updated data).
                  Close Environment A. Garbage collect -- no difference in size. Note, A2 has not been accessed at all.

                  Access Environment A2 again causing more memory to be used. It will grow well past its previous level above.
                  Open Environment A3.
                  Close Environment A2. Garbage collect -- it will free up all the stuff associated with A2, but all the stuff from A still remains.
                  The effect is as if the first shared lru cache is lost. Repeated opening and closing recycles the memory properly. But whatever is accessed in the "first" opening is lost forever.

                  When you use a cache that is 40% of the heap size (3500MB heap) and you lose 40% of the heap, its bad.

                  We are unable to use the shared cache feature due to this bug, though we are thinking about investigating a purposeful open/access/close upon startup to work around it.
                  • 6. Re: Memory leak issue with static basicWriteLockInfo
                    Charles Lamb
                    Charles Lamb Jan 8, 2009 8:53 PM (in response to 678712)
                    Could you please tell us if it happens when you don't open Environment A with isLocking disabled?

                    Thanks.

                    Charles Lamb
                    • 7. Re: Memory leak issue with static basicWriteLockInfo
                      678712
                      678712 Jan 9, 2009 6:43 PM (in response to Charles Lamb)
                      I won't have time today to get to testing with EnvironmentConfig.setLocking(true); I will be unavailable for a few weeks and be able to test with locking enabled when I return.
                      The parameter has always been false for us; this process does not write at all but has multiple readers. We first started seeing memory problems after upgrading to the version with shared caches, we noticed the newer version that had a claimed fix for the leak but it did not help. After some experimentation, we found that it was the environments using the shared cache that leaked, and not the others. Switching all but the very small ones (which we don't care if they leak once) to not use the shared cache is the current work-around.
                      All we know is the following:

                      Older versions (prior to shared caches): locking = false, no leak but we were only using one environment in the process.
                      First version with shared caches, using shared caches and more than one environment: locking = false, leaks with shared caches
                      After the leak fix ( .75), using shared caches with locking = false, leaks. Using non-shared caches with locking = false, does not leak.

                      Thanks!