5 Replies Latest reply on Sep 21, 2012 7:07 PM by Linda Lee-Oracle

    Heap Space Error while Storing Keys in Berkeley Database


      I am trying to store 100,000 keys. I am using 2 environments simultaneously. The problem is I am encountering heap space error. I did not encounter this when the environments are not writing simultaneously.

      Please someone help me with the error. Thanks in advance.
        • 1. Re: Heap Space Error while Storing Keys in Berkeley Database
          "Andrei Costache, Oracle-Oracle"

          So you are seeing OOME errors.
          You mentioned that you are using multiple environments. I assume you are likely using a shared cache for these two environments. Here is some documentation reference:
          Multiple Environments
          Configuring a Shared Cache for Multiple Environments

          To estimate the optimal size for you cache, you could use the DbCacheSize, as discussed in the following documentation sections:
          Sizing the Cache
          How can I estimate my application's optimal cache size?
          and use one of the appropriate methods / constant fields representing the cache properties, to size the cache appropriately:
          EnvironmentMutableConfig.setCacheSize(long) / EnvironmentConfig.MAX_MEMORY
          EnvironmentMutableConfig.setCachePercent(int) / EnvironmentConfig.MAX_MEMORY_PERCENT

          I will need to refine a little the previous answer I gave you, so that I point out that JE does live within a cache, and will not indiscriminately exceed that memory boundary.
          You don't really need to increase the cache size to avoid an OOME. To clarify, the problem lies in what JE thinks the cache size is, relative to the available memory in the Java process.

          By default, JE will use a certain 60% of the JVM heap. However, we assume that there is one environment used in the process. If you use two (or more) environments in the same process (as in your case), and don't reduce their cache sizes, or use a shared cache, they will each use 60%, which of course adds up to 120%, which leads to an OOME.

          So you need to reduce the cache size per environment, or use a shared cache rather than increase the cache size.


          Edited by: Andrei Costache, Oracle on Sep 5, 2012 5:44 PM
          1 person found this helpful
          • 2. Re: Heap Space Error while Storing Keys in Berkeley Database

            I tried the solution of shared cache, no more heap space error. The problem now is that only one environment is writing the records. The other environment creates the database environment but didn't write the records. Please advice some possible solutions.

            Thank you in advance.
            • 3. Re: Heap Space Error while Storing Keys in Berkeley Database
              The use of a shared cache doesn't influence whether records are written or not. There must be something different that you're doing for the two environments, or something you're misunderstanding. The information you've given isn't nearly enough to know what that is.

              Perhaps it would be best if you were to write a small standalone test, in a single .java file with a main method, that demonstrates the problem you're having. If you post this source code, then we should be able to understand the problem you're having.

              • 4. Re: Heap Space Error while Storing Keys in Berkeley Database

                For the writing of records, I made it a GWT application for the two environments to write simultaneously. Is there any way to speed up the writing of records? My current environment config is like this:
                EnvironmentConfig environmentConfig = new EnvironmentConfig();


                Edited by: 955980 on Sep 19, 2012 10:37 PM
                • 5. Re: Heap Space Error while Storing Keys in Berkeley Database
                  Linda Lee-Oracle
                  There are a few things that are unclear about your situation. For example, I don't understand if you are using two different environments (with different environment home directories) or if you have multiple threads with the different handles, or Environment instances, for the same directory. I think you mean the former (two different home directories).

                  Also, you said this:

                  but you must have left something out. You should show us what the durability value is, such as Durability.COMMIT_SYNC, Durability.COMMIT_NO_SYNC, etc. That's can have a big performance impact. You will want to read the javadoc for Durability about the differences.

                  But in the end, we can't answer your question without more information. There is no reason within BDB JE why two environments (with different environment directories) should have any impact on each other, except for system resources. There's no coordination between the two environments, except that they are both using the system's memory and I/O resources. One way to approach this is to take thread stack dumps when the system is not behaving well. That would help you identify a potential bottleneck, whether it is in BDB JE or elsewhere.