This discussion is archived
5 Replies Latest reply: Sep 21, 2012 12:07 PM by Linda Lee RSS

Heap Space Error while Storing Keys in Berkeley Database

958983 Newbie
Currently Being Moderated
Hi,

I am trying to store 100,000 keys. I am using 2 environments simultaneously. The problem is I am encountering heap space error. I did not encounter this when the environments are not writing simultaneously.

Please someone help me with the error. Thanks in advance.
  • 1. Re: Heap Space Error while Storing Keys in Berkeley Database
    Andrei Costache, Oracle Journeyer
    Currently Being Moderated
    Hi,

    So you are seeing OOME errors.
    You mentioned that you are using multiple environments. I assume you are likely using a shared cache for these two environments. Here is some documentation reference:
    Multiple Environments
    Configuring a Shared Cache for Multiple Environments

    To estimate the optimal size for you cache, you could use the DbCacheSize, as discussed in the following documentation sections:
    Sizing the Cache
    How can I estimate my application's optimal cache size?
    and use one of the appropriate methods / constant fields representing the cache properties, to size the cache appropriately:
    EnvironmentMutableConfig.setCacheSize(long) / EnvironmentConfig.MAX_MEMORY
    EnvironmentMutableConfig.setCachePercent(int) / EnvironmentConfig.MAX_MEMORY_PERCENT

    I will need to refine a little the previous answer I gave you, so that I point out that JE does live within a cache, and will not indiscriminately exceed that memory boundary.
    You don't really need to increase the cache size to avoid an OOME. To clarify, the problem lies in what JE thinks the cache size is, relative to the available memory in the Java process.

    By default, JE will use a certain 60% of the JVM heap. However, we assume that there is one environment used in the process. If you use two (or more) environments in the same process (as in your case), and don't reduce their cache sizes, or use a shared cache, they will each use 60%, which of course adds up to 120%, which leads to an OOME.

    So you need to reduce the cache size per environment, or use a shared cache rather than increase the cache size.


    Regards,
    Andrei

    Edited by: Andrei Costache, Oracle on Sep 5, 2012 5:44 PM
  • 2. Re: Heap Space Error while Storing Keys in Berkeley Database
    958983 Newbie
    Currently Being Moderated
    Hi,

    I tried the solution of shared cache, no more heap space error. The problem now is that only one environment is writing the records. The other environment creates the database environment but didn't write the records. Please advice some possible solutions.

    Thank you in advance.
  • 3. Re: Heap Space Error while Storing Keys in Berkeley Database
    greybird Expert
    Currently Being Moderated
    The use of a shared cache doesn't influence whether records are written or not. There must be something different that you're doing for the two environments, or something you're misunderstanding. The information you've given isn't nearly enough to know what that is.

    Perhaps it would be best if you were to write a small standalone test, in a single .java file with a main method, that demonstrates the problem you're having. If you post this source code, then we should be able to understand the problem you're having.

    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
  • 4. Re: Heap Space Error while Storing Keys in Berkeley Database
    958983 Newbie
    Currently Being Moderated
    Hi,

    For the writing of records, I made it a GWT application for the two environments to write simultaneously. Is there any way to speed up the writing of records? My current environment config is like this:
    EnvironmentConfig environmentConfig = new EnvironmentConfig();
    environmentConfig.setTransactional(true);
    environmentConfig.setAllowCreate(true);
    environmentConfig.setSharedCache(true);
    environmentConfig.setDurability(DURABILITY);

    Thanks.

    Edited by: 955980 on Sep 19, 2012 10:37 PM
  • 5. Re: Heap Space Error while Storing Keys in Berkeley Database
    Linda Lee Journeyer
    Currently Being Moderated
    There are a few things that are unclear about your situation. For example, I don't understand if you are using two different environments (with different environment home directories) or if you have multiple threads with the different handles, or Environment instances, for the same directory. I think you mean the former (two different home directories).

    Also, you said this:
    environmentConfig.setDurability(DURABILITY);

    but you must have left something out. You should show us what the durability value is, such as Durability.COMMIT_SYNC, Durability.COMMIT_NO_SYNC, etc. That's can have a big performance impact. You will want to read the javadoc for Durability about the differences.

    But in the end, we can't answer your question without more information. There is no reason within BDB JE why two environments (with different environment directories) should have any impact on each other, except for system resources. There's no coordination between the two environments, except that they are both using the system's memory and I/O resources. One way to approach this is to take thread stack dumps when the system is not behaving well. That would help you identify a potential bottleneck, whether it is in BDB JE or elsewhere.

    Regards,

    Linda

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points