This discussion is archived
4 Replies Latest reply: Mar 2, 2012 8:24 AM by vinothchandar RSS

High GC promotion rate on cursor walk with CacheMode.UNCHANGED

vinothchandar Newbie
Currently Being Moderated
Hi,

Our app has a daily clean up job, which does a cursor walk and cleans up old entries. The normal ops/sec is much much lower (say 1000) than the number of records scanned using the cursor/sec. This caused high eviction rate, high cache miss in bdb and high promotion rate into old gen. My guess was that the primed cache entries due to normal ops, were being evicted and new data brought into the cache by the cursor walk was being promoted into the cache.

Ideally, the cursor walk should not interfere with normal operations. I want to have the entried brought in by the cursor walk to be evicted first and early, so they can be collected in young gen itself. Hence, I set the cursor cache mode to UNCHANGED. But I still observe the same promotion rate, which I have difficulty in understanding.

I know BDB je puts the entries into the cache (as opposed to releasing them right away) and then evicts them, even when cachemode.UNCHANGED is used. is this a factor here?

Can you please share any thoughts on how to achieve this?
  • 1. Re: High GC promotion rate on cursor walk with CacheMode.UNCHANGED
    greybird Expert
    Currently Being Moderated
    Hello,

    Yes, when not all records fit in cache, then the cursor walk will bring records into the cache, and they will not normally be evicted until they become "older" than other entries in the cache, since cache eviction is done using an LRU algorithm. This can cause GC promotion, as you say.

    CacheMode.UNCHANGED won't impact this behavior. It only has an impact on records that are already in cache. Using UNCHANGED will leave their cache "age" unchanged, if they are already in cache, as opposed to DEFAULT, which will update the age to the current time.

    CacheMode.EVICT_LN is probably the best thing you can use to reduce the GC promotion issue. It will evict the record as soon as the cursor moves or is closed. However, it will also evict records encountered during the scan, that are also being used in other/normal operations. Those operations may be impacted, but you may still see much better performance. I suggest trying this.

    JE does not have a CacheMode that evicts a record immediately (when the cursor moves or closes), but only if it was not already in cache. That may be better for your app, but it doesn't exist. So I suggest trying EVICT_LN.

    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
  • 2. Re: High GC promotion rate on cursor walk with CacheMode.UNCHANGED
    vinothchandar Newbie
    Currently Being Moderated
    Really appreciate your help!

    I see. The java doc is slightly misleading though. http://docs.oracle.com/cd/E17277_02/html/java/com/sleepycat/je/CacheMode.html#UNCHANGED says "If the record was added to the cache by this operation, it will have "maximum coldness" and will therefore be colder than other records.". But, you are right. It does not make hard guarantees about when eviction will happen. I went through CursorImpl.performEviction as well to confirm.

    We already considered EVICT_LN (and even EVICT_BIN). Some background is that we are on SSDs with high memory churn. The problem is, it seems it will help different workloads to different extents in terms of earlier eviction. I am looking for a fool proof way that will make sure I can avoid GC entirely, since for all technical purposes, the cursor walk has nothing to do with normal workload. But, seems like this is not available.

    One final question :
    Is it possible to open another env handle (without closing already open env handle), with a non shared cache, for the cursor walk alone? Basic idea is to have a separate cache for the cursor walk alone. Is this doable at all. Please pardon, if my desperation shows :)
  • 3. Re: High GC promotion rate on cursor walk with CacheMode.UNCHANGED
    greybird Expert
    Currently Being Moderated
    I see. The java doc is slightly misleading though. http://docs.oracle.com/cd/E17277_02/html/java/com/sleepycat/je/CacheMode.html#UNCHANGED says "If the record was added to the cache by this operation, it will have "maximum coldness" and will therefore be colder than other records.". But, you are right. It does not make hard guarantees about when eviction will happen. I went through CursorImpl.performEviction as well to confirm.
    Yes, you're right. It will have maximum coldness, but it will still stick around in the cache for a while, until eviction happens naturally. This may be long enough to be promoted.
    We already considered EVICT_LN (and even EVICT_BIN). Some background is that we are on SSDs with high memory churn. The problem is, it seems it will help different workloads to different extents in terms of earlier eviction. I am looking for a fool proof way that will make sure I can avoid GC entirely, since for all technical purposes, the cursor walk has nothing to do with normal workload. But, seems like this is not available.
    EVICT_LN is a good way to avoid GC, as much as is possible. You can specify EVICT_LN just for the cursor doing the scan, for a Database or the entire Environment. I know you have reservations, but I suggest trying it.
    One final question :
    Is it possible to open another env handle (without closing already open env handle), with a non shared cache, for the cursor walk alone? Basic idea is to have a separate cache for the cursor walk alone. Is this doable at all. Please pardon, if my desperation shows >

    Nope. Opening another env handle in the same JVM process will simply share the cache. Opening it in another process will lock out log cleaning, which is not a good thing.

    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
  • 4. Re: High GC promotion rate on cursor walk with CacheMode.UNCHANGED
    vinothchandar Newbie
    Currently Being Moderated
    Yes - mark. That seems to be the way forward. Will explore it. Thanks for your insights. helps..

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points