This discussion is archived
1 2 3 Previous Next 34 Replies Latest reply: Jul 16, 2012 4:13 PM by greybird Go to original post RSS
  • 15. Re: Latency increase in upgrage to 4.1.17 from 4.0.92
    vinothchandar Newbie
    Currently Being Moderated
    do you mean a voldemort operation or a BDB operation?
  • 16. Re: Latency increase in upgrage to 4.1.17 from 4.0.92
    vinothchandar Newbie
    Currently Being Moderated
    Hi Mark,

    Following eviction code from CursorImpl, I see that under the following conditions, the cursor may not evict the LN right away.

    1. Its BIN is dirty and will be migrated soon (i.e Lazy migration)
    2. The file that the LN is in, will be cleaned shortly
    3. Some experimental clustering, which is off by default.

    Don't you think we should do base eviction on 1, 2 only when the LN is dirty? Even when logging BINs, JE logs deltas. So, I think as long as the LN is not dirty, we should let go here..
    Please let me know what you think.

    For now, I just need the confirmation on the fetchMiss/fetch stat interpretation from you. I will try to play with lazy migration to see if I can get more LNs off the JVM heap.
  • 17. Re: Latency increase in upgrage to 4.1.17 from 4.0.92
    greybird Expert
    Currently Being Moderated
    I'm saying that LNFetches may be incremented twice in a single JE operation (single read). So you should ignore it.

    By operation I mean a single read, from the app perspective.

    --mark                                                                                                                                                                                                                                                                                                                                                                                           
  • 18. Re: Latency increase in upgrage to 4.1.17 from 4.0.92
    greybird Expert
    Currently Being Moderated
    1. Its BIN is dirty and will be migrated soon (i.e Lazy migration)
    2. The file that the LN is in, will be cleaned shortly
    3. Some experimental clustering, which is off by default. >

    None of these will happen except 1, and only if you have lazy migration on.
    For now, I just need the confirmation on the fetchMiss/fetch stat interpretation from you. I will try to play with lazy migration to see if I can get more LNs off the JVM heap.
    I claim that if you turn off lazy migration, and you turn on EVICT_LN, that no LNs will be in cache (except for an active operation). The fetch stat will not help to see whether this is happening (as I explained above), so you can either trust me or do more in-depth debugging to get more info.

    If you think that LNs are not being evicted when they should be, the simplest thing is to write a very small test to confirm this. For each get(), you should see one cache miss. (Ignore the fetch stat, just look at the misses.) The test you described earlier is seeing one miss per get(), which confirms that EVICT_LN is working.

    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
  • 19. Re: Latency increase in upgrage to 4.1.17 from 4.0.92
    vinothchandar Newbie
    Currently Being Moderated
    Sure. I will explore lazy_cleaner_migration and see how it goes. Thanks a lot for confirming.

    Thanks
    Vinoth
  • 20. Re: Latency increase in upgrage to 4.1.17 from 4.0.92
    greybird Expert
    Currently Being Moderated
    Vinoth,

    I'm sorry if I confused the issue with what I said earlier about the use of the nLNsFetch and nLNsFetchMiss. The problem is that nLNsFetch is not the same as the number of records read (number of get() operations). So using these two stats to "gauge cache hit/miss ratios" (as the javadoc says!) isn't accurate. What is lacking in the stats are counters for the number of read and write operations.

    I've filed ticket [#21554], so we can think about the need for more stats later on, or at least remember to correct the javadoc.

    Thanks,
    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
  • 21. Re: Latency increase in upgrage to 4.1.17 from 4.0.92
    vinothchandar Newbie
    Currently Being Moderated
    Hi mark,

    That's okay. For verification purposes, I think we will do a 'jmap histo' and see how much LNs are there in the JVM heap. Its not that I don't trust you :) (of course I do, you are awesome). We are seeing heavy CMS activity, usually indicating fragmentation. I assume index nodes are all the same size pretty much, since I expect most of them to contain > 4 children, which means they will have default representation in memory. Since the CMS is real and could only be caused only by LN nodes being in the heap, I am super anxious to get this working.

    We currently have 3 cleaners with lazy migration turned on. I am going to try reducing the cleaners to 1 (its SSD, it should be able to keep up) and/or turning lazy migration off. This should drive away fragmentation. (if not, I am not sure what else to do). Will keep this thread updated

    Thanks
    Vinoth
  • 22. Re: Latency increase in upgrage to 4.1.17 from 4.0.92
    greybird Expert
    Currently Being Moderated
    I think it's worth calculating the LN miss rate using (nFetchMiss / nOps) where nOps is the number of JE operations you perform and is computed by your app. That way, if it is not 100%, then you can add more debugging to find out what ops are not evicting the LN.

    It is also possible the CMS activity is not due to LNs, but to mutations in BIN representations. We don't see this in our tests, but that doesn't mean it can't happen with a different access pattern.

    Looking at INTargetRep is a good tactic. Assuming LNs are not kept in cache, this should mostly be None, with some Sparse that are in the middle of an operation. This is what we see in JE 5 anyway. I assume you've seen the stats for this.

    When you say that the rep should be Default because they contain more than 4 children, that shouldn't be the case with EVICT_LN. It's the number of resident children that determines the rep, and LNs should not be resident. I'm speaking of the BINs, not the upper INs. The BINs dominate the cache.

    You should not trust me completely, seriously, since I often make mistakes and in this case it's possible I'm thinking of JE 5 behavior. I haven't done JE 4 performance testing since we started on JE 5, at least two years ago. It's possible that we improved behavior in this area in JE 5, even though EVICT_LN was present in JE 4.

    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
  • 23. Re: Latency increase in upgrage to 4.1.17 from 4.0.92
    vinothchandar Newbie
    Currently Being Moderated
    Hi mark,

    You were right. The lazy cleaner migration does reduce the number of LNs in the cache by ~75%. But we are still seeing high fragmentation even when cache is dominated by index nodes. We have assumed that the keys are uniform size so far. Is that true? (i.e would nt you have to make it some uniform size so you can compare them?).

    Pasted a histogram of the heap below
    num #instances #bytes class name
    ----------------------------------------------
    1: 98291348 14291470352 [B
    2: 1039956 2074705520 [J
    3: 1005044 1023373616 [[B
    4: 16374 184680608 [I
    5: 1043765 147572144 [Lcom.sleepycat.je.tree.Node;
    6: 1349020 69703856 [C
    7: 393821 53559656 com.sleepycat.je.tree.BIN
    8: 324342 46705248 com.sleepycat.je.tree.DBIN
    9: 348798 44646144 com.sleepycat.je.tree.DIN
    10: 1073152 42926080 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync
    11: 1073103 42924120 com.sleepycat.je.latch.SharedLatch
    12: 1263927 40445664 java.lang.String
    13: 1063045 34017440 java.util.concurrent.ConcurrentHashMap$HashEntry
    14: 985414 31533248 com.sleepycat.je.tree.INTargetRep$Sparse
    15: 997399 25219952 [S
    16: 302535 25094208 [Ljava.util.HashMap$Entry;
    17: 1004026 24096624 com.sleepycat.je.tree.INKeyRep$Default
    18: 718163 17235912 com.sleepycat.je.utilint.TinyHashSet
    19: 1073160 17170560 java.util.concurrent.locks.ReentrantReadWriteLock$Sync$ThreadLocalHoldCounter
    20: 1073152 17170432 java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock
    21: 1073152 17170432 java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock
    22: 298177 14312496 java.util.HashMap
    23: 421027 13472864 com.sleepycat.je.tree.LN
    24: 4164 11680680 [Ljava.util.concurrent.ConcurrentHashMap$HashEntry;
    25: 350754 11224128 com.sleepycat.je.tree.ChildReference
    26: 81357 11137784 <constMethodKlass>
    27: 81357 11074696 <methodKlass>
    28: 257973 8255136 java.util.HashMap$Entry
    29: 6736 7647456 <constantPoolKlass>
    30: 101940 5902336 <symbolKlass>
    31: 6736 5515992 <instanceKlassKlass>
    32: 150468 4814976 java.lang.ThreadLocal$ThreadLocalMap$Entry

    Thanks again
    Vinoth
    33: 5815 4702328 <constantPoolCacheKlass>
  • 24. Re: Latency increase in upgrage to 4.1.17 from 4.0.92
    greybird Expert
    Currently Being Moderated
    Hi Vinoth,

    No, keys are variable length and we can see that INKeyRep$Default is used, meaning one byte array per key. If the key size (without common prefix) is <= 16 bytes, we use a single byte array per BIN (INKeyRep$MaxKeySize). In JE 5 we added a config option for this constant (if 16 isn't best for your app) but that doesn't help you on JE 4.1. If you haven't configured key prefixing, you should, to have a chance of using INKeyRep$MaxKeySize.

    I see that all your BINs are using INTargetRep$Sparse. We added an optimization in JE 5 so that the INTargetRep$Sparse representation reverts to INTargetRep$None (0 bytes) when a BIN is empty. This consumes less cache. But the INTargetRep$Sparse object should not be allocated/freed often, so is probably not your GC issue.

    The main thing that strikes me is that you have as many DIN/DBIN nodes as BINs. This means (if I understand Voldemort) that your use of duplicates is very wasteful of cache memory, although I don't know if that's causing eviction which is then causing GC issues -- it certainly could be, since it triples the number of INs. It looks like many keys have a duplicate, meaning (I think) in Voldemort that these records have two versions. The worst case would be that there are many DBINs that contain only two keys each. I had understood that Voldemort would rarely create a duplicate (2nd version for a record), but what we're seeing in the heap dump is contrary to this, unless there is another explanation for the DIN/DBIN numbers (some other use of duplicates in your test?).

    I'm on vacation until next week.

    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
  • 25. Re: Latency increase in upgrage to 4.1.17 from 4.0.92
    vinothchandar Newbie
    Currently Being Moderated
    Hi mark,

    Thanks for confirming. Makes sense. We will think about dealing with this.

    Regarding the duplicates, voldemort could potentially have a lot of duplicates (~20%) with highly concurrent writes. and yes, if we do better in this front that would be great. I definitely have this noted down
    But right now, the fragmentation is more worrying and hurting us as I type.

    Have a good vacation! We can chat later.

    Thanks
    Vinoth
  • 26. Re: Latency increase in upgrage to 4.1.17 from 4.0.92
    greybird Expert
    Currently Being Moderated
    OK, so you're pretty sure the fragmentation is not caused by eviction, due to the DINs/DBINs using so much JE cache.
  • 27. Re: Latency increase in upgrage to 4.1.17 from 4.0.92
    vinothchandar Newbie
    Currently Being Moderated
    Well. There is always a factor which I tend to not mention in our conversations, to not confuse you. This relates to our multitenancy. In this particular case, there is ~600G of data shared across 20+ environments over a 10GB shared BDB cache. I think the fragmentation is due different key sizes across stores. As a short term fix, I am going to try keyprefixing, to see if it helps put out this fire for now.

    I would need to scope out a long term generic solution for this, since we are not moving away from multitenancy in the future (due to obvious operational advantages). Hope that clarifies where I am coming from.
  • 28. Re: Latency increase in upgrage to 4.1.17 from 4.0.92
    vinothchandar Newbie
    Currently Being Moderated
    Quick question: is there a way to turn on keyPrefixing for all envs/Databases without a code change? (Guess not. Still trying to see if there are some 'backdoors')
  • 29. Re: Latency increase in upgrage to 4.1.17 from 4.0.92
    vinothchandar Newbie
    Currently Being Moderated
    never mind. Found it .. "je.forceKeyPrefixing".

    Thanks
    Vinoth

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points