1 2 Previous Next 24 Replies Latest reply: Jul 7, 2012 4:22 PM by 909806 Go to original post RSS
      • 15. Re: Data Bloat in je 4.1.17
        greybird
        I am sorry to disagree. Your explanation described why a DIN and DBIN must be created even if K does not have duplicates. But I still don't understand why DBIN and DINs must be continuously created for every such update and never cleaned up. In other words, N transactions like these produce N DINs/DBINs. This will just fill up the disk.
        What you're seeing is due to performing a small number of insert/update and not filling multiple log files, so no cleaning takes place.

        If I run your test program, I see double the number of DIN/DBINs as the number of records inserted/updated. This is because the DIN/DBIN is initially created for each record during the update, and is also flushed by the checkpoint since it is dirtied by the update. If I run the test again, I see the number of DIN/DBIN increase again, as you describe. So I do see the behavior you describe, running your test without modifications.

        But if I change your test to insert/update 100,000 records instead of 100, and I run it multiple times, log cleaning does do its job. The end result after several runs is around 150,000 DIN/DBIN, and is not increasing. This is correct behavior.
        However, I am working handling duplicates completely in voldemort code, transparent to JE. (there went my July 4th).
        I would like to know about the following. >

        When you say "handling duplicates completely in voldemort code, transparent to JE", do you mean you won't use JE duplicates, and instead you'll append a discriminator to the key in Voldemort, when there are multiple record versions? If so, I strongly believe that is the right course of action, as we've discussed.
        -- How can we make the DINs and DBINs go away? Would switching back to 4.0.92 auto fix the bloat we currently have?
        -- Can we get a patch to make the bloat go away?>

        Are you saying that you have this bloat issue in production, and you've switched to JE 4.1 in production?

        --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
        • 16. Re: Data Bloat in je 4.1.17
          greybird
          But if I change your test to insert/update 100,000 records instead of 100, and I run it multiple times, log cleaning does do its job. The end result after several runs is around 150,000 DIN/DBIN, and is not increasing. This is correct behavior.
          Sorry, correction: If I run it several more times I see around 200,000 DIN/DBIN. Since the log cleaner keeps the log at around 50% utilization, this is around the right number for a data set containing 100,000 records, where every record is represented by a DIN/DBIN.

          Also, I un-commented the call to cleanLog in your test, to give the cleaner a chance to run. Normally an app is longer running that this test, of course, and cleaner background threads would make the explicit call to cleanLog unnecessary.

          --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
          • 17. Re: Data Bloat in je 4.1.17
            vinothchandar
            Are you saying that you have this bloat issue in production, and you've switched to JE 4.1 in production?
            Yes. The first printLog I posted is from production. Comparing PrintLog from 4.0.92 vs 4.1.17, the DBIN and DINs increases are what we see as the difference. That's why I am sweating this out so much.

            Coming back to experiment, I ran the experiment two times with 1M records (so 4M calls tp put) with the cleanLog() at the end. I think you are right. There are around 1M DIN and DBINs with utilization at 66%

            $ java -cp ~/Documents/mycode/javatestcode/lib/je.jar com.sleepycat.je.util.DbPrintLog -h test4117 -SC
            type,total count,provisional count,total bytes,min bytes,max bytes,avg bytes,entries as % of log
            LN_TX,1011604,0,55439208,45,57,54,11.734780246198994
            MapLN,178,0,203893,842,1640,1145,0.04315789555901036
            NameLN,2,0,64,29,35,32,1.3546837389104398E-5
            DelDupLN_TX,1011604,0,54439208,44,57,53,11.52311091199424
            DupCountLN_TX,2023208,0,96891764,43,48,47,20.509015175804368
            FileSummaryLN,1260,0,11724302,29,187937,9305,2.4816751983554917
            IN,818,150,836615,44,2396,1022,0.17708574003571212
            BIN,39305,6446,44470840,46,2384,1131,9.413113094326242
            DIN,1026192,85900,93965096,72,92,91,19.889529308806
            DBIN,1025727,269565,81608056,65,94,79,17.27392287926446
            Root,65,0,40992,560,709,630,0.008676749347721367
            Commit,1011604,0,32371328,32,32,32,6.852017445083783
            CkptStart,20,0,579,28,29,28,1.2255654450455384E-4
            CkptEnd,21,0,1280,60,61,60,2.70936747782088E-4
            BINDelta,3565,0,422437,38,260,118,0.0894169585334546
            Trace,108,0,17505,47,291,162,0.003705271695254258
            FileHeader,48,0,1824,38,38,38,3.8608486558947534E-4
            key bytes,,,8993326,,,,1.9036113267063235
            data bytes,,,14016534,,,,2.9668704196383286

            Total bytes in portion of log read: 472,434,991
            Total number of entries: 7,155,329
            If I run it several more times I see around 200,000 DIN/DBIN. Since the log cleaner keeps the log at around 50% utilization, this is around the right number for a data set containing 100,000 records, >>where every record is represented by a DIN/DBIN.
            Still trying to understand. You are saying there will always be twice (half of them dead) as much DBINs and DINs are LNs in 4.1.17?
            • 18. Re: Data Bloat in je 4.1.17
              greybird
              Still trying to understand. You are saying there will always be twice (half of them dead) as much DBINs and DINs are LNs in 4.1.17?
              Let's say CLEANER_MIN_UTILIZATION is 50%. Half of the log entries will be obsolete. So there will always be -- very roughly -- around twice as many of everything, as shown by DbPrintLog, compared to what you would see if you do a query. DbPrintLog shows obsolete and active entries -- it doesn't distinguish.

              I'm only pointing this out to help interpret the DbPrintLog output. In other words, if your actual data set (active data) is 1 M records, and utilization is 50%, DbPrintLog will show log entries for around 2 M records, because the log has 50% obsolete information.

              This is a very rough estimate. Some obsolete log entries will be more prevalent than others, and this is expected. If you see around 1 M (rather than 2 M) DIN/DBIN for 1 M records, that's fine. It just means that almost all of the DIN/DBIN in the log are currently active, i.e., very few are obsolete.

              --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
              • 19. Re: Data Bloat in je 4.1.17
                vinothchandar
                Hi Mark,

                Thanks. So, I take it that the bloat is expected and bounded (by #entries in the database). Correct me if I am wrong.

                Given this, we could have a 2X bloat in cases where the databases have millions of small values (< 100bytes say), since the DBIN and DINs are almost as big as LNs. I think this is what we are seeing. The bloat is alarming at the scale of records we have (several hundred millions)

                Technically, when the transaction is committed and we detect v1 is gone and v2 is the only value for key K, can we not link the BIN directly to the LN and throw away the DINand DBIN?
                If not, for general release, can we get a patch implementing this? If not, If I make the changes and send you a code review, will you be able to help us out?

                Thanks
                Vinoth
                • 20. Re: Data Bloat in je 4.1.17
                  greybird
                  Yes, I believe that the large amount of disk usage is expected/normal, based on the way duplicates work in JE 4.1, and the way your update transaction works.

                  A patch that collapses the DIN/DBIN back into a single BIN slot is possible, but quite a lot of work. We will strongly resist doing this because it is not applicable to our current release, JE 5. We are focused on improving JE 5 and hope to only do critical bug fixes for JE 4.1. In JE 5 the duplicate storage architecture has been reworked, partially because (as you are now painfully aware), the performance of duplicates in JE 4.1 is poor. It would be very wasteful of engineering resources to go back to JE 4.1 and try to improve duplicates at this point in time. Doing this would have to be negotiated with sales, and we would have to put aside other work.

                  No, I'm sorry, but if you were to try to do this on your own, I couldn't help you (without a sales contract) because I would have to put aside other work in order to do that. This is not a simple change even for us, in other words, I don't believe you (or anyone) can do it without a large amount of help. It really makes more sense for us to do it, if it has to be done.

                  From a purely technical viewpoint (forgetting schedules and sales contracts), I strongly believe it is the wrong thing to do. The right thing is to append the version to the key, and not use duplicate databases, whether you're using JE 4.1 or JE 5. Using duplicates for this use case -- where 80% of the keys have one record and the other 20% have two -- is just not going to perform well.

                  I'm trying to be as honest as possible so that you know what to expect. I'm sorry to have to say no.

                  --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
                  • 21. Re: Data Bloat in je 4.1.17
                    vinothchandar
                    Alright. Thanks for helping out in the investigation.

                    Thanks
                    Vinoth
                    • 22. Re: Data Bloat in je 4.1.17
                      909806
                      Just to clarify my understanding of one point: With CLEANER_MIN_UTILIZATION at 50% (the default), you'll basically always end up with 50% "wasted" space no matter what (in a heavily used environment), independent of the issue at hand. So the issue being discussed in this thread will end up putting pressure on the cleaning system (by creating "junk" faster that would otherwise happen), but won't actually increase the total space used (assuming the cleaning can keep up). Correct?
                      • 23. Re: Data Bloat in je 4.1.17
                        greybird
                        That's correct. This cleaner behavior is documented in the Getting Started Guide, and the only reason I specifically mentioned it here is to clarify what is shown by DbPrintLog.

                        --mark                                                                                                                                                                                                                                                                                                                                                                                       
                        • 24. Re: Data Bloat in je 4.1.17
                          909806
                          Yep, I was just underscoring that the apparent bloat wouldn't actually cause more space to be consumed in the long run (in case someone skims this thread later and missses that point). But at the same time, it does have the potential to contribute to a cleaner backlog, by giving the cleaner more work to do, so it's not completely without consequence. (Though relative to other sources of cleaner work, I don't know if this is significant.)
                          1 2 Previous Next