10 Replies Latest reply: May 28, 2013 4:28 PM by vinothchandar RSS

    Slight Disk space increase in 4.1.17 -> 5.0.73 upgrade

    vinothchandar
      Hi,

      I am upgrading our Voldemort servers from 4.1.17 to 5.0.73. First of all, the performance seems to be much better esp cleaning. But,I notice a curious pattern with the servers running BDB JE 5. They all have slightly higher disk space than ones running JE4 (about 5-6GB roughly).. The cleaners are keeping up though. Is this expected?

      (We are running with maxDelta=100 and binDelta=75.. i.e maxing out on them to reduce BIN logging). I would be happy to provide anymore info you might need.
        • 1. Re: Slight Disk space increase in 4.1.17 -> 5.0.73 upgrade
          Bogdan Coman-Oracle
          Hi,
          vinothchandar wrote:
          They all have slightly higher disk space than ones running JE4 (about 5-6GB roughly).
          How much disk space do they take with JE4 and how much with JE5?

          - Much changed internally in JE 5 and some differences should be expected.
          - Cleaner utilization is calculated differently and this could mean that X% gives a different result than before. The CLEANER_MIN_UTILIZATION setting is means as a knob you can turn rather than an exact control. You can experiment by turning it down a little.
          - The checkpoint interval changed, could have something to do with it:
          The behavior, although not the syntax or intent, of
          <code>EnvironmentConfig.CHECKPOINTER_BYTES_INTERVAL</code> has changed.
          Previously, this interval defined the byte distance between the end of one
          checkpoint and the start of the next. Now it defines the byte distance between
          the start of one checkpoint and start of the next. In other words, now the
          interval includes the checkpoint itself, which in some cases can be large.
          This now more accurately reflects the intention of the parameter, which is to
          bound the recovery interval, which is proportional to the time to recover (open
          the <code>Environment</code>) after a crash. It does mean, however, that
          checkpoints may occur more often for the same configured interval, and some
          applications may wish to adjust their configured setting accordingly. [#19704]
          Thanks,
          Bogdan
          • 2. Re: Slight Disk space increase in 4.1.17 -> 5.0.73 upgrade
            vinothchandar
            How much disk space do they take with JE4 and how much with JE5?
            We have a lot of Voldemort stores. So, no one number.. I would put it around 10%.
            Let me may be do a PrintLog and see what the difference is.. And get back to you.

            Thanks for the quick response,
            Vinoth
            • 3. Re: Slight Disk space increase in 4.1.17 -> 5.0.73 upgrade
              vinothchandar
              Here are the DbPrintLog outputs for two environment.. Looks like it might related to much of the data is migrated to the new format, depending on the amount of new record types I see (INS_LN_TX, UPD_LN_TX records)?

              Store1 : (JE5 has higher disk space)

              JE5 PrintLog
              type,total count,provisional count,total bytes,min bytes,max bytes,avg bytes,entries as % of log
              MapLN,99,0,242657,731,13418,2451,6.554211342604134E-4
              NameLN,2,0,71,34,37,35,1.9177233927926808E-7
              FileSummaryLN,14325,0,12337224,24,18092,861,0.03332307474214548
              IN,375,0,5753829,46,48777,15343,0.01554120066398439
              BIN,6274,6211,158558941,57,64243,25272,0.4282706905523021
              DbTree,13,0,4013,270,368,308,1.0839188697573279E-5
              Commit,3772575,0,128267550,34,34,34,0.34645307207211945
              CkptStart,12,0,371,29,33,30,1.002077998205753E-6
              CkptEnd,10,0,2254,221,233,225,6.088096517400989E-6
              BINDelta,44102,38378,866312424,34,49872,19643,2.3399261985517343
              Trace,1396,0,246050,49,396,176,6.645856912628719E-4
              FileHeader,589,0,22382,38,38,38,6.045420419364194E-5
              DEL_LN_TX,500346,0,58954504,40,908,117,0.1592372273680136
              INS_LN_TX,1293391,0,188509127,68,94081,145,0.5091667078913047
              UPD_LN_TX,1978838,0,24525104549,74,216611,12393,66.24277000606125
              UPD_LN,597913,0,11078750189,58,191410,18529,29.923913239931878
              key bytes,,,383534132,,,,1.0359329251701914
              data bytes,,,35347157842,,,,95.47334008780092

              Total bytes in portion of log read: 37,023,066,135
              Total number of entries: 8,210,260

              JE4 PrintLog
              type,total count,provisional count,total bytes,min bytes,max bytes,avg bytes,entries as % of log
              LN_TX,9663571,0,53691413154,48,201049,5556,91.78359239172033
              MapLN,1086,0,20079437,1090,24445,18489,0.03432509507203998
              NameLN,2,0,73,35,38,36,1.2479094609370364E-7
              FileSummaryLN,27919,0,25014647,30,16586,895,0.04276166390863048
              IN,770,0,9954710,45,44322,12928,0.01701722847929387
              BIN,147181,76668,4221776453,56,63873,28684,7.21697914747932
              Root,55,0,27881,476,542,506,4.766159408271988E-5
              Commit,9663571,0,328561414,34,34,34,0.5616642425060965
              CkptStart,28,0,865,29,31,30,1.4786872379596389E-6
              CkptEnd,28,0,1901,66,68,67,3.2496929934812412E-6
              BINDelta,72275,0,200690019,40,11182,2776,0.34307256633662137
              Trace,1681,0,277673,49,288,165,4.74672279105164E-4
              FileHeader,931,0,35378,38,38,38,6.047745330004174E-5
              key bytes,,,846886312,,,,1.4477225220313352
              data bytes,,,52517994139,,,,89.77767363766291

              Total bytes in portion of log read: 58,497,833,605
              Total number of entries: 19,579,098

              Store2 : (JE4 has higher disk space)

              JE5 PrintLog
              type,total count,provisional count,total bytes,min bytes,max bytes,avg bytes,entries as % of log
              LN_TX,28114,0,3143245010,64,1760704,111803,16.34923101134753
              LN,104523,0,3440810064,55,3488644,32919,17.896981757239946
              MapLN,244,0,1182467,557,7136,4846,0.006150467457926571
              NameLN,2,0,78,35,43,39,4.057081184661158E-7
              FileSummaryLN,24483,0,3139177,24,4092,128,0.016328071720539822
              IN,329,0,846640,44,7615,2573,0.0044037015566429786
              BIN,2163,1704,10595004,53,11492,4898,0.05510870689719194
              DbTree,38,0,42085,277,3642,1107,2.1890033545700623E-4
              Commit,58905,0,1942330,32,33,32,0.010102808330003729
              CkptStart,25,0,775,29,33,31,4.0310742539902534E-6
              CkptEnd,22,0,3717,62,233,168,1.9333552260750673E-5
              BINDelta,8497,6204,7242640,38,7520,852,0.03767176727086448
              Trace,1708,0,321087,47,388,187,0.0016700974696657659
              FileHeader,307,0,11666,38,38,38,6.0679370641355226E-5
              INS_LN_TX,1,0,3239,3239,3239,3239,1.684728968861217E-5
              UPD_LN_TX,30790,0,3528095044,67,1393067,114585,18.350984641934208
              UPD_LN,127432,0,9088163338,52,3562230,71317,47.271046771445064
              key bytes,,,3607516,,,,0.018764083701236004
              data bytes,,,19190168288,,,,99.8154752458026

              Total bytes in portion of log read: 19,225,644,361
              Total number of entries: 387,583

              JE4 PrintLog
              type,total count,provisional count,total bytes,min bytes,max bytes,avg bytes,entries as % of log
              LN_TX,58265,0,6918799041,70,1649607,118747,53.67394922792035
              LN,150174,0,5956928399,55,2771407,39666,46.21204786373891
              MapLN,335,0,1387185,2848,5715,4140,0.01076136144705416
              NameLN,2,0,78,35,43,39,6.051003960324142E-7
              FileSummaryLN,6862,0,1060960,29,2993,154,0.008230606617622437
              IN,114,0,311511,45,6675,2732,0.0024166080701083768
              BIN,1652,1449,9008093,55,11643,5452,0.06988205951021562
              Root,21,0,16483,199,3234,784,1.2787012599746518E-4
              Commit,58265,0,1922745,33,33,33,0.014916073858581336
              CkptStart,7,0,214,29,31,30,1.6601472403966236E-6
              CkptEnd,7,0,459,64,66,65,3.5607830997292065E-6
              BINDelta,1789,0,907982,40,2638,507,0.007043849587055173
              Trace,369,0,71919,47,293,194,5.57925838234041E-4
              FileHeader,206,0,7828,38,38,38,6.072725513002229E-5
              key bytes,,,2583364,,,,0.020040956147383097
              data bytes,,,12867837779,,,,99.82479148915078

              Total bytes in portion of log read: 12,890,422,897
              Total number of entries: 278,068
              • 4. Re: Slight Disk space increase in 4.1.17 -> 5.0.73 upgrade
                vinothchandar
                Hi,

                Any updates on this?

                Thanks
                Vinoth
                • 5. Re: Slight Disk space increase in 4.1.17 -> 5.0.73 upgrade
                  Bogdan Coman-Oracle
                  Hi Vinoth,
                  vinothchandar wrote:
                  Here are the DbPrintLog outputs for two environment.. Looks like it might related to much of the data is migrated to the new format, depending on the amount of new record types I see (INS_LN_TX, UPD_LN_TX records)?
                  Did you also change the nodeMaxEntries when upgrading to JE5? Did both stores (Store1 and Store2) go through the same format change?

                  Thanks,
                  Bogdan
                  • 6. Re: Slight Disk space increase in 4.1.17 -> 5.0.73 upgrade
                    vinothchandar
                    Hi Bogdan,

                    No.. We simply used the preupgrade script and bounced the servers. (No fanout changes). Only thing that changed was, we set the BIN_DELTA and MAX_DELTA to their maximums (75% and 100 respectively)
                    • 7. Re: Slight Disk space increase in 4.1.17 -> 5.0.73 upgrade
                      Bogdan Coman-Oracle
                      vinothchandar wrote:
                      Here are the DbPrintLog outputs for two environment.. Looks like it might related to much of the data is migrated to the new format, depending on the amount of new record types I see (INS_LN_TX, UPD_LN_TX records)?
                      You may be right. I need to investigate more to see if I can get to a clear conclusion. While store2 looks okay, as the size/entry is similar, for store1 it takes 33% more space in JE5. All this is happening while there are exponentially more records in store1 (so maybe more running time is needed after the upgrade?). Can you please post a new set of statistics for store1, see if any improvement occurred since last week?

                      Thanks,
                      Bogdan
                      • 8. Re: Slight Disk space increase in 4.1.17 -> 5.0.73 upgrade
                        vinothchandar
                        Hi Bogdan,

                        I was trying to get another set of DbPrintLogs for you. But our admin is totally swamped.. Instead, I just checked on the disk space for these stores again and here is how they look..

                        Store      je4 diskspace/util      je5 diskspace/util
                        store1      59GB/53%      37GB/47%
                        store2      13.08GB/43%      20.8GB/46%

                        Effectively no change, since I posted this thread.. So, I don't expect anything has changed..
                        • 9. Re: Slight Disk space increase in 4.1.17 -> 5.0.73 upgrade
                          Greybird-Oracle
                          Vinoth,

                          In my opinion, if the only symptom is slightly higher disk usage on JE 5, this is not something to be concerned about or spend too much time analyzing. As Bogdan said, the cleaner calculates utilization differently in JE 5 (due to many internal changes), so the most likely explanation is simply that when setting CLEANER_MIN_UTILIZATION to a particular value, say 50% (the default), the cleaner actually allows a lower utilization. This setting has never controlled log utilization according to the exact amount configured -- it's simply a knob you can turn to trade-off cleaner efficiency and increased disk usage. So if slightly more disk space is being used than desired, you may want to simply try raising this config setting slightly.

                          If you suspect there is a bug or undesirable behavior change in JE that caused this, the only way I would know to find the underlying cause is to write a very small standalone test that can be run with both JE 4.1 and 5.0, and tweek the settings and data access pattern to try to reproduce the problem. If you're able to do that, then we can run the program here and do more analysis.

                          Bogdan is on vacation now, but he asked me to ask you whether the data access pattern and data records (key and data size) are different in the two test cases you described.

                          --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
                          • 10. Re: Slight Disk space increase in 4.1.17 -> 5.0.73 upgrade
                            vinothchandar
                            Hi Mark,

                            No. We have not hit any perf or correctness issues so far. The performance (cleaning) is rather improved a lot.. Nor do we see unbounded disk growth.. I was just concerned due the other disk bloat we saw last year when moving with sorted duplicates.. But this, so far does not look like a bloat or anything..

                            Let me load some data into both versions from scratch and update the thread if I find any big differences..

                            Thanks
                            Vinoth