8 Replies Latest reply: Mar 9, 2012 12:39 PM by vinothchandar RSS

    Latency increase in je 4.0.117

    vinothchandar
      Hi,

      We are on je 4.0.92 and we would like to move forward to a release with Evict_LN. (We cannot do 5.0 due to data compatibility issues). We evaluated je 4.0.117 and found the 99th percentile latency to shoot up. Is this known? Is there a later release we can try, in case some fixes went in for similar issues? 4.1.117??

      Please let me know

      Thanks
      Vinoth
        • 1. Re: Latency increase in je 4.0.117
          Greybird-Oracle
          No, nothing comes to mind. I suggest reading the change logs and experimenting with later versions. Anything that was improved or fixed will be in the change logs.

          What data compatibility issues do you have with JE 5.0?

          --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
          • 2. Re: Latency increase in je 4.0.117
            vinothchandar
            Move to BDB5 requires we run a "prepare" script on the existing data, before bringing the environment up, I think. We cannot undertake this exercise now, due to the amount of data and operational complexity.

            Our intention of upgrading is simply get evict_ln/evict_bin support. Nothing in the change logs that I could relate to.

            Thanks.
            • 3. Re: Latency increase in je 4.0.117
              Greybird-Oracle
              >
              Move to BDB5 requires we run a "prepare" script on the existing data, before bringing the environment up, I think. We cannot undertake this exercise now, due to the amount of data and operational complexity.
              >

              Do you have databases configured for duplicates? If so, how large are they and what is the data (as in key-data) byte length?

              --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
              • 4. Re: Latency increase in je 4.0.117
                vinothchandar
                If you are referring to setSortedDuplicates(), yes. its enabled. We have lots of data sets, with maximum of around 300Gb data bases. data size vary from several bytes to 200K.

                We are pushing back for now, since we were unable to bring up one of our databases, even after the "preprocess".
                Apart from Disk ordered scans, do you think bdb5 improves performance significantly?
                • 5. Re: Latency increase in je 4.0.117
                  Greybird-Oracle
                  Are you using Voldemort by chance? Just a guess, since it uses duplicates databases for all data.

                  When you say that you "were unable to bring up one of our databases", so you mean that the Environment took too long to open the first time? This is when duplicate conversion takes place. If you see DupConvert on the stack (when doing a thread dump), this is what is happening. I'm just trying to identify the problem you're talking about.

                  Yes, see the change log for the other performance improvements.

                  --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
                  • 6. Re: Latency increase in je 4.0.117
                    vinothchandar
                    To be more specific I run something like ,

                    java -jar /je-4.1.17.jar DbPreUpgrade_4_1 -h <env_dir>

                    which goes well and still get the following when I bring up the environment

                    com.sleepycat.je.EnvironmentFailureException: (JE 5.0.34) Before upgrading to JE 5.0, the following utility must be run using JE 4.1: DbPreUpgrade_4_1 using the JE old version. See the release notes. UNEXPECTED_STATE: Unexpected internal state, may have side effects

                    This is our blocker for bdb5.
                    • 7. Re: Latency increase in je 4.0.117
                      Greybird-Oracle
                      This should not happen, as long as you are performing the upgrade step correctly. If this is a JE bug, it needs to be fixed.

                      Please save the environment directory and send me email -- mark.hayes at o.com (o == oracle) -- so that we can work together to diagnose the problem. However, I will not be able to get back to you until Monday, so I hope the problem can wait until next week.

                      --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
                      • 8. Re: Latency increase in je 4.0.117
                        vinothchandar
                        yes this is voldemort.