5 Replies Latest reply: Dec 19, 2013 1:52 AM by aditya_kar RSS

    E-commerce MDEX Engine Performance Issues

    AndrewBenedict

      Hi,

      We have migrated an existing e-commerce site to Endeca E commerce.

      Configuration:

      RAM :24 GB

      Processor:Quad Core

      MDEX Engines: 2 (each with 2 individual dgraphs running on 2 different ports)

      No of threads: 8 threads for each dgraph

      Avg RSS: around 350 Mb

       

      Below are the issues/queries we have

       

      Most Expensive Query: around 10800 ms(Details Tab)

      HTTP Total Request time:around 14000 ms(Details Tab)- Sometimes this can reach 100000 ms

      Qns:

      1.why is the TRT so huge inspite of having a high configuration and optimal MDEX setting?

      2.Is there a  relation between most Exp Query and HTTP TRT?

      3.Is there a relation between  Total Requests Received on  Performance Tab and HTTP Count of  Server on Details Tab?

      4.Is there a way to improve the performance of this system and reduce HTTP request time?

       


        • 1. Re: E-commerce MDEX Engine Performance Issues
          aditya_kar

          Hi,

           

          At a first glance, I do see that the number of threads assigned to each Dgraph is too high. Since your MDEX servers are using a quad-core processor, it has a maximum of four threads available. And you have two instances of Dgraphs running on each MDEX server with 8 threads each, which means that the Dgraphs will try to use 16 threads from a total of only four threads available. This will cause CPU contention and requests will queue up.

           

          Also, please make sure that there are no other processes trying to utilize the CPU.

           

          I would suggest using one Dgraph per MDEX server using 4 threads. If there are other processes also running on the MDEX server which would likely contest for the CPU, then reduce the number of threads to 3.

           

          When you run the performance tests, please keep an eye on the CPU utilization and load average.

           

          How much cache memory have you assigned to each Dgraph? And what is the amount of RAM on the MDEX servers?

           

          Thanks.

           

          -Adi

          • 2. Re: E-commerce MDEX Engine Performance Issues
            Michael Peel-Oracle

            AdityaKar's point on threading is valid, although it is hard to imagine a server build with 24Gb RAM but only a single quad core processor. 

             

            I'd suggest reading the Performance Tuning Guide, then setting up a load test and running the MDEX Request Analyzer afterwards - that will give you a clearer picture with min/max/average engine time, min/max/average response time, details on any queuing, request profiling etc.

             

            Michael

            • 3. Re: E-commerce MDEX Engine Performance Issues
              AndrewBenedict

              The amount of Cache is 2 GB

              • 4. Re: E-commerce MDEX Engine Performance Issues
                AndrewBenedict

                I performed a load test on the MDEX engines  using eneperf tool ,the input for this was filtered from the DGraph logs using the reqlogparser and then used them to perform load tests.I was able to notice improvement in the overall throughput after changing the no of threads to 8 from 4 hence decided to go with 8,but I would change it as you have suggested . However before that let me explain the scenario in a detailed manner

                These MDEX engines communicates to an ATG Server via a Load Balancer (f5 ) which balances the load between the two MDEX Engines.

                The reported issue is that the HTTP response time under the Server subsection of details Tab shows a Max response time of around 14000 -16000 ms ,but the average is very low ,my assumption is that the MDEX engine is not responsible for the slow HTTP response since the load test with eneperf gave positive results-the throughput was very good infact.

                Could the issue be due to network problems or issues with MDEX<-> F5 Load Balancer<-> ATG communication?

                Please correct me if I'm wrong

                • 5. Re: E-commerce MDEX Engine Performance Issues
                  aditya_kar

                  There are a couple of tests that you can run to identify if the root cause is the MDEX server's capacity or network latency.

                   

                  Before you start, please verify the number of cores on your MDEX server. Use only one Dgraph on each MDEX server. And set the Dgraph to use no more than the number of cores available on the server.

                   

                  Use the request log analyzer (available with Platform Services) to analyse the Dgraph logs.

                  1. Clear out the Dgraph logs. You can use "admin?op=logroll" to quickly do this.
                  2. Run the load test as before from the front-end.
                  3. Capture the Dgraph1.reqlog from Dgraph1.
                  4. Run the reqloganalyzer on this log file.
                  5. Compare the MDEX Engine Only Processing time against the Total Round Trip Response time in the log analysis results. If the difference between the two metrics is very high, then there maybe a network issue.

                   

                  Next, you can take out network latency from the equation and run the test again.

                  1. Parse the reqlog captured in the previous test using the Request Log Parser.
                  2. Clear out the Dgraph1 logs.
                  3. Use Eneperf on the MDEX server and the parsed log file to run a load test against Dgraph1.
                  4. Capture the Dgraph1.reqlog and run the reqloganalyzer on this log file.
                  5. Check the MDEX Engine Only Processing time and the Total Round Trip Response time again in the log analysis results.

                   

                  I hope this helps identify the root cause. Let us know how it goes.

                   

                  -Adi