7 Replies Latest reply: Oct 5, 2012 10:31 AM by Branchbird - Pat RSS

    High Memory utilization for a single project

    SunilN
      Hi All,
      I am facing one issue with one of the projects configured. We have endeca installed on a Windows server and there are multiple projects setup. I am observing that dgraph.exe for a single project is taking as much as 80% of memory on server. I do see that the project is receiving around 300 to 350 requests every day which is slightly higher than other projects. I am required to manually restart the project before business starts and that releases the memory. I am not able to identify whats causing this issue. Any ideas on how to troubleshoot the issue?

      Any suggestions will be of great help..

      Thank You,
      Sunil
        • 1. Re: High Memory utilization for a single project
          sabdelhalim
          Hi Sunil,
          did you check what was the size of the index the Dgraph loads in memory (either the dgidx_output directory on ITL or dgraph_input on your MDEX ???) that should give you a hint about the minimum size for the dgraph, besides did you check in your appconfig for the running project how many threads does the dgraph run ? (dgraph default section), I suppose you can try and monitor memory usage by running sysload or other tools ...
          hope that helps
          regards
          Saleh
          • 2. Re: High Memory utilization for a single project
            Pravin Chikhale
            Try identifying the queries that are taking too much of time. You can get these from Dgraph.reqlog or from http://<host>:<dgraph_port>/admin?op=stats page.
            Refer Details -> Most Expensive Queries

            Also refer Details ->Hotspots to get information like which type of queries are overloading server.

            After getting expensive queries, all you need to do is optimize it by removing unnecessary parameters.

            I am assuming hardware configuration is same as recommended by Endeca (MDEXInstallGuide.pdf)

            Ideal dgraph setup is you should have separate server for serving response for each project.

            HTH

            -- Pravin.
            • 3. Re: High Memory utilization for a single project
              Pravin Chikhale
              Duplicate post.

              Edited by: Pravin Chikhale on Oct 4, 2012 7:13 PM
              • 4. Re: High Memory utilization for a single project
                Branchbird - Pat
                The three factors that have the most impact on a dgraph's memory footprint are:

                Size of the index on disk (i.e. how big is it)
                Size of the cache memory (set via the cmem flag)
                Are you running a lot of partial updates?

                If you could tell us how much memory you have, how big your index is and what your cmem is set to (or if you just left it as default), we can estimate how much memory that dgraph "should" be using.
                • 5. Re: High Memory utilization for a single project
                  SunilN
                  Hi Branchbird ,

                  Please find my comments below in bold

                  Size of the index on disk (i.e. how big is it) : *511 MB*
                  Size of the cache memory (set via the cmem flag): it is default
                  Are you running a lot of partial updates? No i am not running partial update, only running full update once in a day

                  I have 16 GB REM, the dgraph is taking more then 9 GB for this project.
                  We have enabled the wildcard search for typeAhead feature.

                  Any suggestions will be of great help..

                  Thanks,
                  Sunil
                  • 6. Re: High Memory utilization for a single project
                    sabdelhalim
                    Hi,
                    complementary question : Did you check in your appconfig.xml at the default section for Dgraphs what is the default n° of threads (starting from MDEX version 6.1.4 default n° of threads is 2) ??
                    what MDEX release are you running ?? I fail to understand how a 511Mb index can be turned into 9 Gb !!!
                    regards
                    Saleh
                    • 7. Re: High Memory utilization for a single project
                      Branchbird - Pat
                      The number of threads will not have any impact on memory usage, however, I agree with Saleh that this doesn't make any sense.

                      I would suggest the following:

                      Grab the stats page XML (http://[your server name]:[your port]/admin?op=stats).
                      Restart the dgraph and see how much memory is taken up on the initial load. Then, assuming the memory consumption is reasonable, issue a couple N=0, N=something queries to the index and track what's happening with how much memory is being taken up.

                      If you're still at a normal level, it seems like there must be some process or "bad query" that causes the memory to spike. A possible culprit might be found on the "most expensive queries" section of the stats page but it's possible it won't be there as that only tracks queries that have completed, not queries that are still in process or taking forever.

                      The only other explanation I can think of is a rogue "bulk export" query where a user is trying to (for example) export the entire index to excel or something like that. Does your application offer that functionality?

                      Regards,

                      Patrick Rafferty
                      Branchbird