This content has been marked as final. Show 7 replies
did you check what was the size of the index the Dgraph loads in memory (either the dgidx_output directory on ITL or dgraph_input on your MDEX ???) that should give you a hint about the minimum size for the dgraph, besides did you check in your appconfig for the running project how many threads does the dgraph run ? (dgraph default section), I suppose you can try and monitor memory usage by running sysload or other tools ...
hope that helps
Try identifying the queries that are taking too much of time. You can get these from Dgraph.reqlog or from http://<host>:<dgraph_port>/admin?op=stats page.
Refer Details -> Most Expensive Queries
Also refer Details ->Hotspots to get information like which type of queries are overloading server.
After getting expensive queries, all you need to do is optimize it by removing unnecessary parameters.
I am assuming hardware configuration is same as recommended by Endeca (MDEXInstallGuide.pdf)
Ideal dgraph setup is you should have separate server for serving response for each project.
The three factors that have the most impact on a dgraph's memory footprint are:
Size of the index on disk (i.e. how big is it)
Size of the cache memory (set via the cmem flag)
Are you running a lot of partial updates?
If you could tell us how much memory you have, how big your index is and what your cmem is set to (or if you just left it as default), we can estimate how much memory that dgraph "should" be using.
Hi Branchbird ,
Please find my comments below in bold
Size of the index on disk (i.e. how big is it) : *511 MB*
Size of the cache memory (set via the cmem flag): it is default
Are you running a lot of partial updates? No i am not running partial update, only running full update once in a day
I have 16 GB REM, the dgraph is taking more then 9 GB for this project.
We have enabled the wildcard search for typeAhead feature.
Any suggestions will be of great help..
complementary question : Did you check in your appconfig.xml at the default section for Dgraphs what is the default n° of threads (starting from MDEX version 6.1.4 default n° of threads is 2) ??
what MDEX release are you running ?? I fail to understand how a 511Mb index can be turned into 9 Gb !!!
The number of threads will not have any impact on memory usage, however, I agree with Saleh that this doesn't make any sense.
I would suggest the following:
Grab the stats page XML (http://[your server name]:[your port]/admin?op=stats).
Restart the dgraph and see how much memory is taken up on the initial load. Then, assuming the memory consumption is reasonable, issue a couple N=0, N=something queries to the index and track what's happening with how much memory is being taken up.
If you're still at a normal level, it seems like there must be some process or "bad query" that causes the memory to spike. A possible culprit might be found on the "most expensive queries" section of the stats page but it's possible it won't be there as that only tracks queries that have completed, not queries that are still in process or taking forever.
The only other explanation I can think of is a rogue "bulk export" query where a user is trying to (for example) export the entire index to excel or something like that. Does your application offer that functionality?