Forum Stats

  • 3,759,185 Users
  • 2,251,510 Discussions
  • 7,870,528 Comments

Discussions

Share cleaner threads for multiple environments

Nitin.Patel-Oracle
Nitin.Patel-Oracle Member Posts: 5
edited Mar 20, 2019 12:22PM in Berkeley DB Java Edition

Hi,

I believe that each open environment will have its own set of cleaner threads which perform compaction. Our application (Oracle Unified Directory) opens a large number of environments (for isolation etc), resulting in too many cleaner threads, and we wanted to see if this can be avoided. Hence the question - Is there a way to share these cleaner threads across all the environments (similar to shared cache)?

Any help is highly appreciated.

Thanks

Nitin

Best Answer

  • Greybird-Oracle
    Greybird-Oracle Member Posts: 2,690
    edited Jan 29, 2019 9:19AM Accepted Answer

    Hi Nitin,

    There is no built-in way to share cleaner, checkpointer, evictor and IN compressor threads among JE Environments in a single JVM process and I don't recommend large numbers of JE Environments per JVM process. The only way to share threads is to disable the JE built-in threads (by configuring EnvironmentConfig.ENV_RUN_CLEANER, etc, to false) and use a thread in your application to periodically call Environment.cleanLog(), checkpoint(), evict() and compress().

    --mark

«1

Answers

  • Greybird-Oracle
    Greybird-Oracle Member Posts: 2,690
    edited Jan 29, 2019 9:19AM Accepted Answer

    Hi Nitin,

    There is no built-in way to share cleaner, checkpointer, evictor and IN compressor threads among JE Environments in a single JVM process and I don't recommend large numbers of JE Environments per JVM process. The only way to share threads is to disable the JE built-in threads (by configuring EnvironmentConfig.ENV_RUN_CLEANER, etc, to false) and use a thread in your application to periodically call Environment.cleanLog(), checkpoint(), evict() and compress().

    --mark

  • Nitin.Patel-Oracle
    Nitin.Patel-Oracle Member Posts: 5
    edited Jan 30, 2019 3:19AM

    Hi Mark,

    Thanks for the reponse. Few follow-up questions:

    1. What is the maximum number of JE environments per JVM that you would recommend? I know this might be subjective, but if you have any number, that would really help.

    2. We can explore the approach that you are suggesting, i.e., have common cleaner/compressor threads at application level.

    3. Currently, to be able to scale to say 100 tenants, we were thinking of isolating them using different JE environments, each having the same tables/databases. Do you have any better approach in mind which we should follow instead?

    Thanks

    Nitin

  • Greybird-Oracle
    Greybird-Oracle Member Posts: 2,690
    edited Jan 30, 2019 9:33AM

    I don't have any numbers on maximum JE envs per JVM, but I can tell you that in NoSQL DB which is a sharded, multi-tenant server product using JE, we use one env per JVM process. In addition to the issue with threads, multiple envs don't perform as well sharing a single disk, or a spinning disk anyway since head will be moving during writes from multiple envs. There is also the issue of replication. I know you're not using JE HA but you may want to consider resource usage for LDAP replication with various approaches.

    There are many ways to separate tenant data. NoSQL DB uses logical tables for each tenant, using a key prefix for the table ID. Another approach is to use separate JE Databases for each tenant; this is probably the simplest approach.

    --mark

  • Nitin.Patel-Oracle
    Nitin.Patel-Oracle Member Posts: 5
    edited Feb 1, 2019 6:27AM

    Hi Mark,

    Thanks for providing insights, based on that I think we would park the idea of using one database env per tenant.

    Today, OUD defines a set of JE Databases, and we were planning to utilize the same JE databases for each tenant in MT setup, in different JE Environments. That way, we could easily move tenants around from instance to another by just moving db files. Not sure if we can easily move tenants around with a shared JE Environment, because all data pertaining to all tenants may end up into a single jdb file?

    Regarding the approaches, just to make sure I have got it right (due to terminology):

    "NoSQL DB uses logical tables for each tenant, using a key prefix for the table ID" - If there is a NOSQL "table" called EMPLOYEES, you mean NOSQL internally creates/opens different JE Databases such as TENANT1_EMPLOYEES, TENANT2_EMPLOYEES etc?

    "separate JE Databases for each tenant" - By separate databases, do you mean separate JVMs by any chance?

    Thanks

    Nitin

  • Greybird-Oracle
    Greybird-Oracle Member Posts: 2,690
    edited Feb 1, 2019 9:38AM

    Yes, with a single env per JVM (which I recommend), moving tenant data to different machines will require copying the individual records rather than just copying jdb files.

    No, NoSQL DB stores the data for multiple tables in a single JE Database, not a JE Database per table. The key prefix of the record identifies the table.

    By separate JE Databases, I do not mean JVMs or envs, I mean separate Databases in a single env.  However, if you have too many Databases per env with lots of small tenants, you may run into performance problems due to the per-Database overhead in JE. You will need to consider the maximum number of tenants per JVM and do tests to measure the overhead.

    I'm on vacation right now, so I may not reply back quickly.

    --mark

  • Nitin.Patel-Oracle
    Nitin.Patel-Oracle Member Posts: 5
    edited Mar 12, 2019 5:44AM

    Hi Mark,

    We have tried the approach where we have a single JE env to store the data for all the tenants, and would like to know your thoughts on the same.

    In this approach, we have used separate set of tables for each tenant, so we have around 3500 JE Databases overall, and the total (file-system) size of the entire data is around 10GB. We have compared the performance of this Multi-tenant system against a vanilla OUD instance which has only 35 JE Databases, with similar data size of around 10GB. And, we have found that the performance of both these systems is comparable, i.e., the application throughputs are almost same. So, it seems like there is not much overhead added by more number of JE Databases.

    Also note that, this multi-tenant solution/design is primarily meant for free/trial customers. So, we will have control on the number of tenants can be supported on a JVM and the data each tenant can load and query.

    So, do you think this approach can work for us?

    PS: Based on the below architecture whitepaper, with increasing number of JE databases, the overhead seems to be in looking up the Mapping tree to identify the B+Tree given the Database name. Request you to clarify/correct if there could be any other overhead with more databases:

    https://www.oracle.com/technetwork/database/database-technologies/berkeleydb/learnmore/bdb-je-architecture-whitepaper-36…

    Thanks

    Nitin

  • Greybird-Oracle
    Greybird-Oracle Member Posts: 2,690
    edited Mar 12, 2019 3:52PM

    Nitin,

    This approach sounds good to me. But when comparing the single and multi-tenant runs, in addition to disk space and throughput, also please look at the cache size: the CacheTotalBytes stat. The per-database overhead not mentioned in the whitepaper is that disk utilization information is stored in memory. There is one map entry per-database and per-file in which the database's records appear. The map entry contains a fairly small object, but if the databases are spread across all files and there are a large number of files, this memory overhead is something to account for. Your test should show this difference.

    --mark

  • Greybird-Oracle
    Greybird-Oracle Member Posts: 2,690
    edited Mar 12, 2019 4:51PM

    Nitin,

    To ensure that you are testing the worst case scenario, alternate writes among the entire set of Databases/tenants. This way the records for all Databases are spread across all files.

    --mark

  • pranjal_ranjan-Oracle
    pranjal_ranjan-Oracle Member Posts: 1
    edited Mar 18, 2019 1:01AM

    Mark,

               I am collaborating on the same thing. We tried to alternate writes among the entire set of tenants, so that each tenant's records are spread across the whole set of files. For configuration purpose, we kept the db-local-backend-workflow-element cache size to only 1% and JVM heap size to 2GB to check the performance while relying on the filesystem. We checked that the CacheTotalBytes stat was around 20mb. The performance numbers in this test look stressed. etime parameter for a search result returning only one entry as a result , was around 200-300 as well, which is almost 100 times the etime in a different OUD setup with a significantly lesser number of tables. Is this sort of a result expected considering the tuning? Also, does the cache ( which is set to 1% ) also contain the disk utilization ratio map you mentioned earlier?

    -Pranjal

  • Greybird-Oracle
    Greybird-Oracle Member Posts: 2,690
    edited Mar 18, 2019 9:30AM

    I don't understand the purpose of this test. I thought you were trying to compare the use of many databases (multi-tenant) to your normal product.

    Configuring a tiny JE cache and looking at latency doesn't seem meaningful to me. Setting the JE cache size to 20 MB in a 2 GB heap is not making use of the memory in the heap and 20 MB is ridiculously low.

    I suggest comparing realistic configurations with the normal and multi-tenant approaches, and look at memory and disk usage to see the overhead added by having many JE databases. I thought that's what Nitin was doing. It seems to me that you are not in sync with Nitin. This makes it extremely difficult for me to support you.

    Also, to measure memory usage you will need to look at the eviction stats as well, when comparing runs. If the cache is too small to hold the data set, eviction may occur. To compare memory usage in the two approaches by looking at CacheTotalBytes, there must be no eviction. you need to set the JE cache to as large a value as possible -- large enough so that no eviction occurs.

    --mark