1 Reply Latest reply: Feb 29, 2012 4:27 PM by bsjudd-Oracle RSS

    running out of mutexes

    620392
      We have a system currently using BerkeleyDB 5.1.25 with fairly large databases and associated cache configurations. (Mpool = 1.5 GB, Data Files at about 4GB). During runtime, all transactions proceeded correctly for some time, and then we began receiving the error "Exception in thread "pool-2-thread-1598878" java.lang.OutOfMemoryError: Cannot allocate memory: unable to allocate memory for mutex; resize mutex region". This error then repeats through all subsequent transactions.

      When this occurred a check with "db_stat -x" showed the following:

      96MB 704KB Mutex region size
      19382 The number of region locks that required waiting (0%)
      4 Mutex alignment
      1200 Mutex test-and-set spins
      666981 Mutex total count
      1 Mutex free count
      666980 Mutex in-use count
      666981 Mutex maximum in-use count
      Mutex counts
      1 Unallocated
      5 db handle
      1 env dblist
      2 env handle
      1 env region
      243 lock region
      6 logical lock
      1 log filename
      1 log flush
      2 log region
      5 mpoolfile handle
      *404537 mpool buffer*
      5 mpool filehandle
      17 mpool file bucket
      1 mpool handle
      262147 mpool hash bucket
      1 mpool region
      1 mutex region
      1 twister
      1 txn active list
      1 transaction checkpoint
      1 txn region

      Upon stopping the service, and restarting it, another check of db_stat -x shows many of the same values with the exception of the mutexes being used for the mpool buffer, which has dropped down to about 40k, but is also growing gradually as the system runs.

      I am trying to determine how to resolve the issue with the environment running out of mutexes.

      From looking at the code, when an environment is created, the mutex region is by default computed to a size which includes some # of mutexes based (I think) on the mpool region size. If the issue here is that I need to increase that value because the computation is not right, how many mutexes do I need for the region? Is there a way that I can compute the maximum number of mutexes needed by the mpool region?

      What could be causing the gradual increase in the number of mutexes associated with the mpool buffer? It is accurate to say that the mpool region (cache) is filling with pages from the data region as the system runs, so is this just a question of having to set a higher maximum?
        • 1. Re: running out of mutexes
          bsjudd-Oracle
          Hi Bob,

          Take a look at the following document:

          Properly Sizing the Number of Berkeley DB Mutexes with DB_ENV->set_mp_pagesize() (Doc ID 1423974.1)
          Goal
          To determine when DB_ENV->set_mp_pagesize() is needed to properly size the number of mutexes required by a Berkeley DB application.

          Solution

          If the total number of mutexes required by an application is too small, the error:


          "unable to allocate memory for mutex; resize mutex region"


          is raised. The maximum size of the mutex region is inferred by the Berkeley DB library from the sizes of internal memory structures. In general the number of mutexes needed at run-time is based on several factors. For example, each active transaction requires a mutex, each environment shared memory region requires a mutex, each open database handle requires a mutex, each mpool hash bucket requires a mutex and each page in the cache requires a mutex. When an application specifies a cache size, Berkeley DB assumes a page size of 4K and allocates a number of mutexes needed in the cache for that page size. If your page size is different than 4K DB_ENV->set_mp_pagesize() is needed for Berkeley DB to allocate the proper number of mutexes. Setting DB_ENV->set_mp_pagesize() allows Berkeley DB to allocate the proper number of mutexes by setting the page size to the value actually used and using that to allocate the hash table and the number of mutexes expected to be needed by the buffer pool. If the application page size is greater than 4K then using the default 4K size to allocate mutexes wastes memory. The far worse scenario is with page sizes less than 4K because in this case insufficient mutexes are allocated.

          Note that you can programmatically call DB_ENV->mutex_set_max() or use a DB_CONFIG file with mutex_set_max to override Berkeley DB's mutex calculation, but that is not recommended.

          If the error:


          "unable to allocate memory for mutex; resize mutex region"


          is raised use db_stat -x/db_stat -e output to view the statistics for the mutex region.

          The db_stat output will show a value for the "Mutex total count" as well as "Mutex in-use count" and "Mutex maximum in-use count". There will also be a value for the "mpool hash bucket" and of specific
          interest in this case "mpool buffer" which represents the number of pages expected to be in the cache. "mpool buffer" is computed approximately as the size of the cache / page size. If the "mpool buffer" i.e. the number of pages in cache does not reflect the size of the cache / page size verify that ENV->set_mp_pagesize() is being invoked to set the page size used to allocate the hash table and the number of mutexes expected to be needed by the buffer pool.

          In addition to DB_ENV->set_mp_pagesize() for the C API, see DbEnv::set_mp_pagesize()
          for the C++ API and EnvironmentConfig.setCachePageSize() for the Java API.


          Also, if you can consider a newer release, starting in 5.2 you have the option for the enhanced memory allocation. For this you could look at:
          How Can BDB Increase the Mutex Region Size Dynamically by Using the Enhanced Memory Allocation (Doc ID 1357873.1)

          Brian