Hi BDB exports,
I met a problem that put db failed for "Cannot allocate memory". Seen from shell cmd "ps", the app memory consumption was not high, it seemed just BDB can't allocate memory. Here are my questions:
1) if use db or cursor get with DBT flag 0, where BDB will allocate memory from? (DB cache?)
2) i had set cache 16m by db_env->set_cachesize, there are about 10 databases in the environment, does it mean all 10 db share 16m cache or each has 16m cache?
3) if there is no memory leak in the app, what would be possible reasons for "Cannot allocate memory"? How could I debug this?
4) one database in the app has about 100k records each about 500 bytes data, this database needs to be traversed many times by cursor. If use DBT flag 0, will the memory allocated by cursor be freed immediately after close cursor? How much memory would be allocated during traversing the database?
5) is cache multi-thread safe? if one thread changes the memory allocated by BDB using cusror get with DBT flag 0, will other threads read dirty data possibly?
The error of 'Cannot allocate memory' can be caused by using a lot of resources while you don't allocate spaces for them in advance, the resources include locks, mutexes, etc.
1) The memory will be allocated from heap if you haven't set any alloc functions.
2) All the databases in an environment share the same cache(16m here).
3) As mentioned, the error can be from failing to allocating lock, mutex, and other resources. You need to get the exact location of where the error happens.
4) I think you have asked the question about the DBT flag 0. The memory will be allocated from heap by default, and be related to the cursor. It is transient, which means that it can be overwritten by next DBC->get, and freed until the cursor closes. More information can be found from here:
From the simple description about the program, the error seems to be caused by not setting enough locks and lock objects. Generally Berkeley DB uses page lock, and if you enable lock subsystem, by default every page will be locked until the cursor closes. If your database has many pages, then many locks and lock objects are required, so you need to set the number before opening environment, since the default number(1000) may be not enough in this case.
To avoid setting such a big number of locks and lock objects, you can try the DB->set_lk_exclusive, since it locks on the database instead of pages.
Another way is to re-structure your program to divide so much operations into many cursors/transactions, and start the next curor/transaction after previous finishes. And if you are just using the cursors to do so many read operations, you can also evaluate using an isolation level like DB_READ_COMMITTED instead of the default level, and this can avoid dividing.
This thread may help you on this topic:
5 The data from DBC->get with flag 0 has no business with the cache. Cache is thread-safe to provide read/write atomicity. If you care about your program's consistence in multi-threads environment, you need to turn on lock and transaction subsystems.
Winter, Oracle Berkeley DB