I want to create some database in a DatabaseEnvironment which will be store a large data, such as 10TB. But I always get an DatabaseException with message "Not enough space", of course there are enough space to store these data in the driver. It throws a database exception while put data and the database size reach a size(or 125% size) specified by CacheInfo, even though I close and then reopen the database environment and database. In addition, it will throw an exception when I give the param "gbytes" a value larger than 1 in CacheInfo, such as "new CacheInfo(2, 0, 1)". The OS is windows 8. Why?
static void Main(string args)
string Dir = @"D:\Temp2\";
string Name = "TestEnv";
DatabaseEnvironmentConfig envConfig = new DatabaseEnvironmentConfig();
envConfig.Create = true;
envConfig.CreationDir = Dir;
envConfig.MPoolSystemCfg = new MPoolConfig();
//envConfig.MPoolSystemCfg.CacheSize = new CacheInfo(2, 0, 1);
envConfig.MPoolSystemCfg.CacheSize = new CacheInfo((uint)0, (uint)64 * 1024 * 1024, 1);
envConfig.MPoolSystemCfg.CacheSize.Gigabytes.ToString() + "GB and " +
envConfig.MPoolSystemCfg.CacheSize.Bytes.ToString() + "Bytes");
envConfig.UseMPool = true;
envConfig.ErrorPrefix = Name;
DatabaseEnvironment Env = DatabaseEnvironment.Open(Dir, envConfig);
HashDatabaseConfig cfg = new HashDatabaseConfig();
cfg.Env = Env;
cfg.Creation = CreatePolicy.IF_NEEDED;
HashDatabase database1 = HashDatabase.Open(null, "db1", cfg);
HashDatabase database2 = HashDatabase.Open(null, "db2", cfg);
Stopwatch sw = new Stopwatch();
for (int i = 0; i < 100000000; i++)
DatabaseEntry dbKey = new DatabaseEntry(BitConverter.GetBytes(i));
DatabaseEntry dbValue = new DatabaseEntry(Encoding.UTF8.GetBytes(new string('A', 10 * 1024) + i.ToString()));
if (i % 100 == 0)
Console.WriteLine(i.ToString() + ":" + sw.Elapsed.TotalSeconds.ToString());
Console.WriteLine("TimeSpan:" + sw.Elapsed.TotalSeconds.ToString());
Here is the message of exception:
Message=Not enough space
在 BerkeleyDB.DatabaseException.ThrowException(Int32 err)
在 BerkeleyDB.Internal.DB.put(DB_TXN txn, DatabaseEntry key, DatabaseEntry data, UInt32 flags)
在 BerkeleyDB.Database.Put(DatabaseEntry key, DatabaseEntry data, Transaction txn, UInt32 flags)
在 BerkeleyDB.Database.Put(DatabaseEntry key, DatabaseEntry data, Transaction txn)
在 BerkeleyDB.Database.Put(DatabaseEntry key, DatabaseEntry data)
在 TestDatabaseEnvironment.Program.Main(String args) 位置 D:\Mzsoft\Sooule\Test\TestDatabaseEnvironment\Program.cs:行号 50
在 System.AppDomain._nExecuteAssembly(Assembly assembly, String args)
在 System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String args)
在 System.Threading.ThreadHelper.ThreadStart_Context(Object state)
在 System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
It seems you are running out of memory while inserting records into the database. You are inserting around 200 millions records totally into 2 databases, and the size of each record is around 10KB. You are using a hash database. So even though you insert the records with sorted keys, it is very likely that record (1, AAA1) and (2, AAA2) reside in different pages.
I suggest you to try setting a bigger cache or other access method such as BTree. I see you try to resize the cache after environment creation. You should check if the cache is resized successfully to the expected size you want to change to. The correct way to check that is calling DatabaseEnvironment.CacheSize. DatabaseEnvironmentConfig.MPoolSystemCfg.CacheSize only returns the cache size you pass to configure the environment during its creation.
I am trying to reproduce the issue that sets the cache size as 2GB. May I know which platform you build BDB libraries on? Windows 32 bit or 64 bit?
The problem have been found in the method HashDatabase.Open(). The parameter of filename must be specified like below, or it will throw the DatabaseException with message "Not enough space" while the total size (in bytes) of dababases reach the size of CacheInfo:
HashDatabase database1 = HashDatabase.Open("filename.db", "db1", cfg);
HashDatabase database2 = HashDatabase.Open("filename.db", "db2", cfg);
This may be a bug. I set the parameter of filename to "null" guide by Berkeley DB documentation.
I have run your program on both 32 and 64 bit Windows, but there is no error of "Not enough space". The program is able to inserts the records until it fills up the disk. I have tried running it with CacheInfo(0, 64 * 1024 * 1024, 1) and CacheInfo(1, 64 * 1024 * 1024, 1).
For the cachesize problem, I can reproduce the failure when the program requests a single 2GB cache. But this is normal.
32-bit Windows hardly ever has a 2GB contiguous free region in its address map. There are .DLLs and data segments mapped scattered all over the address space, include just below 2GB. Since the most data that a 32-bit application can possibly have is 3 GB (and even that is unusual), there is no single 2GB space for a mapping.
This is not a BDB bug. If you really need 2GB cache, you could try several smaller regions to get closer to 2GB.
Moreover, if you pass the null db file name to HashDatabase.Open(). The database will be created in-memory. That means the database stays inside the environment cache only. So if the database grows bigger than the cache, it will return the "Not enough space" error.
If there is any question, please let me know.
According to your last post, there is no bug in BDB indicated by your application.
In the method HashDatabase.Open(), when the parameter Filename is set null, the database will be created in-memory. So when the database can't be fit in the environment cache, it will return the error "Not enough space".
You could either create a bigger cache for the in-memory database, or create the database file on disk by passing a non-null Filename to the database open method.
It is right that if "when the parameter Filename is set null, the database will be created in-memory", but the database will be store in a file named "__db.003" in default. I can read data from the same database by another application after I close the application with the databaseEnvironment above. In other words, the database in disk, not in-memory, and size of the database is size of databaseEnvironment's CacheInfo.
I suggest that DB should add a function put all data in-memory, include databaseEnvironment and database, like RamDirectory in Lucene.