This discussion is archived
8 Replies Latest reply: Oct 14, 2013 12:52 AM by Oracle,CindyZeng RSS

"Not enough space" DatabaseException while using DatabaseEnvironment

3c325edf-a6cb-4015-beeb-de7a8b3d3be8 Newbie
Currently Being Moderated

I want to create some database in a DatabaseEnvironment which will be store a large data, such as 10TB. But I always get an DatabaseException with message "Not enough space",  of course there are enough space to store these data in the driver. It throws a database exception while put data and the database size reach a size(or 125% size) specified by CacheInfo, even though I close and then reopen the database environment and database. In addition,  it will throw an exception when I give the param "gbytes" a value larger than 1 in CacheInfo,  such as  "new CacheInfo(2, 0, 1)". The OS is windows 8. Why?

 

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
using System.Diagnostics;
using BerkeleyDB;

namespace TestDatabaseEnvironment
{
    class Program
    {
        static void Main(string[] args)
        {
            string Dir = @"D:\Temp2\";
            string Name = "TestEnv";

            if (Directory.Exists(Dir))
                Directory.Delete(Dir, true);

            DatabaseEnvironmentConfig envConfig = new DatabaseEnvironmentConfig();
            envConfig.Create = true;
            envConfig.DataDirs.Add(Dir);
            envConfig.CreationDir = Dir;
            envConfig.MPoolSystemCfg = new MPoolConfig();
            //envConfig.MPoolSystemCfg.CacheSize = new CacheInfo(2, 0, 1);
            envConfig.MPoolSystemCfg.CacheSize = new CacheInfo((uint)0, (uint)64 * 1024 * 1024, 1);
            Console.WriteLine("CacheSize:" +
                envConfig.MPoolSystemCfg.CacheSize.Gigabytes.ToString() + "GB and " +
                envConfig.MPoolSystemCfg.CacheSize.Bytes.ToString() + "Bytes");
            envConfig.UseMPool = true;
            envConfig.ErrorPrefix = Name;

            if (!Directory.Exists(Dir))
                Directory.CreateDirectory(Dir);

            DatabaseEnvironment Env = DatabaseEnvironment.Open(Dir, envConfig);
            HashDatabaseConfig cfg = new HashDatabaseConfig();
            cfg.Env = Env;
            cfg.Creation = CreatePolicy.IF_NEEDED;
            HashDatabase database1 = HashDatabase.Open(null, "db1", cfg);
            HashDatabase database2 = HashDatabase.Open(null, "db2", cfg);
            Stopwatch sw = new Stopwatch();
            sw.Start();
            for (int i = 0; i < 100000000; i++)
            {
                DatabaseEntry dbKey = new DatabaseEntry(BitConverter.GetBytes(i));
                DatabaseEntry dbValue = new DatabaseEntry(Encoding.UTF8.GetBytes(new string('A', 10 * 1024) + i.ToString()));
                database1.Put(dbKey, dbValue);
                database2.Put(dbKey, dbValue);
                if (i % 100 == 0)
                {
                    Console.WriteLine(i.ToString() + ":" + sw.Elapsed.TotalSeconds.ToString());
                }
            }
            Console.WriteLine("TimeSpan:" + sw.Elapsed.TotalSeconds.ToString());
            database1.Close();
            database2.Close();
            Env.Close();
        }
    }
}

  • 1. Re: "Not enough space" DatabaseException while using DatabaseEnvironment
    3c325edf-a6cb-4015-beeb-de7a8b3d3be8 Newbie
    Currently Being Moderated

    Here is the message of exception:

     

    未处理 BerkeleyDB.DatabaseException

      Message=Not enough space

      Source=libdb_dotnet53

      StackTrace:

           在 BerkeleyDB.DatabaseException.ThrowException(Int32 err)

           在 BerkeleyDB.Internal.DB.put(DB_TXN txn, DatabaseEntry key, DatabaseEntry data, UInt32 flags)

           在 BerkeleyDB.Database.Put(DatabaseEntry key, DatabaseEntry data, Transaction txn, UInt32 flags)

           在 BerkeleyDB.Database.Put(DatabaseEntry key, DatabaseEntry data, Transaction txn)

           在 BerkeleyDB.Database.Put(DatabaseEntry key, DatabaseEntry data)

           在 TestDatabaseEnvironment.Program.Main(String[] args) 位置 D:\Mzsoft\Sooule\Test\TestDatabaseEnvironment\Program.cs:行号 50

           在 System.AppDomain._nExecuteAssembly(Assembly assembly, String[] args)

           在 System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)

           在 Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()

           在 System.Threading.ThreadHelper.ThreadStart_Context(Object state)

           在 System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)

           在 System.Threading.ThreadHelper.ThreadStart()

      InnerException:

  • 2. Re: "Not enough space" DatabaseException while using DatabaseEnvironment
    Oracle,CindyZeng Newbie
    Currently Being Moderated

    Hi,

     

    It seems you are running out of memory while inserting records into the database. You are inserting around 200 millions records totally into 2 databases, and the size of each record is around 10KB. You are using a hash database. So even though you insert the records with sorted keys, it is very likely that record (1, AAA1) and (2, AAA2) reside in different pages.

     

    I suggest you to try setting a bigger cache or other access method such as BTree. I see you try to resize the cache after environment creation. You should check if the cache is resized successfully to the expected size you want to change to. The correct way to check that is calling DatabaseEnvironment.CacheSize. DatabaseEnvironmentConfig.MPoolSystemCfg.CacheSize only returns the cache size you pass to configure the environment during its creation.

     

    I am trying to reproduce the issue that sets the cache size as 2GB. May I know which platform you build BDB libraries on? Windows 32 bit or 64 bit?

     

    Regards,

    Cindy

  • 3. Re: "Not enough space" DatabaseException while using DatabaseEnvironment
    3c325edf-a6cb-4015-beeb-de7a8b3d3be8 Newbie
    Currently Being Moderated

    The problem have been found in the method HashDatabase.Open(). The parameter of filename must be specified like below, or it will throw the DatabaseException with message "Not enough space" while the total size (in bytes) of dababases reach the size of CacheInfo:

          HashDatabase database1 = HashDatabase.Open("filename.db", "db1", cfg);

          HashDatabase database2 = HashDatabase.Open("filename.db", "db2", cfg);

     

    This may be a bug. I set the parameter of filename to "null" guide by Berkeley DB documentation.

  • 4. Re: "Not enough space" DatabaseException while using DatabaseEnvironment
    Oracle,CindyZeng Newbie
    Currently Being Moderated

    Hi,

     

    I have run your program on both 32 and 64 bit Windows, but there is no error of "Not enough space". The program is able to inserts the records until it fills up the disk. I have tried running it with CacheInfo(0, 64 * 1024 * 1024, 1) and CacheInfo(1, 64 * 1024 * 1024, 1).

     

    For the cachesize problem, I can reproduce the failure when the program requests a single 2GB cache. But this is normal.

     

    32-bit Windows hardly ever has a 2GB contiguous free region in its address map. There are .DLLs and data segments mapped scattered all over the address space, include just below 2GB. Since the most data that a 32-bit application can possibly have is 3 GB (and even that is unusual), there is no single 2GB space for a mapping.

     

    This is not a BDB bug. If you really need 2GB cache, you could try several smaller regions to get closer to 2GB.

     

    Moreover, if you pass the null db file name to HashDatabase.Open(). The database will be created in-memory. That means the database stays inside the environment cache only. So if the database grows bigger than the cache, it will return the "Not enough space" error.

     

    If there is any question, please let me know.

     

    Regards,

    Cindy Zeng

  • 5. Re: "Not enough space" DatabaseException while using DatabaseEnvironment
    Oracle,CindyZeng Newbie
    Currently Being Moderated

    Hi,

     

    According to your last post, there is no bug in BDB indicated by your application.

     

    In the method HashDatabase.Open(), when the parameter Filename is set null, the database will be created in-memory. So when the database can't be fit in the environment cache, it will return the error "Not enough space".


    You could either create a bigger cache for the in-memory database, or create the database file on disk by passing a non-null Filename to the database open method.


    Regards,

    Cindy

  • 6. Re: "Not enough space" DatabaseException while using DatabaseEnvironment
    hmz Newbie
    Currently Being Moderated

    It is right that if "when the parameter Filename is set null, the database will be created in-memory", but the database will be store in a file named "__db.003" in default. I can read data from the same database by another application after I close the application with the databaseEnvironment above. In other words, the database in disk, not in-memory, and size of the database is size of databaseEnvironment's CacheInfo.

    I suggest that DB should add a function put all data in-memory, include databaseEnvironment and database, like RamDirectory in Lucene.

  • 7. Re: "Not enough space" DatabaseException while using DatabaseEnvironment
    Oracle,CindyZeng Newbie
    Currently Being Moderated

    Hi,

     

    If you don't want to allocate the region memory by the filesystem, you can open the Environment with flag DB_PRIVATE. However, turning on this flag means allocation memory from heap and the environment regions can not be shared between processes.

     

    Regards,

    Cindy

  • 8. Re: "Not enough space" DatabaseException while using DatabaseEnvironment
    Oracle,CindyZeng Newbie
    Currently Being Moderated

    Moreover, you may open the Environment with DB_SYSTEM_MEM to allocate region memory from system shared memory.

     

    Regards,

    Cindy

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points