This discussion is archived
5 Replies Latest reply: Sep 24, 2013 5:01 AM by Hanhui Liang Branched from an earlier discussion. RSS

Re: A big bug in BDB on Win7 x64

3c325edf-a6cb-4015-beeb-de7a8b3d3be8 Newbie
Currently Being Moderated


I suggest that Berkeley DB (V5.3.21 to V6.0.20) should be tested on x64, the bug of "NOT RELEASE ANY MEMORY" is very clear. Any application use BDB will take memory more and more continually until out of memory (towards 100%  watch on taskmgr.exe) under on platform of x64, and the memory size of it took is the size of databases mainly. Of course, the above application with BDB runs well on win32.

I use Berkeley DB (V6.0.20, C#) and run my application under Windows server 2008(x64) with .Net Framework 2.0 or 3.5, the bug  do not let me release my application on x64.

I hope the engineer of Oracle solve this bug as fast as possible, thanks!

  • 1. Re: A big bug in BDB on Win7 x64
    3c325edf-a6cb-4015-beeb-de7a8b3d3be8 Newbie
    Currently Being Moderated

    I wrote a test method named UseEnv() (with bold) changed from the examples (excs_env.cs), it runs well on win32, but will encounter a bug described above on x64 (windows server 2008).  All the codes of file 'excs_env.cs' changed from Berkeley DB examples below:

     

    /*-
    * See the file LICENSE for redistribution information.
    *
    * Copyright (c) 2009, 2013 Oracle and/or its affiliates.  All rights reserved.
    *
    */
    using System;
    using System.IO;
    using System.Diagnostics;
    using System.Text;
    using System.Collections;
    using System.Collections.Generic;
    using BerkeleyDB;

    namespace excs_env
    {
        class Program
        {
            const int EXIT_FAILURE = 1;
            const int EXIT_SUCCESS = 0;
            const string progName = "excs_env";

            static void Main(string[] args)
            {
                string data_dir, home;

                /*
                 * excs_env is meant to be run from build_windows\AnyCPU, in either
                 * the Debug or Release directory. The required core libraries,
                 * however, are in either build_windows\Win32 or build_windows\x64,
                 * depending upon the platform.  That location needs to be added to
                 * the PATH environment variable for the P/Invoke calls to work.
                 */
                try
                {
                    String pwd = Environment.CurrentDirectory;
                    pwd = Path.Combine(pwd, "..");
                    pwd = Path.Combine(pwd, "..");
                    if (IntPtr.Size == 4)
                        pwd = Path.Combine(pwd, "Win32");
                    else
                        pwd = Path.Combine(pwd, "x64");
    #if DEBUG
                    pwd = Path.Combine(pwd, "Debug");
    #else
                    pwd = Path.Combine(pwd, "Release");
    #endif
                    pwd += ";" + Environment.GetEnvironmentVariable("PATH");
                    Environment.SetEnvironmentVariable("PATH", pwd);
                }
                catch (Exception e)
                {
                    Console.WriteLine(
                        "Unable to set the PATH environment variable.");
                    Console.WriteLine(e.Message);
                    return;
                }

                data_dir = home = null;
                switch (args.Length)
                {
                    case 0:
                        data_dir = "envData";
                        home = "envHome";
                        break;
                    case 2:
                        home = args[0];
                        data_dir = args[1];
                        break;
                    case 1:
                        home = args[0];
                        break;
                    default:
                        usage();
                        return;
                }

                if (!Directory.Exists(home))
                {
                    Console.WriteLine("Creating home directory: {0}", home);
                    try
                    {
                        Directory.CreateDirectory(home);
                    }
                    catch
                    {
                        Console.WriteLine("Unable to create {0}", home);
                        return;
                    }
                }
                if (data_dir != null && !Directory.Exists(home + "/" + data_dir))
                {
                    Console.WriteLine(
                        "Creating home directory: {0}/{1}", home, data_dir);
                    try
                    {
                        Directory.CreateDirectory(home + "/" + data_dir);
                    }
                    catch
                    {
                        Console.WriteLine(
                            "Unable to create {0}/{1}", home, data_dir);
                        return;
                    }
                }

                /* Set up environment. */
                if (SetUpEnv(home, data_dir) == EXIT_FAILURE)
                {
                    Console.WriteLine("Fail to set up the environment.");
                    return;
                }
                Console.WriteLine("Set up the environment.");

                /* Tear down the environment and remove its files. */
                if (TearDownEnv(home) == EXIT_FAILURE)
                {
                    Console.WriteLine(
                        "Fail to tear down the environment.");
                    return;
                }
                Console.WriteLine("Tear down the environment.");
            }

            /*
             * Set up environment.
             */
            public static int SetUpEnv(string home, string data_dir)
            {
                DatabaseEnvironment env;
                DatabaseEnvironmentConfig envConfig;

                /* Configure an environment. */
                envConfig = new DatabaseEnvironmentConfig();
                envConfig.MPoolSystemCfg = new MPoolConfig();
                envConfig.MPoolSystemCfg.CacheSize = new CacheInfo(
                    0, 1024 * 1024, 1);
                envConfig.Create = true;
                envConfig.DataDirs.Add(data_dir);
                envConfig.CreationDir = data_dir;
                envConfig.ErrorPrefix = progName;
                //envConfig.UseLogging = true;
                envConfig.UseLocking = true;
                envConfig.UseMPool = true;
                envConfig.UseTxns = true;

                /* Create and open the environment. */
                try
                {
                    env = DatabaseEnvironment.Open(home, envConfig);
                }
                catch (Exception e)
                {
                    Console.WriteLine("{0}", e.Message);
                    return EXIT_FAILURE;
                }

                //Console.ReadLine();
                UseEnv(env);

                env.Close();
                return EXIT_SUCCESS;
            }

            /*
             * Tear down environment and remove its files.
             * Any log or database files and the environment
             * directory are not removed.
             */
            public static int TearDownEnv(string home)
            {
                /* Remove environment regions. */
                try
                {
                    DatabaseEnvironment.Remove(home);
                }
                catch (Exception e)
                {
                    Console.WriteLine("{0}: {1}\n{2}",
                        e.Source, e.Message, e.StackTrace);
                    return EXIT_FAILURE;
                }

                return EXIT_SUCCESS;
            }

            public static void usage()
            {
                Console.WriteLine("Usage: excs_env [home] [data dir]");
            }

     

            private static void UseEnv(DatabaseEnvironment env)
            {
                const int COUNT = 10;
                BTreeDatabase[] databases = new BTreeDatabase[COUNT];
                BTreeDatabaseConfig[] cfgs = new BTreeDatabaseConfig[COUNT];
                for (int i = 0; i < COUNT; i++)
                {
                    cfgs[i] = new BTreeDatabaseConfig();
                    cfgs[i].Creation = CreatePolicy.IF_NEEDED;
                    cfgs[i].Env = env;
                    string dbName = "db" + i.ToString();
                    databases[i] = BTreeDatabase.Open("file_test.db", dbName, cfgs[i]);
                }
                for (int i = 0; i < 100000; i++)
                {
                    DatabaseEntry dbKey = new DatabaseEntry(BitConverter.GetBytes(i));
                    for (int j = 0; j < COUNT; j++)
                    {
                        DatabaseEntry dbValue = new DatabaseEntry(Encoding.UTF8.GetBytes(new string((char)('A' + j), 512 * 1024) + i.ToString()));
                        databases[j].Put(dbKey, dbValue);
                        if (i > 0 && i % 10 == 0)
                        {
                            Console.WriteLine(string.Format("{0} items have been written in db{1}.", i, j));
                            databases[j].Sync();
                        }
                        dbValue.Dispose();
                    }
                    dbKey.Dispose();
                    env.SyncMemPool();
                }

                for (int i = 0; i < COUNT; i++)
                {
                    databases[i].Close();
                }
            }
        }
    }

  • 2. Re: A big bug in BDB on Win7 x64
    userBDBDMS Guru Moderator
    Currently Being Moderated

    thank you for the test case, this is very helpful.

     

    We will take a look at it and see what we find.

     

    thanks

    mike


  • 3. Re: A big bug in BDB on Win7 x64
    Hanhui Liang Newbie
    Currently Being Moderated

    Hi,

     

    I am a little confused about the problem you met.

    1. You are a C# API user, is the problem you met the same as the problem described by the C++ API user above?

    2. Is the message "NOT RELEASE ANY MEMORY" your summary or the error message you got from any tool?

    3. How do you define "Lost memory", "BDB uses an unknown memory as large as database Files, and it keeps growing until system out of memory", or something else?

     

    Thanks,

    Hanhui

    Oracle Berkeley DB

  • 4. Re: A big bug in BDB on Win7 x64
    userBDBDMS Guru Moderator
    Currently Being Moderated

    This is looking a bug in the OS and not in BDB.    We have a little more digging to do and once we confirm we will post additional information.

     

     

    thanks

    mike


  • 5. Re: A big bug in BDB on Win7 x64
    Hanhui Liang Newbie
    Currently Being Moderated

    Hi,


    Thanks for your test code. It's very useful. I run it on our test machines, and see what the problem is. The utilization of physical RAM would grow fast and continually. Finally, available memory is almost exhausted, and the system was painfully slow.

     

    We already have a thread discuss this problem. This appears to be an issue with the Windows System Cache.  Please see thread: https://forums.oracle.com/thread/2571599

     

    The root cause of the issue is "Windows System File Cache consumes most of the physical RAM" from Windows. For your reference, please see this: http://support.microsoft.com/kb/976618

     

     

    Regards,

    Hanhui