1 Reply Latest reply on Jun 13, 2014 4:54 AM by Winter Zhang

    Batch operation slower - doing something wrong?


      I thought the batch Put operation will be much faster than single Put operation in loop but it's not. Am I doing something wrong here? The code is in C#.


      //Batch Operation Methods

      public void Initialize(uint pageSize)


           var envConfig = new DatabaseEnvironmentConfig;

           envConfig.Create = true;

                envConfig.UseMPool = true;

                envConfig.UseTxns = true;

                envConfig.AutoCommit = true;

                envConfig.TxnNoSync = true;       


           _dbEnv = DatabaseEnvironment.Open("Home", envConfig);

           Transaction transaction = _dbEnv.BeginTransaction();


                _btreeConfig = new BTreeDatabaseConfig();

                _btreeConfig.Env = _dbEnv;

                _btreeConfig.Creation = CreatePolicy.IF_NEEDED;

                _btreeConfig.PageSize = pageSize; // 8 * 1024

                _btreeDb = BTreeDatabase.Open("TestData.db", "Primary", _btreeConfig, transaction);



      public void AddRows(List<TestData> testDataList)
                var transaction = _dbEnv.BeginTransaction();


                var keyValuePairList = new List<KeyValuePair<DatabaseEntry, DatabaseEntry>>();

                for (var index = 0; index < testDataList.Count; index++)
                          keyValuePairList.Add(new KeyValuePair<DatabaseEntry, DatabaseEntry>(
                                               new DatabaseEntry(BitConverter.GetBytes(testDataList[index].Index)),
                                               new DatabaseEntry(testDataList[index].Serialize())));


                _btreeDb.Put(new MultipleKeyDatabaseEntry(keyValuePairList, false), transaction);



      //One row at a time Methods

      public void Initialize(uint pageSize)
                  _btreeConfig = new BTreeDatabaseConfig
                      Creation = CreatePolicy.ALWAYS,
                      CacheSize = new CacheInfo(0, 64 * 1024, 1),
                      ErrorPrefix = "prototype",
                      PageSize = pageSize //8 * 1024


           _btreeDb = BTreeDatabase.Open(DbName, _btreeConfig);


      public static void AddRow(string dataKey, TestData dataValue)
                  var key = new DatabaseEntry(Encoding.ASCII.GetBytes(dataKey));

                  var data = new DatabaseEntry(dataValue.Serialize());

                  _btreeDb.Put(key, data);

        • 1. Re: Batch operation slower - doing something wrong?
          Winter Zhang



          Batch operation wrappers individual operations into a single operation, so it saves time for CPU. But as far as I see, most applications spend the major time on I/O cost, so CPU time saving does not seem so greatly, unless your store the databases pure in-memory.


          But since operations can be in batch, you can sort your key-value pairs before putting, and this usually helps in performance. So have a try to see if this works.



          Winter, Oracle Berkeley DB