1 Reply Latest reply: Jun 12, 2014 11:54 PM by Winter Zhang RSS

    Batch operation slower - doing something wrong?

    2082ae58-b094-443d-8ef2-17541e3e6fb5

      I thought the batch Put operation will be much faster than single Put operation in loop but it's not. Am I doing something wrong here? The code is in C#.

       

      //Batch Operation Methods

      public void Initialize(uint pageSize)

      {

           var envConfig = new DatabaseEnvironmentConfig;

           envConfig.Create = true;

                envConfig.UseMPool = true;

                envConfig.UseTxns = true;

                envConfig.AutoCommit = true;

                envConfig.TxnNoSync = true;       

       

           _dbEnv = DatabaseEnvironment.Open("Home", envConfig);

           Transaction transaction = _dbEnv.BeginTransaction();

       

                _btreeConfig = new BTreeDatabaseConfig();

                _btreeConfig.Env = _dbEnv;

                _btreeConfig.Creation = CreatePolicy.IF_NEEDED;

                _btreeConfig.PageSize = pageSize; // 8 * 1024

                _btreeDb = BTreeDatabase.Open("TestData.db", "Primary", _btreeConfig, transaction);

      }

       

      public void AddRows(List<TestData> testDataList)
      {
                var transaction = _dbEnv.BeginTransaction();

               

                var keyValuePairList = new List<KeyValuePair<DatabaseEntry, DatabaseEntry>>();

                for (var index = 0; index < testDataList.Count; index++)
                {
                          keyValuePairList.Add(new KeyValuePair<DatabaseEntry, DatabaseEntry>(
                                               new DatabaseEntry(BitConverter.GetBytes(testDataList[index].Index)),
                                               new DatabaseEntry(testDataList[index].Serialize())));
                }

               

                _btreeDb.Put(new MultipleKeyDatabaseEntry(keyValuePairList, false), transaction);

      }

       

      //One row at a time Methods

      public void Initialize(uint pageSize)
      {
                  _btreeConfig = new BTreeDatabaseConfig
                  {  
                      Creation = CreatePolicy.ALWAYS,
                      CacheSize = new CacheInfo(0, 64 * 1024, 1),
                      ErrorPrefix = "prototype",
                      PageSize = pageSize //8 * 1024
                  };

                

           _btreeDb = BTreeDatabase.Open(DbName, _btreeConfig);
      }

       

      public static void AddRow(string dataKey, TestData dataValue)
      {
                  var key = new DatabaseEntry(Encoding.ASCII.GetBytes(dataKey));

                  var data = new DatabaseEntry(dataValue.Serialize());

                  _btreeDb.Put(key, data);
      }

        • 1. Re: Batch operation slower - doing something wrong?
          Winter Zhang

          Hi,

           

          Batch operation wrappers individual operations into a single operation, so it saves time for CPU. But as far as I see, most applications spend the major time on I/O cost, so CPU time saving does not seem so greatly, unless your store the databases pure in-memory.

           

          But since operations can be in batch, you can sort your key-value pairs before putting, and this usually helps in performance. So have a try to see if this works.

           

          Regards,

          Winter, Oracle Berkeley DB