14 Replies Latest reply: Apr 28, 2011 2:25 AM by 518667 RSS

    Bulk update example

    AMARCIONEK
      Are there any examples of the new bulk update feature in 4.8? I'm having a hard time understanding it.

      We use transactions and have some number of secondary DBs associated.

      The bulk inserts are definitely multiple data items with potentially unique or potentially duplicate keys.

      Here is some pseudocode:

      // Initialization of the Dbt:

      Dbt dbtBulkKey;

      dbtBulkKey.set_data([some buffer]);
      dbtBulkKey.set_ulen([buffer size]); // Mulitple of 4.
      dbtBulkKey.set_flags(DB_DBT_USERMEM);

      Every time I add a key I use this (there is essentially a for loop around this:)

           // Get Sequence number
           db_seq_t iS64SeqNum = GetNextSequenceNumber();

           Dbt keyRecord(&iS64SeqNum, sizeof(iS64SeqNum));

           DbMultipleKeyDataBuilder mkdb(dbtBulkKey);

           mkdb.append(&keyRecord, keyRecord.get_dlen(), pDataRec, cbDataRec);

      // Now put

      pDbEnvironment->txn_begin(NULL, &pTxn, DB_READ_COMMITTED);
      pDbPrimary->put(pTxn, &dbtBulkKey, &dataRecord, DB_MULTIPLE_KEY); // The data Record is a dummy?

      etc.

      Hows that look?
        • 1. Re: Bulk update example
          600261
          I have asked the bdb team for examples, but no reply now. Example is very important for newbie or skilled to try the new feature. I'v tried the bulk update, but with no luck, it does not work.
          • 2. Re: Bulk update example
            524395
            Adam, Steve,

            Hey, I apologize for the silence. This past week has been a busy one for me with the two product launches. I did read your post last week and I flagged it, but I'd not come back to it yet. Again, apologies. I'm alerting the support team and the developer responsible for the feature. We should have included some new example code for all the new features, another oversight on my part. This feature works quite well, so don't give up!

            cheers,

            -greg
            Product Manager, Oracle Berkeley DB
            • 3. Re: Bulk update example
              Oracle, Sandra Whitman
              Hi Adam, Steve,

              I will work on getting an example together for you.

              Sandra
              • 4. Re: Bulk update example
                Oracle, Sandra Whitman
                Hi Adam, Steve,

                I created a patch on top of the 4.8.24 bench_001.c example program which provides arguments to use bulk update and bulk delete. Please send me an email at
                sandra.whitman@<this company>.com and I will send that to you. If that sample
                does not answer your questions, just let me know.

                Thanks,
                Sandra
                • 5. Re: Bulk update example
                  712359
                  Hi everyone,

                  Were you able to figure out usage of the bulk load operation? I am looking for code examples to better understand the usage.. but couldn't find any on the internet. There are a few urls that discuss this, but they are not complete and I found them difficult to understand.

                  If you have some working code using this feature, it would be helpful for others if you could post it here..

                  Thanks
                  AndOr
                  • 6. Re: Bulk update example
                    Oracle, Sandra Whitman
                    Hello,

                    The examples_c/bench_001.c example program provided with the 11gR2 release illustrates the use of the bulk feature. See the arguments -B/-D perform bulk updates/perform bulk deletes.

                    Thanks,
                    Sandra
                    • 7. Re: Bulk update example
                      766000
                      Great!
                      • 8. Re: Bulk update example
                        518667
                        Hello everybody,

                        are there any examples/tests for the C++ bulk API or are there only the C examples available?

                        Thanks.

                        Best regards
                        Martin
                        • 9. Re: Bulk update example
                          518667
                          Is nobody using the DbMultipleKeyDataBuilder class? I think this feature is not used by anyone. Google returns as a result only this post.

                          The documentation is really poor, there is even no information what to do with the data parameter in the Db::put method. Should it be 0 or an empty Dbt?

                          There should be more examples or better more unit tests written for the C++ API as it is done for Java (MultipleCursorTest::testMultiplePut)
                          • 10. Re: Bulk update example
                            656853
                            Hi all,

                            There are two ways to do bulk update in BDB C++ API:
                            1. Construct bulk buffers in the key Dbt and data Dbt with DbMultipleDataBuilder, and fill the buffers with each key/data pair by DbMultipleDataBuilder.append() or DbMultipleDataBuilder.reserve(). Then, call Db::put() with DB_MULTIPLE.

                            2. Construct a single bulk buffer in the key Dbt with DbMultipleKeyDataBuilder, and fill the buffer with key/data pairs by DbMultipleKeyDataBuilder.append() or DbMultipleKeyDataBuilder.reserve(). Then, call Db::put() with DB_MULTIPLE_KEY.

                            Here is an example to do bulk update in above #1:
                            Dbt mkey, mdata;
                            char keybuff[1024];
                            memset(keybuff, 0, 1024);
                            mkey.set_ulen(1024 * sizeof(char));
                            mkey.set_data(&keybuff);
                            
                            char databuff[1024];
                            memset(databuff, 0, 1024);
                            mdata.set_ulen(1024 * sizeof(char));
                            mdata.set_data(&databuff);
                            
                            DbMultipleDataBuilder keybuilder(mkey);
                            DbMultipleDataBuilder databuilder(mdata);
                            
                            int key = 10;
                            for (int i = 0; i < 500; i++) {
                                 keybuilder.append(&key, len);
                                 databuilder.append(&i, 4);
                            }
                            
                            if ((ret = dbp->put(NULL, &mkey, &mdata, DB_MULTIPLE)) != 0) {
                                 dbp->err(ret, "Db::put");
                                 throw DbException(ret);
                            }
                            To do bulk retrieval in C++ API:
                            1. To retrieve multiple data items for a single key, call Db::get() or Dbc::get() with DB_MULTIPLE, where the data and ulen field of the data Dbt are referred to a buffer that will be filled with returned data, and its flags is DB_DBT_USERMEM. Create a DbMultipleDataIterator on the returned data Dbt, and call DbMultipleDataIterator.next() to iterate over each data item.

                            2. To retrieve multiple key/data pairs, call Db::get() or Dbc::get() with DB_MULTIPLE_KEY, where the data and ulen field of the data Dbt are referred to a buffer that will be filled with returned key/data pairs, and its flags is DB_DBT_USERMEM. Create a DbMultipleKeyDataIterator on the returned data Dbt, and call DbMultipleKeyDataIterator.next() to iterate over each key/data item.

                            Here is an example to do bulk retrieval in above #1:
                            Dbt key;
                            
                            char buffer[1024];
                            memset(buffer, 0, 1024);
                            
                            Dbt mdata;
                            mdata.set_ulen(1024 * sizeof(char));
                            mdata.set_data(&buffer);
                            mdata.set_flags(DB_DBT_USERMEM);
                            
                            if ((ret = dbcp->get(&key, &mdata, DB_MULTIPLE | DB_NEXT)) != 0) {
                                 dbp->err(ret, "DBcursor->get");
                                 throw DbException(ret);
                            }
                            
                            Dbt sdata;
                            DbMultipleDataIterator iterator(mdata);
                            while (iterator.next(sdata)) {
                                 // Each data item is returned in sdata.          
                            }
                            Please refer to all C++ bulk APIs at:
                            * http://download.oracle.com/docs/cd/E17076_02/html/api_reference/CXX/dbt.html#dbtlist.

                            Also, the bulk C API doc and reference book might be helpful:
                            * http://download.oracle.com/docs/cd/E17076_02/html/programmer_reference/am_misc_bulk.html
                            * http://download.oracle.com/docs/cd/E17076_02/html/api_reference/C/dbt.html#dbtlist

                            If you are able to download the latest library, you could also refer to the bulk C example, named ex_bulk.

                            Hope it helps!

                            Regards,
                            Emily Fu, Oracle Berkeley DB
                            • 11. Re: Bulk update example
                              518667
                              Thank you Emily for your example.

                              Could you also add some C++ examples to the BDB source code examples for the next release, please :) C examples are not enough.
                              • 12. Re: Bulk update example
                                518667
                                Hello Emily,

                                I have one more question:

                                I want to create a Dbt buffer for lets say 1000 int keys. So I have to create a buffer of size = 1000 * sizeof(int). After that I append in a loop those 1000 integers into the Dbt builder. But at this point I got an segmentation fault. How big should be the buffer for the Dbt's?


                                Thanks in advance.

                                Martin
                                • 13. Re: Bulk update example
                                  858191
                                  Has anyone repplied to your question? I'm having the same problem, at the moment of the creation of the DbMultipleDataBuilder I get a segmentation fault...

                                  Please help..
                                  • 14. Re: Bulk update example
                                    518667
                                    Unfortunately, none has replied to my question.

                                    AMARCIONEK has written in the opening message that the size of the Dbt has to be the multiple of 4. When you have 1000 ints I additionally multiply 'sizeof(int) * 1000' it with that number. Somehow this is working, but don't ask me why, it's kind of magic :) I had no time to check the code for it.