5 Replies Latest reply: Aug 2, 2013 10:12 AM by Winter Zhang RSS

    Berkley Database stalls adding 65,536 entry

    user634811

      I've been playing around using BDB on a couple of different platforms.  I noticed a similar situation.  When I sit in a loop doing adds, at the 65,536th add BDB stalls for minute or two.  After a minute or two, the add succeeds.

       

      This situation seems to happen when I have around 43 10MB log files.  During the stall, I notice many log files are being written (another 25 or so), which is a much quicker rate than was being written prior to the stall. 

       

      The stall only happens once. I added another 350,000 entries and no more stalls.

       

      Trying to understand what BDB is doing during the stall. 

       

      Thanks for any help,

       

      Mark

        • 1. Re: Berkley Database stalls adding 65,536 entry
          Winter Zhang

          Hi Mark,

           

          Can you provide more information about you application ? Like the following:

           

          1 What is the access method you are using ? Btree, Hash, Recno, Queue or Heap ? And what's the page size you are using ?

          2 What is the entry like ? The average length of the key and data.

          3 How do you configure your environment ? I see you are using logs, so are you opening the environment using DB_INIT_TXN, and DB_INIT_LOCK ? What's the cache size of your environment and are you doing checkpoint periodically ?  Are you using replication ?

           

          It is best if you can provide a sample program or at least provide the code about environment configuration.

           

          Regards,

          -Winter, Oracle Berkeley DB

          • 2. Re: Berkley Database stalls adding 65,536 entry
            user634811

            I'm using BDB as a backend from OpenLDAP.  So, I'm not sure I know all your answers, but I hope your familiar with OpenLDAP's use of BDB.

             

            1) Btree - at least I've certainly seen btree routines being called.

            2) There are 7 data items in an entry (at least in OpenLDAP terms).

            3) Some values from my DB_CONFIG file:

            set_cachesize 0  20971520 1

            set_lg_regionmax  1048576

            set_lg_max        10485760  

            set_lg_bsize      2097152

            set_lk_max_locks   2000

            set_lk_max_objects 2000

            set_open_flags db_private

             

            I was hoping the question was really a general question in nature and the details were not important, but maybe not.

             

            I guess it's possible the OpenLDAP is causing the delay, but it seemed like BDB was doing all the activity in this period.

            • 3. Re: Berkley Database stalls adding 65,536 entry
              Winter Zhang

              Hi Mark,

               

              From the DB_CONFIG, you are using a private environment which prevents db_stat utility from getting statistics out of the environment.

               

              I have no idea of how OpenLDAP uses BerkeleyDB. I will take a look, but that will take some time, it will take me a little longer to get to you again.

               

              Regards,

              -Winter

              • 4. Re: Berkley Database stalls adding 65,536 entry
                user634811

                I asked this question to OpenLDAP and they believe it is an OpenLDAP condition that happens relating to indexes.  If my understanding is correct, after 65535, the max number of index slots in the index table, the table is collapsed into ranges.  This reorganizing of the table is causing my delay.

                 

                Thanks for any time you spent looking at this issue.

                • 5. Re: Berkley Database stalls adding 65,536 entry
                  Winter Zhang

                  Hi Mark,

                   

                  It is good to know that you get the answer from OpenLDAP.  Sorry for not being able to get your further.

                   

                  Regards,

                  --Winter