3 Replies Latest reply: Aug 23, 2013 1:26 AM by Oracle,CindyZeng RSS

    DB_NOTFOUND on DB->open() with several (~800) partitions

    995163
      Hello!

      I'm testing the impact of the number of partitions on the performances of a BDB based application.

      When I increase the number of partitions to about 800, the DB won't open correctly, and the following message will be returned by the DB->open() function:

      BDB0073 DB_NOTFOUND: No matching key/data pair found+

      Is there any undocumented limit on the number of partitions that could cause this?

      Thanks!

      Regards,

      Nizar
        • 1. Re: DB_NOTFOUND on DB->open() with several (~800) partitions
          userBDBDMS-Oracle
          Hi Nizar,

          No undocumented limit. We have not tested with this number of partitions in our internal testing. Do you have test code that can be used to reproduce this problem?


          thanks
          mike
          • 2. Re: DB_NOTFOUND on DB->open() with several (~800) partitions
            995163
            Thank you for your interest Mike.
            Here is a program that tries to open a DB, with N Partitions.

            #include <stdlib.h>
            #include <string.h>
            #include "db.h"

            void exitError();

            /*
            *
            * Initializes the DB and exits
            *
            * This program DOESN'T use a Berkeley DB Environment
            *
            */
            main(int argc, char **argv)
            {


            int i;
            int ret;
                 
                 struct timeval tp1,ttp2;
                 double duration;

            DB *CATALOG_dbp  = NULL;

                 printf("Start OK \n");

                 gettimeofday(&tp1,NULL);


                 /* Creating the database */
                 if ((ret = db_create(&CATALOG_dbp, NULL, 0)) != 0) {
                 fprintf(stderr,"Initialize error");
                 //goto exit_error;
                 exitError();
                 return ret;
                 }
                 

                 /* Creating partitions */
                 int n_partitions=800;
                 int size_partition=1000000;
                 DBT partKeys[n_partitions];
                 int *pntr;
                 for (i=0; i<n_partitions; i++){
                 /* Setup the partition keys */
                 pntr = (int *)malloc(sizeof(int));
                 pntr = size_partition(i+1);
            memset(&partKeys, 0, sizeof(DBT));
            partKeys[i].data = pntr;
            partKeys[i].size = sizeof(size_partition);
                 }
            CATALOG_dbp->set_partition(CATALOG_dbp, n_partitions, partKeys, NULL);

                 /* Other settings */
                 CATALOG_dbp->set_errfile(CATALOG_dbp, stderr);

                 /* Setting the cache size */
                 int dbCacheSizeB = 0 ;     // Bytes
                 int dbCacheSizeGB = 1;     // GBytes
                 CATALOG_dbp->set_cachesize(CATALOG_dbp, dbCacheSizeGB, dbCacheSizeB, 1);
                 
                 /* Opening the database */
                 if ((ret = CATALOG_dbp->open(CATALOG_dbp, NULL, "CATALOG.db", NULL, DB_BTREE, DB_CREATE, 0644)) != 0){
                 CATALOG_dbp->err(CATALOG_dbp, ret, "DB->open: %s", "CATALOG.db");
                 goto exit_error;
                 }
                 printf("Initialize OK \n");
            gettimeofday(&ttp2,NULL);
            duration+=(double) (ttp2.tv_sec-tp1.tv_sec) + 1E-6 * (ttp2.tv_usec-tp1.tv_usec);
            printf("TIME: %.3lf s \n",duration);
            fflush(stdout);


            exit_error:

            /*
            * Close the database
            */
            if (CATALOG_dbp != NULL)
            ret = CATALOG_dbp->close(CATALOG_dbp, 0);

            return ret;
            }

            void exitError(DB *CATALOG_dbp){

            if (CATALOG_dbp != NULL)
            CATALOG_dbp->close(CATALOG_dbp, 0);

            }
            • 3. Re: DB_NOTFOUND on DB->open() with several (~800) partitions
              Oracle,CindyZeng

              Hi,

               

              There are also some requirements of partition keys.

               

              Each DBT in the partition keys should not have NULL for its data field. Moreover there should not be any duplicate keys in the 'keys' parameter.

               

              If any is not satisfied, DB->open may fail or some partitions may never be used.

               


                   /* Creating partitions */
                   int n_partitions=800;
                   int size_partition=1000000;
                   DBT partKeys[n_partitions];
                   int *pntr;
                   for (i=0; i<n_partitions; i++){
                   /* Setup the partition keys */
                   pntr = (int *)malloc(sizeof(int));
                   pntr = size_partition(i+1);
              memset(&partKeys, 0, sizeof(DBT));
              partKeys[i].data = pntr;
              partKeys[i].size = sizeof(size_partition);
                    }
              CATALOG_dbp->set_partition(CATALOG_dbp, n_partitions, partKeys, NULL);

               

              In your program, you keep memset the first DBT in each round of the for loop by "memset(&partKeys, 0, sizeof(DBT));".  So after the for loop is done, the first DBT in the partition key array has NULL for its data field. This does not satisfy one of the  requirements listed above.

               

              If there is any question, please let me know.

               

              Regards,

              Cindy