This discussion is archived
3 Replies Latest reply: Aug 22, 2013 11:26 PM by Oracle,CindyZeng RSS

DB_NOTFOUND on DB->open() with several (~800) partitions

995163 Newbie
Currently Being Moderated
Hello!

I'm testing the impact of the number of partitions on the performances of a BDB based application.

When I increase the number of partitions to about 800, the DB won't open correctly, and the following message will be returned by the DB->open() function:

BDB0073 DB_NOTFOUND: No matching key/data pair found+

Is there any undocumented limit on the number of partitions that could cause this?

Thanks!

Regards,

Nizar
  • 1. Re: DB_NOTFOUND on DB->open() with several (~800) partitions
    userBDBDMS Guru Moderator
    Currently Being Moderated
    Hi Nizar,

    No undocumented limit. We have not tested with this number of partitions in our internal testing. Do you have test code that can be used to reproduce this problem?


    thanks
    mike
  • 2. Re: DB_NOTFOUND on DB->open() with several (~800) partitions
    995163 Newbie
    Currently Being Moderated
    Thank you for your interest Mike.
    Here is a program that tries to open a DB, with N Partitions.

    #include <stdlib.h>
    #include <string.h>
    #include "db.h"

    void exitError();

    /*
    *
    * Initializes the DB and exits
    *
    * This program DOESN'T use a Berkeley DB Environment
    *
    */
    main(int argc, char **argv)
    {


    int i;
    int ret;
         
         struct timeval tp1,ttp2;
         double duration;

    DB *CATALOG_dbp  = NULL;

         printf("Start OK \n");

         gettimeofday(&tp1,NULL);


         /* Creating the database */
         if ((ret = db_create(&CATALOG_dbp, NULL, 0)) != 0) {
         fprintf(stderr,"Initialize error");
         //goto exit_error;
         exitError();
         return ret;
         }
         

         /* Creating partitions */
         int n_partitions=800;
         int size_partition=1000000;
         DBT partKeys[n_partitions];
         int *pntr;
         for (i=0; i<n_partitions; i++){
         /* Setup the partition keys */
         pntr = (int *)malloc(sizeof(int));
         pntr = size_partition(i+1);
    memset(&partKeys, 0, sizeof(DBT));
    partKeys[i].data = pntr;
    partKeys[i].size = sizeof(size_partition);
         }
    CATALOG_dbp->set_partition(CATALOG_dbp, n_partitions, partKeys, NULL);

         /* Other settings */
         CATALOG_dbp->set_errfile(CATALOG_dbp, stderr);

         /* Setting the cache size */
         int dbCacheSizeB = 0 ;     // Bytes
         int dbCacheSizeGB = 1;     // GBytes
         CATALOG_dbp->set_cachesize(CATALOG_dbp, dbCacheSizeGB, dbCacheSizeB, 1);
         
         /* Opening the database */
         if ((ret = CATALOG_dbp->open(CATALOG_dbp, NULL, "CATALOG.db", NULL, DB_BTREE, DB_CREATE, 0644)) != 0){
         CATALOG_dbp->err(CATALOG_dbp, ret, "DB->open: %s", "CATALOG.db");
         goto exit_error;
         }
         printf("Initialize OK \n");
    gettimeofday(&ttp2,NULL);
    duration+=(double) (ttp2.tv_sec-tp1.tv_sec) + 1E-6 * (ttp2.tv_usec-tp1.tv_usec);
    printf("TIME: %.3lf s \n",duration);
    fflush(stdout);


    exit_error:

    /*
    * Close the database
    */
    if (CATALOG_dbp != NULL)
    ret = CATALOG_dbp->close(CATALOG_dbp, 0);

    return ret;
    }

    void exitError(DB *CATALOG_dbp){

    if (CATALOG_dbp != NULL)
    CATALOG_dbp->close(CATALOG_dbp, 0);

    }
  • 3. Re: DB_NOTFOUND on DB->open() with several (~800) partitions
    Oracle,CindyZeng Newbie
    Currently Being Moderated

    Hi,

     

    There are also some requirements of partition keys.

     

    Each DBT in the partition keys should not have NULL for its data field. Moreover there should not be any duplicate keys in the 'keys' parameter.

     

    If any is not satisfied, DB->open may fail or some partitions may never be used.

     


         /* Creating partitions */
         int n_partitions=800;
         int size_partition=1000000;
         DBT partKeys[n_partitions];
         int *pntr;
         for (i=0; i<n_partitions; i++){
         /* Setup the partition keys */
         pntr = (int *)malloc(sizeof(int));
         pntr = size_partition(i+1);
    memset(&partKeys, 0, sizeof(DBT));
    partKeys[i].data = pntr;
    partKeys[i].size = sizeof(size_partition);
          }
    CATALOG_dbp->set_partition(CATALOG_dbp, n_partitions, partKeys, NULL);

     

    In your program, you keep memset the first DBT in each round of the for loop by "memset(&partKeys, 0, sizeof(DBT));".  So after the for loop is done, the first DBT in the partition key array has NULL for its data field. This does not satisfy one of the  requirements listed above.

     

    If there is any question, please let me know.

     

    Regards,

    Cindy

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points