5 Replies Latest reply: Apr 18, 2012 8:44 AM by 922962 RSS

    BDB3017 error

    dmikh
      Hi,

      I'm trying to change default page size (16K on my system) using set_pagesize method of Db class. But any values greater than 16K causes the following error during execution of my code

      BDB3017 unable to allocate space from the buffer cache
      Abort (core dumped)

      What can be wrong?

      Thanks,
      Dmitry
        • 1. Re: BDB3017 error
          Oracle, Sandra Whitman
          Hello,

          I would suggest starting by collecting the db_stat -MA statistics
          to collect information about the cache. Is the system actually out
          of memory? You can try increasing the cache. Without having any
          other details on how the program is configured and what it is doing
          that could resolve the issue or not.

          Thanks,
          Sandra
          • 2. Re: BDB3017 error
            dmikh
            My system has 6 GB of memory so free memory should not be an issue. Below is output of db_stat -MA but it does not give any clues for me:

            Default cache region information:
            264KB     Total cache size
            1     Number of caches
            1     Maximum number of caches
            264KB     Pool individual cache size
            264KB     Pool individual cache max
            0     Maximum memory-mapped file size
            0     Maximum open file descriptors
            0     Maximum sequential buffer writes
            0     Sleep after writing maximum sequential buffers
            0     Requested pages mapped into the process' address space
            93721     Requested pages found in the cache (99%)
            448     Requested pages not found in the cache
            2066     Pages created in the cache
            448     Pages read into the cache
            2455     Pages written from the cache to the backing file
            56     Clean pages forced from the cache
            2455     Dirty pages forced from the cache
            0     Dirty pages written by trickle-sync thread
            3     Current total page count
            0     Current clean page count
            3     Current dirty page count
            37     Number of hash buckets used for page location
            37     Number of mutexes for the hash buckets
            4096     Assumed page size used
            497016     Total number of times hash chains searched for a page
            11     The longest hash chain searched for a page
            0     Total number of hash chain entries checked for page
            0     The number of hash bucket locks that required waiting (0%)
            0     The maximum number of times any hash bucket lock was waited for (0%)
            0     The number of region locks that required waiting (0%)
            0     The number of buffers frozen
            0     The number of buffers thawed
            0     The number of frozen buffers freed
            2535     The number of page allocations
            5888     The number of hash buckets examined during allocations
            646     The maximum number of hash buckets examined for an allocation
            2511     The number of pages examined during allocations
            172     The max number of pages examined for an allocation
            0     Threads waited on page I/O
            0     The number of times a sync is interrupted
            =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
            Pool File: experiment_dir
            1024     Page size
            0     Requested pages mapped into the process' address space
            0     Requested pages found in the cache (0%)
            1     Requested pages not found in the cache
            0     Pages created in the cache
            1     Pages read into the cache
            0     Pages written from the cache to the backing file
            =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
            Pool File: coverage
            65536     Page size
            0     Requested pages mapped into the process' address space
            0     Requested pages found in the cache (0%)
            1     Requested pages not found in the cache
            0     Pages created in the cache
            1     Pages read into the cache
            0     Pages written from the cache to the backing file
            =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
            Pool File: line_dir
            1024     Page size
            0     Requested pages mapped into the process' address space
            87169     Requested pages found in the cache (99%)
            443     Requested pages not found in the cache
            2065     Pages created in the cache
            443     Pages read into the cache
            2455     Pages written from the cache to the backing file
            =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
            Pool File: annotaton
            65536     Page size
            0     Requested pages mapped into the process' address space
            6552     Requested pages found in the cache (99%)
            3     Requested pages not found in the cache
            1     Pages created in the cache
            3     Pages read into the cache
            0     Pages written from the cache to the backing file
            =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
            Mpool REGINFO information:
            Mpool     Region type
            3     Region ID
            test/__db.003     Region name
            0xfea86000     Region address
            0xfea86000     Region allocation head
            0xfea860c4     Region primary address
            0     Region maximum allocation
            0     Region allocated
            Region allocations: 209 allocations, 2363 failures, 188 frees, 1 longest
            Allocations by power-of-two sizes:
            1KB     24
            2KB     2513
            4KB     0
            8KB     0
            16KB     0
            32KB     0
            64KB     0
            128KB     38
            256KB     0
            512KB     0
            1024KB     0
            REGION_JOIN_OK     Region flags
            =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
            MPOOL structure:
            3     MPOOL region mutex [0/5551 0% 11467/1]
            0/0     Maximum checkpoint LSN
            37     Hash table entries
            37     Hash table mutexes
            =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
            DB_MPOOL handle information:
            0     DB_MPOOL handle mutex [!Set]
            1     Underlying cache regions
            =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
            DB_MPOOLFILE structures:
            =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
            MPOOLFILE structures:
            File #1: experiment_dir
            148     Mutex [0/5 0% !Own]
            0     Revision count
            1     Reference count
            0     Block count
            1     Last page number
            1     Original last page number
            0     Maximum page number
            0     Type
            0     Priority
            0     Page's LSN offset
            32     Page's clear length
            39 59 1 0 8 0 ad 2 de 36 3 4d fc 39 3 0 0 0 0 0     ID
                 Flags
            File #2: coverage
            141     Mutex [0/5 0% !Own]
            0     Revision count
            1     Reference count
            0     Block count
            1     Last page number
            1     Original last page number
            0     Maximum page number
            0     Type
            0     Priority
            0     Page's LSN offset
            32     Page's clear length
            36 59 1 0 8 0 ad 2 e7 0 46 4f bc 2c 0 0 0 0 0 0     ID
                 Flags
            File #3: line_dir
            151     Mutex [0/9734 0% !Own]
            0     Revision count
            1     Reference count
            0     Block count
            2066     Last page number
            1     Original last page number
            0     Maximum page number
            0     Type
            0     Priority
            0     Page's LSN offset
            32     Page's clear length
            3a 59 1 0 8 0 ad 2 35 ad 60 4d 9c c0 4 0 0 0 0 0     ID
            file written     Flags
            File #4: annotaton
            145     Mutex [0/13 0% !Own]
            0     Revision count
            1     Reference count
            3     Block count
            2     Last page number
            1     Original last page number
            0     Maximum page number
            0     Type
            0     Priority
            0     Page's LSN offset
            32     Page's clear length
            38 59 1 0 8 0 ad 2 6c ea 63 4c 5c b3 1 0 0 0 0 0     ID
            file written     Flags
            =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
            Cache #1:
            12     Hash table last-checked
            96235     Hash table LRU priority
            0     Hash table LRU generation
            96235     Put counter
            BH hash table (37 hash slots)
            bucket #: priority, I/O wait, [mutex]
                 pageno, file, ref, LSN, address, priority, flags
            bucket 4: 0 (1 dirty)[0/146 0% rd 0/786 0% 1 reader 11467/1]
                 0, #4, 0, 0/1, 0x010fc4, 96232 (dirty)
            bucket 5: 0 (1 dirty)[0/267 0% rd 0/8130 0% 1 reader 11467/1]
                 1, #4, 0, 0/1, 0x022004, 96233 (dirty)
            bucket 6: 0 (1 dirty)[0/234 0% rd 0/1396 0% 1 reader 11467/1]
                 2, #4, 0, 0/1, 0x000dfc, 96234 (dirty, created)
            • 3. Re: BDB3017 error
              Oracle, Sandra Whitman
              Hello,

              It looks like you are selecting the default cache size of 255KB.
              Take a look at the, "Selecting a cache size" documentation at:
              http://download.oracle.com/docs/cd/E17076_02/html/programmer_reference/general_am_conf.html#am_conf_cachesize

              And try increasing the cache and see if that resolves the error.

              Thanks,
              Sandra
              • 4. Re: BDB3017 error
                dmikh
                Increasing cache size to 656KB helped, thanks.
                But for me it looks strange that a lack of cache causes failure. I'd expect just slowdown in such a case.
                • 5. Re: BDB3017 error
                  922962
                  Hi I have a similar error message " BDB3017 unable to allocate space from the buffer cache" when using DbMpoolFile for a general purpose in-memory cache.
                  It appears when the allocated cache size gets full, th application is not able to evict pages to the backing store and pagein requested pages.

                  Am using berkelydb 5.3.15 on ubunt 11.10 64bit.

                  Please find my sample code below. I will be grateful to know what am doing wrong. Thanks so much

                  #include <sys/stat.h>
                  #include <cstdlib>
                  #include <errno.h>
                  #include <stdlib.h>
                  #include <string.h>
                  #include <unistd.h>
                  #include <db_cxx.h>

                  #define     DATAFILE "mpool"
                  #define _cachesize   10240
                  #define _npages      10000
                  #define _pagesize    512

                  using namespace std;

                  struct record {
                  int key;
                  int data;
                  record(const int & theKey = 0,const int & theData = 0)
                  : key(theKey) , data(theData) { }
                  } ;
                  typedef struct record pgdata;

                  int cacheEnvInit __P((DB_ENV **dbenv, char *db_home, u_int32_t flags,
                  size_t cachesize , const char* progname));
                  int cacheEnvClose __P((DB_ENV **dbenv));

                  int cachedFileOpen __P((DB_ENV dbenv, DB_MPOOLFILE *mfp, const char *fileName,
                  u_int32_t flags, size_t bktsize));
                  int cachedFileClose __P((DB_ENV dbenv, DB_MPOOLFILE *mfp));

                  int initCachedFile __P((const DB_ENV dbenv, DB_MPOOLFILE mfp, size_t bktsize, int nbuckets));

                  int displayCacheStats __P((DB_ENV *dbenv, u_int32_t flags));

                  int runCacheTest __P((DB_ENV dbenv, DB_MPOOLFILE mfp, int nhits, int nbuckets));


                  int main(int argc, char** argv) {

                  DB_ENV *dbenv;
                  DB_MPOOLFILE *mfp;
                  DBT pgcookie ;
                  char *ckVal = "CookieVal";
                  int fType = 10, fType2 = 0 ;
                  char *myHome = "datadir";

                  (void) system("rm -rf datadir/*");

                  cacheEnvInit(&dbenv, myHome, DB_CREATE | DB_INIT_MPOOL | DB_INIT_CDB ,
                  _cachesize , "O2cache");

                  fprintf(stderr, "Pass 1\n");
                  umask((mode_t) 0012);

                  cachedFileOpen(dbenv, &mfp, DATAFILE, DB_CREATE , _pagesize);   
                  memset(&pgcookie, 0, sizeof(DBT));
                  pgcookie.data = ckVal;
                  pgcookie.size = (u_int32_t) strlen(ckVal) + 1;
                  mfp->set_pgcookie(mfp, &pgcookie);

                  mfp->set_ftype(mfp, fType);

                  fprintf(stderr, "Pass 2 fType = %d\n", fType);

                  mfp->get_ftype(mfp, &fType2);

                  fprintf(stderr, "Pass 3 fType2 = %d\n", fType2);

                  dbenv->memp_register(dbenv, fType, NULL,NULL);

                  initCachedFile(dbenv, mfp, pagesize ,npages );

                  cachedFileClose(dbenv, &mfp);

                  fprintf(stderr, "Pass 4 \n");
                  // return EXIT_SUCCESS ;

                  cachedFileOpen(dbenv, &mfp, DATAFILE, DB_RDONLY, _pagesize);

                  runCacheTest(dbenv,mfp, npages , pagesize);

                  cachedFileClose(dbenv, &mfp);
                  displayCacheStats(dbenv, 0);
                  cacheEnvClose(&dbenv);

                  return EXIT_SUCCESS ;

                  return 0;
                  }

                  int cacheEnvInit(DB_ENV **dbenv, char *db_home, u_int32_t flags,
                  size_t cachesize, const char* progname)
                  {
                  *dbenv = NULL;
                  int ret;
                  u_int32_t cgbyte, cbytes;

                  fprintf(stderr,"cacheEnvInit Entry\n");
                  printf("%s: cachesize: %d;\n", "cacheEnvInit", (int)cachesize);

                  /*
                  * Open a memory pool, specify a cachesize, output error messages
                  * to stderr.
                  */
                  if ((ret = db_env_create(dbenv, 0)) != 0) {
                  fprintf(stderr,
                  "%s: db_env_create: %s\n", "cacheEnvInit", db_strerror(ret));
                  return (ret);
                  }

                  (*dbenv)->set_errpfx(*dbenv, progname);
                  (*dbenv)->set_errfile(*dbenv, stderr);
                  (*dbenv)->set_msgfile(*dbenv, stderr);
                  /* Open the environment. */

                  (*dbenv)->set_cache_max(*dbenv, 0, 524288);

                  if ((ret = (*dbenv)->open(*dbenv, db_home, flags, 0)) != 0) { //0777
                  (*dbenv)->err(*dbenv, ret, "DB_ENV->open: %s", "cacheEnvInit");
                  return (ret);
                  }
                  (*dbenv)->get_cache_max(*dbenv, &cgbyte, &cbytes);
                  fprintf(stderr, "Max Cache GByte = %d, Max Cache Bytes = %d\n", cgbyte,cbytes);

                  /* Set the cachesize. */
                  if ((ret = (*dbenv)->set_cachesize(*dbenv, 0, (u_int32_t)cachesize, 1)) != 0) {
                  (*dbenv)->err(*dbenv, ret, "DB_ENV->set_cachesize: %s","set_cacheEnvInit");
                  return (ret);
                  }
                  fprintf(stderr,"cacheEnvInit Normal Exit\n");
                  return EXIT_SUCCESS;
                  }

                  int cacheEnvClose(DB_ENV **dbenv)
                  {
                  int ret = 0;
                  /* Close the Cache pool Environment */
                  if (*dbenv != NULL) {
                  if ((ret = (*dbenv)->close(*dbenv, 0)) != 0) {
                  fprintf(stderr,
                  "%s: db_env_close: %s\n", "cachedEnvClose", db_strerror(ret));
                  return (ret);
                  }
                  }
                  return (ret);
                  }
                  /* Open the file in the environment. */

                  int cachedFileOpen(DB_ENV dbenv, DB_MPOOLFILE *mfp, const char *fileName,
                  u_int32_t flags, size_t bktsize )
                  {
                  *mfp = NULL ;
                  int ret = 0;
                  fprintf(stderr, "cachedFileOpen Entry\n");

                  /* Open the cache mpool environment. */
                  if ((ret = dbenv->memp_fcreate(dbenv, mfp, 0)) != 0) {
                  dbenv->err(dbenv, ret, "DB_ENV->memp_fcreate: %s", "cachedFileOPen");
                  return (ret);
                  }

                  /* Attach the cache pool to a file */

                  if ((ret = (*mfp)->open(*mfp, fileName, flags, 0776, bktsize)) != 0) {
                  dbenv->err(dbenv, ret, "DB_MPOOLFILE->open: %s", fileName);
                  return (ret);
                  }

                  fprintf(stderr, "cachedFileOpen Normal Exit\n");
                  return ret;
                  }

                  /*
                  * Close the file.
                  */
                  int cachedFileClose(DB_ENV dbenv, DB_MPOOLFILE *mfp)
                  {
                  int ret = 0 ;
                  fprintf(stderr, "cachedFileClose Entry\n");
                  if ((ret = (*mfp)->sync(*mfp)) != 0) {
                  dbenv->err(dbenv, ret, "DB_ENV->memp_sync %s", "cachedFileClose");
                  return (ret);
                  }

                  fprintf(stderr, "cachedFileClose Pass 1\n");

                  if ((ret = (*mfp)->close(*mfp, 0)) != 0) {
                  dbenv->err(dbenv, ret, "DB_MPOOLFILE->close %s", "cachedFileClose" );
                  return (ret);
                  }
                  fprintf(stderr, "cachedFileClose Exit\n");
                  return ret ;
                  }
                  /*
                  * init - Create a backing file.
                  */
                  int initCachedFile(const DB_ENV dbenv, DB_MPOOLFILE mfp, size_t bktsize, int nbuckets)
                  {
                  int cnt, ret = 0;
                  void *bkt ;
                  db_pgno_t pageno, pgnoaddr;
                  /*
                  * Create a file with the right number of pages, and store a page
                  * number on each page.
                  */
                  printf("initCachedFile Entry ... \n");
                  printf("Creating %d buckets... \n", nbuckets);
                  /*
                  * The pages are numbered from 0, not 1.
                  *
                  * Write the index of the page at the beginning of the page in order
                  * to verify the retrieved page (see run()).
                  */

                  for (cnt = 0; cnt < nbuckets; ++cnt) {
                  pageno = cnt ;

                  if ((ret = mfp->get(mfp, &pageno, NULL,
                  DB_MPOOL_CREATE, &bkt)) != 0) {
                  dbenv->err(dbenv, ret, "unable to create page %lu", (u_long) pageno);
                  return ret;
                  }

                  ((pgdata *)bkt)->key = pageno; // pageno;

                  fprintf(stderr,"initCachedFile: Page created is %lu\n",
                  (u_long)((pgdata *)bkt)->key );

                  pgnoaddr = ((pgdata*) bkt)->key ;

                  if ((ret = mfp->put(mfp, bkt, DB_PRIORITY_DEFAULT, 0)) != 0) {
                  dbenv->err(dbenv, ret, "unable to write page %lu", (u_long) pgnoaddr);
                  return ret;
                  } else {
                  fprintf(stderr,"initCachedFile: Page written is %lu\n",
                  (u_long) ((pgdata *) bkt)->key);
                  }
                  }
                  fprintf(stderr, "Exiting %s: %d buckets Created... \n",
                  "initCachedFile", nbuckets);
                  return (ret);
                  }

                  int displayCacheStats(DB_ENV *dbenv, u_int32_t flags)
                  {
                  DB_MPOOL_STAT *gsp;
                  DB_MPOOL_FSTAT **fsp; // declaration of (*fsp)[] gives warning

                  char *funcName = "displayCacheStats";

                  int ret = 0 ;

                  if ((gsp = (DB_MPOOL_STAT *) malloc (sizeof(DB_MPOOL_STAT))) == NULL) {
                  fprintf(stderr, "%s: %s\n", funcName, strerror(ENOMEM));
                  return (1);
                  }

                  if ((ret = dbenv->memp_stat(dbenv, &gsp, &fsp, flags)) != 0) {
                  dbenv->err(dbenv, ret, "displayCacheStats");
                  return ret ;
                  }

                  if ( (ret = dbenv->memp_stat_print(dbenv, flags)) != 0) {
                  dbenv->err(dbenv, ret, "displayCacheStats");
                  return ret ;
                  }
                  return ret ;
                  }
                  int runCacheTest(DB_ENV dbenv, DB_MPOOLFILE mfp, int nhits, int nbuckets)
                  {
                  db_pgno_t bucketNo;
                  int cnt, ret = 0;
                  void *p;
                  // DataBucket *bkt ;
                  const char *funcName = "runCacheTest";

                  printf("%s on file with %d buckets ", funcName, nbuckets);
                  printf(" to retrieve %d random blocks... \n", nhits);

                  srand((u_int)time(NULL));

                  for (cnt = 0; cnt < nhits-2; ++cnt) {
                  bucketNo = rand() % nbuckets;
                  if ((ret = mfp->get(mfp, &bucketNo, NULL, 0 , &p)) != 0) {
                  dbenv->err(dbenv, ret, "unable to retrieve bucketk %lu",
                  (u_long) cnt);
                  return ret;
                  }
                  /* Verify the block's number that was written in init(). */
                  if ( ((pgdata*)p)->key != bucketNo) {
                  dbenv->errx(dbenv, "wrong block retrieved (%lu != %d)",
                  (u_long)bucketNo, *(int *)p);
                  return ret;
                  } else {

                  /* if ((ret = mfp->put(mfp, p, DB_PRIORITY_DEFAULT, 0)) != 0) {
                  dbenv->err(dbenv, ret, "unable to write page %lu", (u_long) bucketNo);
                  return ret;
                  }**/
                  printf("Bucket number read is %lu\n", (u_long)((pgdata *)p)->key );
                  }
                  }

                  printf("Successful Read All Buckets.\n");

                  return (ret);
                  }