This discussion is archived
5 Replies Latest reply: Apr 18, 2012 6:44 AM by 922962 RSS

BDB3017 error

dmikh Newbie
Currently Being Moderated
Hi,

I'm trying to change default page size (16K on my system) using set_pagesize method of Db class. But any values greater than 16K causes the following error during execution of my code

BDB3017 unable to allocate space from the buffer cache
Abort (core dumped)

What can be wrong?

Thanks,
Dmitry
  • 1. Re: BDB3017 error
    Oracle, Sandra Whitman Journeyer
    Currently Being Moderated
    Hello,

    I would suggest starting by collecting the db_stat -MA statistics
    to collect information about the cache. Is the system actually out
    of memory? You can try increasing the cache. Without having any
    other details on how the program is configured and what it is doing
    that could resolve the issue or not.

    Thanks,
    Sandra
  • 2. Re: BDB3017 error
    dmikh Newbie
    Currently Being Moderated
    My system has 6 GB of memory so free memory should not be an issue. Below is output of db_stat -MA but it does not give any clues for me:

    Default cache region information:
    264KB     Total cache size
    1     Number of caches
    1     Maximum number of caches
    264KB     Pool individual cache size
    264KB     Pool individual cache max
    0     Maximum memory-mapped file size
    0     Maximum open file descriptors
    0     Maximum sequential buffer writes
    0     Sleep after writing maximum sequential buffers
    0     Requested pages mapped into the process' address space
    93721     Requested pages found in the cache (99%)
    448     Requested pages not found in the cache
    2066     Pages created in the cache
    448     Pages read into the cache
    2455     Pages written from the cache to the backing file
    56     Clean pages forced from the cache
    2455     Dirty pages forced from the cache
    0     Dirty pages written by trickle-sync thread
    3     Current total page count
    0     Current clean page count
    3     Current dirty page count
    37     Number of hash buckets used for page location
    37     Number of mutexes for the hash buckets
    4096     Assumed page size used
    497016     Total number of times hash chains searched for a page
    11     The longest hash chain searched for a page
    0     Total number of hash chain entries checked for page
    0     The number of hash bucket locks that required waiting (0%)
    0     The maximum number of times any hash bucket lock was waited for (0%)
    0     The number of region locks that required waiting (0%)
    0     The number of buffers frozen
    0     The number of buffers thawed
    0     The number of frozen buffers freed
    2535     The number of page allocations
    5888     The number of hash buckets examined during allocations
    646     The maximum number of hash buckets examined for an allocation
    2511     The number of pages examined during allocations
    172     The max number of pages examined for an allocation
    0     Threads waited on page I/O
    0     The number of times a sync is interrupted
    =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
    Pool File: experiment_dir
    1024     Page size
    0     Requested pages mapped into the process' address space
    0     Requested pages found in the cache (0%)
    1     Requested pages not found in the cache
    0     Pages created in the cache
    1     Pages read into the cache
    0     Pages written from the cache to the backing file
    =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
    Pool File: coverage
    65536     Page size
    0     Requested pages mapped into the process' address space
    0     Requested pages found in the cache (0%)
    1     Requested pages not found in the cache
    0     Pages created in the cache
    1     Pages read into the cache
    0     Pages written from the cache to the backing file
    =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
    Pool File: line_dir
    1024     Page size
    0     Requested pages mapped into the process' address space
    87169     Requested pages found in the cache (99%)
    443     Requested pages not found in the cache
    2065     Pages created in the cache
    443     Pages read into the cache
    2455     Pages written from the cache to the backing file
    =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
    Pool File: annotaton
    65536     Page size
    0     Requested pages mapped into the process' address space
    6552     Requested pages found in the cache (99%)
    3     Requested pages not found in the cache
    1     Pages created in the cache
    3     Pages read into the cache
    0     Pages written from the cache to the backing file
    =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
    Mpool REGINFO information:
    Mpool     Region type
    3     Region ID
    test/__db.003     Region name
    0xfea86000     Region address
    0xfea86000     Region allocation head
    0xfea860c4     Region primary address
    0     Region maximum allocation
    0     Region allocated
    Region allocations: 209 allocations, 2363 failures, 188 frees, 1 longest
    Allocations by power-of-two sizes:
    1KB     24
    2KB     2513
    4KB     0
    8KB     0
    16KB     0
    32KB     0
    64KB     0
    128KB     38
    256KB     0
    512KB     0
    1024KB     0
    REGION_JOIN_OK     Region flags
    =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
    MPOOL structure:
    3     MPOOL region mutex [0/5551 0% 11467/1]
    0/0     Maximum checkpoint LSN
    37     Hash table entries
    37     Hash table mutexes
    =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
    DB_MPOOL handle information:
    0     DB_MPOOL handle mutex [!Set]
    1     Underlying cache regions
    =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
    DB_MPOOLFILE structures:
    =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
    MPOOLFILE structures:
    File #1: experiment_dir
    148     Mutex [0/5 0% !Own]
    0     Revision count
    1     Reference count
    0     Block count
    1     Last page number
    1     Original last page number
    0     Maximum page number
    0     Type
    0     Priority
    0     Page's LSN offset
    32     Page's clear length
    39 59 1 0 8 0 ad 2 de 36 3 4d fc 39 3 0 0 0 0 0     ID
         Flags
    File #2: coverage
    141     Mutex [0/5 0% !Own]
    0     Revision count
    1     Reference count
    0     Block count
    1     Last page number
    1     Original last page number
    0     Maximum page number
    0     Type
    0     Priority
    0     Page's LSN offset
    32     Page's clear length
    36 59 1 0 8 0 ad 2 e7 0 46 4f bc 2c 0 0 0 0 0 0     ID
         Flags
    File #3: line_dir
    151     Mutex [0/9734 0% !Own]
    0     Revision count
    1     Reference count
    0     Block count
    2066     Last page number
    1     Original last page number
    0     Maximum page number
    0     Type
    0     Priority
    0     Page's LSN offset
    32     Page's clear length
    3a 59 1 0 8 0 ad 2 35 ad 60 4d 9c c0 4 0 0 0 0 0     ID
    file written     Flags
    File #4: annotaton
    145     Mutex [0/13 0% !Own]
    0     Revision count
    1     Reference count
    3     Block count
    2     Last page number
    1     Original last page number
    0     Maximum page number
    0     Type
    0     Priority
    0     Page's LSN offset
    32     Page's clear length
    38 59 1 0 8 0 ad 2 6c ea 63 4c 5c b3 1 0 0 0 0 0     ID
    file written     Flags
    =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
    Cache #1:
    12     Hash table last-checked
    96235     Hash table LRU priority
    0     Hash table LRU generation
    96235     Put counter
    BH hash table (37 hash slots)
    bucket #: priority, I/O wait, [mutex]
         pageno, file, ref, LSN, address, priority, flags
    bucket 4: 0 (1 dirty)[0/146 0% rd 0/786 0% 1 reader 11467/1]
         0, #4, 0, 0/1, 0x010fc4, 96232 (dirty)
    bucket 5: 0 (1 dirty)[0/267 0% rd 0/8130 0% 1 reader 11467/1]
         1, #4, 0, 0/1, 0x022004, 96233 (dirty)
    bucket 6: 0 (1 dirty)[0/234 0% rd 0/1396 0% 1 reader 11467/1]
         2, #4, 0, 0/1, 0x000dfc, 96234 (dirty, created)
  • 3. Re: BDB3017 error
    Oracle, Sandra Whitman Journeyer
    Currently Being Moderated
    Hello,

    It looks like you are selecting the default cache size of 255KB.
    Take a look at the, "Selecting a cache size" documentation at:
    http://download.oracle.com/docs/cd/E17076_02/html/programmer_reference/general_am_conf.html#am_conf_cachesize

    And try increasing the cache and see if that resolves the error.

    Thanks,
    Sandra
  • 4. Re: BDB3017 error
    dmikh Newbie
    Currently Being Moderated
    Increasing cache size to 656KB helped, thanks.
    But for me it looks strange that a lack of cache causes failure. I'd expect just slowdown in such a case.
  • 5. Re: BDB3017 error
    922962 Newbie
    Currently Being Moderated
    Hi I have a similar error message " BDB3017 unable to allocate space from the buffer cache" when using DbMpoolFile for a general purpose in-memory cache.
    It appears when the allocated cache size gets full, th application is not able to evict pages to the backing store and pagein requested pages.

    Am using berkelydb 5.3.15 on ubunt 11.10 64bit.

    Please find my sample code below. I will be grateful to know what am doing wrong. Thanks so much

    #include <sys/stat.h>
    #include <cstdlib>
    #include <errno.h>
    #include <stdlib.h>
    #include <string.h>
    #include <unistd.h>
    #include <db_cxx.h>

    #define     DATAFILE "mpool"
    #define _cachesize   10240
    #define _npages      10000
    #define _pagesize    512

    using namespace std;

    struct record {
    int key;
    int data;
    record(const int & theKey = 0,const int & theData = 0)
    : key(theKey) , data(theData) { }
    } ;
    typedef struct record pgdata;

    int cacheEnvInit __P((DB_ENV **dbenv, char *db_home, u_int32_t flags,
    size_t cachesize , const char* progname));
    int cacheEnvClose __P((DB_ENV **dbenv));

    int cachedFileOpen __P((DB_ENV dbenv, DB_MPOOLFILE *mfp, const char *fileName,
    u_int32_t flags, size_t bktsize));
    int cachedFileClose __P((DB_ENV dbenv, DB_MPOOLFILE *mfp));

    int initCachedFile __P((const DB_ENV dbenv, DB_MPOOLFILE mfp, size_t bktsize, int nbuckets));

    int displayCacheStats __P((DB_ENV *dbenv, u_int32_t flags));

    int runCacheTest __P((DB_ENV dbenv, DB_MPOOLFILE mfp, int nhits, int nbuckets));


    int main(int argc, char** argv) {

    DB_ENV *dbenv;
    DB_MPOOLFILE *mfp;
    DBT pgcookie ;
    char *ckVal = "CookieVal";
    int fType = 10, fType2 = 0 ;
    char *myHome = "datadir";

    (void) system("rm -rf datadir/*");

    cacheEnvInit(&dbenv, myHome, DB_CREATE | DB_INIT_MPOOL | DB_INIT_CDB ,
    _cachesize , "O2cache");

    fprintf(stderr, "Pass 1\n");
    umask((mode_t) 0012);

    cachedFileOpen(dbenv, &mfp, DATAFILE, DB_CREATE , _pagesize);   
    memset(&pgcookie, 0, sizeof(DBT));
    pgcookie.data = ckVal;
    pgcookie.size = (u_int32_t) strlen(ckVal) + 1;
    mfp->set_pgcookie(mfp, &pgcookie);

    mfp->set_ftype(mfp, fType);

    fprintf(stderr, "Pass 2 fType = %d\n", fType);

    mfp->get_ftype(mfp, &fType2);

    fprintf(stderr, "Pass 3 fType2 = %d\n", fType2);

    dbenv->memp_register(dbenv, fType, NULL,NULL);

    initCachedFile(dbenv, mfp, pagesize ,npages );

    cachedFileClose(dbenv, &mfp);

    fprintf(stderr, "Pass 4 \n");
    // return EXIT_SUCCESS ;

    cachedFileOpen(dbenv, &mfp, DATAFILE, DB_RDONLY, _pagesize);

    runCacheTest(dbenv,mfp, npages , pagesize);

    cachedFileClose(dbenv, &mfp);
    displayCacheStats(dbenv, 0);
    cacheEnvClose(&dbenv);

    return EXIT_SUCCESS ;

    return 0;
    }

    int cacheEnvInit(DB_ENV **dbenv, char *db_home, u_int32_t flags,
    size_t cachesize, const char* progname)
    {
    *dbenv = NULL;
    int ret;
    u_int32_t cgbyte, cbytes;

    fprintf(stderr,"cacheEnvInit Entry\n");
    printf("%s: cachesize: %d;\n", "cacheEnvInit", (int)cachesize);

    /*
    * Open a memory pool, specify a cachesize, output error messages
    * to stderr.
    */
    if ((ret = db_env_create(dbenv, 0)) != 0) {
    fprintf(stderr,
    "%s: db_env_create: %s\n", "cacheEnvInit", db_strerror(ret));
    return (ret);
    }

    (*dbenv)->set_errpfx(*dbenv, progname);
    (*dbenv)->set_errfile(*dbenv, stderr);
    (*dbenv)->set_msgfile(*dbenv, stderr);
    /* Open the environment. */

    (*dbenv)->set_cache_max(*dbenv, 0, 524288);

    if ((ret = (*dbenv)->open(*dbenv, db_home, flags, 0)) != 0) { //0777
    (*dbenv)->err(*dbenv, ret, "DB_ENV->open: %s", "cacheEnvInit");
    return (ret);
    }
    (*dbenv)->get_cache_max(*dbenv, &cgbyte, &cbytes);
    fprintf(stderr, "Max Cache GByte = %d, Max Cache Bytes = %d\n", cgbyte,cbytes);

    /* Set the cachesize. */
    if ((ret = (*dbenv)->set_cachesize(*dbenv, 0, (u_int32_t)cachesize, 1)) != 0) {
    (*dbenv)->err(*dbenv, ret, "DB_ENV->set_cachesize: %s","set_cacheEnvInit");
    return (ret);
    }
    fprintf(stderr,"cacheEnvInit Normal Exit\n");
    return EXIT_SUCCESS;
    }

    int cacheEnvClose(DB_ENV **dbenv)
    {
    int ret = 0;
    /* Close the Cache pool Environment */
    if (*dbenv != NULL) {
    if ((ret = (*dbenv)->close(*dbenv, 0)) != 0) {
    fprintf(stderr,
    "%s: db_env_close: %s\n", "cachedEnvClose", db_strerror(ret));
    return (ret);
    }
    }
    return (ret);
    }
    /* Open the file in the environment. */

    int cachedFileOpen(DB_ENV dbenv, DB_MPOOLFILE *mfp, const char *fileName,
    u_int32_t flags, size_t bktsize )
    {
    *mfp = NULL ;
    int ret = 0;
    fprintf(stderr, "cachedFileOpen Entry\n");

    /* Open the cache mpool environment. */
    if ((ret = dbenv->memp_fcreate(dbenv, mfp, 0)) != 0) {
    dbenv->err(dbenv, ret, "DB_ENV->memp_fcreate: %s", "cachedFileOPen");
    return (ret);
    }

    /* Attach the cache pool to a file */

    if ((ret = (*mfp)->open(*mfp, fileName, flags, 0776, bktsize)) != 0) {
    dbenv->err(dbenv, ret, "DB_MPOOLFILE->open: %s", fileName);
    return (ret);
    }

    fprintf(stderr, "cachedFileOpen Normal Exit\n");
    return ret;
    }

    /*
    * Close the file.
    */
    int cachedFileClose(DB_ENV dbenv, DB_MPOOLFILE *mfp)
    {
    int ret = 0 ;
    fprintf(stderr, "cachedFileClose Entry\n");
    if ((ret = (*mfp)->sync(*mfp)) != 0) {
    dbenv->err(dbenv, ret, "DB_ENV->memp_sync %s", "cachedFileClose");
    return (ret);
    }

    fprintf(stderr, "cachedFileClose Pass 1\n");

    if ((ret = (*mfp)->close(*mfp, 0)) != 0) {
    dbenv->err(dbenv, ret, "DB_MPOOLFILE->close %s", "cachedFileClose" );
    return (ret);
    }
    fprintf(stderr, "cachedFileClose Exit\n");
    return ret ;
    }
    /*
    * init - Create a backing file.
    */
    int initCachedFile(const DB_ENV dbenv, DB_MPOOLFILE mfp, size_t bktsize, int nbuckets)
    {
    int cnt, ret = 0;
    void *bkt ;
    db_pgno_t pageno, pgnoaddr;
    /*
    * Create a file with the right number of pages, and store a page
    * number on each page.
    */
    printf("initCachedFile Entry ... \n");
    printf("Creating %d buckets... \n", nbuckets);
    /*
    * The pages are numbered from 0, not 1.
    *
    * Write the index of the page at the beginning of the page in order
    * to verify the retrieved page (see run()).
    */

    for (cnt = 0; cnt < nbuckets; ++cnt) {
    pageno = cnt ;

    if ((ret = mfp->get(mfp, &pageno, NULL,
    DB_MPOOL_CREATE, &bkt)) != 0) {
    dbenv->err(dbenv, ret, "unable to create page %lu", (u_long) pageno);
    return ret;
    }

    ((pgdata *)bkt)->key = pageno; // pageno;

    fprintf(stderr,"initCachedFile: Page created is %lu\n",
    (u_long)((pgdata *)bkt)->key );

    pgnoaddr = ((pgdata*) bkt)->key ;

    if ((ret = mfp->put(mfp, bkt, DB_PRIORITY_DEFAULT, 0)) != 0) {
    dbenv->err(dbenv, ret, "unable to write page %lu", (u_long) pgnoaddr);
    return ret;
    } else {
    fprintf(stderr,"initCachedFile: Page written is %lu\n",
    (u_long) ((pgdata *) bkt)->key);
    }
    }
    fprintf(stderr, "Exiting %s: %d buckets Created... \n",
    "initCachedFile", nbuckets);
    return (ret);
    }

    int displayCacheStats(DB_ENV *dbenv, u_int32_t flags)
    {
    DB_MPOOL_STAT *gsp;
    DB_MPOOL_FSTAT **fsp; // declaration of (*fsp)[] gives warning

    char *funcName = "displayCacheStats";

    int ret = 0 ;

    if ((gsp = (DB_MPOOL_STAT *) malloc (sizeof(DB_MPOOL_STAT))) == NULL) {
    fprintf(stderr, "%s: %s\n", funcName, strerror(ENOMEM));
    return (1);
    }

    if ((ret = dbenv->memp_stat(dbenv, &gsp, &fsp, flags)) != 0) {
    dbenv->err(dbenv, ret, "displayCacheStats");
    return ret ;
    }

    if ( (ret = dbenv->memp_stat_print(dbenv, flags)) != 0) {
    dbenv->err(dbenv, ret, "displayCacheStats");
    return ret ;
    }
    return ret ;
    }
    int runCacheTest(DB_ENV dbenv, DB_MPOOLFILE mfp, int nhits, int nbuckets)
    {
    db_pgno_t bucketNo;
    int cnt, ret = 0;
    void *p;
    // DataBucket *bkt ;
    const char *funcName = "runCacheTest";

    printf("%s on file with %d buckets ", funcName, nbuckets);
    printf(" to retrieve %d random blocks... \n", nhits);

    srand((u_int)time(NULL));

    for (cnt = 0; cnt < nhits-2; ++cnt) {
    bucketNo = rand() % nbuckets;
    if ((ret = mfp->get(mfp, &bucketNo, NULL, 0 , &p)) != 0) {
    dbenv->err(dbenv, ret, "unable to retrieve bucketk %lu",
    (u_long) cnt);
    return ret;
    }
    /* Verify the block's number that was written in init(). */
    if ( ((pgdata*)p)->key != bucketNo) {
    dbenv->errx(dbenv, "wrong block retrieved (%lu != %d)",
    (u_long)bucketNo, *(int *)p);
    return ret;
    } else {

    /* if ((ret = mfp->put(mfp, p, DB_PRIORITY_DEFAULT, 0)) != 0) {
    dbenv->err(dbenv, ret, "unable to write page %lu", (u_long) bucketNo);
    return ret;
    }**/
    printf("Bucket number read is %lu\n", (u_long)((pgdata *)p)->key );
    }
    }

    printf("Successful Read All Buckets.\n");

    return (ret);
    }

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points