This discussion is archived
1 2 3 Previous Next 30 Replies Latest reply: Aug 23, 2012 3:15 AM by userBDBDMS Go to original post RSS
  • 15. Re: Problems with increasing/decreasing cache size when live
    897965 Newbie
    Currently Being Moderated
    Hi,
    Thanks for you answer but actually it doesn't help much...
    Let me please explain once again.

    I have some code that compacts 32 env sequentially and to do so smoothly, it dynamically increases the cache size of the env being compacted and decreases when done. This code works normally with BDB 4.7.25 but leaks so I tried to upgrade to 5.1 and then to 5.3 but with those newer versions this code crashes when decreasing the cache size to its original value (see previous stack trace).

    Let me summarize here what I believe is wrong with BDB code (or at least not following documentation):

    Before open, I do:
    DBEnvironment->set_cachesize((u_int32_t)0, (u_int32_t)512*1024*1024/32, 1);
    DBEnvironment->set_cache_max(1*1024*1024*1024, 0);
    DBEnvironment->get_cache_max(&gbytes, &obytes); // gives gbytes=1 and obytes=0

    Here everything is good but after open, I have the following:

    DBEnvironment->get_cache_max(&gbytes, &obytes); // gives gbytes=0 and obytes=9502720
    memp_stat: cache_size=18644992 cache_ncache=1, max_ncache=58

    which is strange, right? But please have a look at the following:

    1. DBEnvironment->set_cache_size(0, 9502720, 0)

    Here I increase cache to cache_max and get:
    memp_stat: outlinks cache_size=27328512 cache_ncache=54 max_ncache=58

    This doesn't make sense, why cache_size ((ui64)sp->st_gbytes * GIGA + sp->st_bytes)) was increased? why my cache is now more than 1GB when I specify 9MB...

    2. DBEnvironment->set_cache_size(0, 18644992, 0)

    Then I try to decrease cache to its original value and get: cannot resize to 142 cache regions: maximum is 58.

    This doesn't make sense either.

    3. DBEnvironment->set_cache_size(0, 9502720/2, 0)

    If I try to decrease cache to cache_max / 2, I get the stack trace I posted before:

    Program terminated with signal 11, Segmentation fault.
    #0 __memp_fput (dbmfp=0x11a62b780, ip=0x0, pgaddr=0x214155058,
    priority=DB_PRIORITY_UNCHANGED)
    at /place/home/bdb5.0/mp/mp_fput.c:123
    in /place/home/bdb5.0/mp/mp_fput.c

    Is there some magic formula out there to apply on cache sizes for these methods to behave correctly?

    Hope that helps. I really would like to make this work but really don't understand what I do wrong or what workaround I can use here.
    Thanks a lot.
  • 16. Re: Problems with increasing/decreasing cache size when live
    Oracle,CindyZeng Newbie
    Currently Being Moderated
    Hi,

    Thanks for providing the information.

    Could you please help confirm the details so that I can reproduce the issue better?

    Here is what I think the case is.

    //Before open
    DBEnvironment->set_cachesize((u_int32_t)0, (u_int32_t)512*1024*1024/32, 1);
    DBEnvironment->set_cache_max(1*1024*1024*1024, 0);
    DBEnvironment->get_cache_max(&gbytes, &obytes); // gives gbytes=1 and obytes=0

    //After open
    DBEnvironment->get_cache_max(&gbytes, &obytes); // gives gbytes=0 and obytes=9502720
    memp_stat: cache_size=18644992 cache_ncache=1, max_ncache=58

    //Resize cache to maximum size just got
    DBEnvironment->set_cache_size(0, 9502720, 0)
    memp_stat: outlinks cache_size=27328512 cache_ncache=54 max_ncache=58

    //Resize cache to original cache size after open
    DBEnvironment->set_cache_size(0, 18644992, 0)
    get: cannot resize to 142 cache regions: maximum is 58

    //Resize the cache to half of the maximum size
    DBEnvironment->set_cache_size(0, 9502720/2, 0)
    get: segmentation falult

    Is that correct?
  • 17. Re: Problems with increasing/decreasing cache size when live
    897965 Newbie
    Currently Being Moderated
    Oracle, Cindy Zeng wrote:
    Hi,

    Thanks for providing the information.

    Could you please help confirm the details so that I can reproduce the issue better?

    Here is what I think the case is.
    Hi,

    I confirm that your description is correct, that's exactly how it goes for me.

    I'm using FreeBSD 8.2, Berkeley DB 5.3.15: (December 19, 2011) and DB_BTREE tables.
    Here are my env (open) flags: DB_CREATE | DB_INIT_LOCK | DB_INIT_LOG | DB_INIT_MPOOL | DB_INIT_TXN | DB_RECOVER | DB_PRIVATE | DB_THREAD;
    Here are my env flags: 0 | DB_AUTO_COMMIT | DB_TXN_NOSYNC;

    Thanks you again.
  • 18. Re: Problems with increasing/decreasing cache size when live
    897965 Newbie
    Currently Being Moderated
    Oracle, Cindy Zeng wrote:
    Hi,

    Thanks for providing the information.

    Could you please help confirm the details so that I can reproduce the issue better?
    Hi,

    Is there any news about what could be done here?

    Thank you.
  • 19. Re: Problems with increasing/decreasing cache size when live
    924602 Newbie
    Currently Being Moderated
    Hi

    it is quite interesting to know if the problem is reproduced and can we expect to get it fixed. We will appreciate any details about is status and possible resolution time.

    Thanks in advance.

    --
    p.s.
  • 20. Re: Problems with increasing/decreasing cache size when live
    526060 Explorer
    Currently Being Moderated
    Hi,

    Sorry for the long delay. We have reproduced some issues with the cache resizing and are working to fix them. I can't give you a timeline for the fix. I will let you know when we have more information. Thanks for your patience.

    Regards,
    Alex Gorrod,
    Oracle Berkeley DB
  • 21. Re: Problems with increasing/decreasing cache size when live
    924602 Newbie
    Currently Being Moderated
    Thanks a lot for the update. Looking forward to get the patch and try it.

    BTW we can try some preliminary patches if you wish.

    p.s.
  • 22. Re: Problems with increasing/decreasing cache size when live
    924602 Newbie
    Currently Being Moderated
    Hi

    Do you have any news regarding the issue?

    --
    p.s.
  • 23. Re: Problems with increasing/decreasing cache size when live
    924602 Newbie
    Currently Being Moderated
    Hi

    We are still waiting for any update and for the fix of course. We need it for the functionality developed several months ago that can't work in production due to crash in BDB.

    Could you please provide any information that might help us to adjust our expectation? Currently situation is completely unclear and it looks like no one is working on it.

    Thanks in advance.

    --
    p.s
  • 24. Re: Problems with increasing/decreasing cache size when live
    carol.sandstrom Newbie
    Currently Being Moderated
    Hi,

    I regret and apologize that we've been moving so slowly on this issue, but we do have some progress to report.

    At this point, we have reproduced several errors, and applied fixes so that increasing the cache size works reliably. We are working on fixes so decreasing the cache size will work as well. Mostly what I can tell you is that we have not forgotten about you or about this issue.

    Regards,
    Carol
  • 25. Re: Problems with increasing/decreasing cache size when live
    924602 Newbie
    Currently Being Moderated
    Thanks for the update, Carol

    Do you have any estimation when this work can be completed?

    --
    p.s.
  • 26. Re: Problems with increasing/decreasing cache size when live
    carol.sandstrom Newbie
    Currently Being Moderated
    Hi Sergey,

    My best guess is that it will be fixed this summer. It is hard to provide a good estimate since it is not the highest priority work for the engineers assigned to it.

    Regards,
    Carol
  • 27. Re: Problems with increasing/decreasing cache size when live
    924602 Newbie
    Currently Being Moderated
    Hi Carol

    I hope that my repeated question won't annoy you :) Do you have any good news for us?

    p.s.
  • 28. Re: Problems with increasing/decreasing cache size when live
    carol.sandstrom Newbie
    Currently Being Moderated
    Hi Sergey,

    Thanks for reminding me about this. An (at least partial) fix has been checked into the current code line, which means it will be part of the next Berkeley DB release. I have asked an engineer to look into whether that fix can be applied against the 5.3 release. I'll let you know what I find out.

    Regards,
    Carol
  • 29. Re: Problems with increasing/decreasing cache size when live
    924602 Newbie
    Currently Being Moderated
    Thanks for these good news!

    What is BDB version where this change is going to be included and when we can expect its release?

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points