This discussion is archived
1 2 Previous Next 27 Replies Latest reply: Aug 24, 2012 9:47 AM by vinothchandar RSS

Quick question on DiskOrderedCursorConfig#setInternalMemoryLimit

vinothchandar Newbie
Currently Being Moderated
Hi,

I stumbled across this method

http://docs.oracle.com/cd/E17277_02/html/java/com/sleepycat/je/DiskOrderedCursorConfig.html#setInternalMemoryLimit%28long%29

and was curious. What does non je cache memory exactly mean? Does this mean that the disk ordered scan will not interfere with the JE cache for that environment and work in separate memory area in the JVM heap?

Thanks
Vinoth
  • 1. Re: Quick question on DiskOrderedCursorConfig#setInternalMemoryLimit
    greybird Expert
    Currently Being Moderated
    Yes, a disk ordered cursor will read data from the JE cache but will not add anything to it. It reads other (non-cached) records separately, directly from the log files, and uses memory allocated separately (not included in the JE cache budget) to do this.

    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
  • 2. Re: Quick question on DiskOrderedCursorConfig#setInternalMemoryLimit
    vinothchandar Newbie
    Currently Being Moderated
    This is awesome. And one final question -- this memory gets allocated when the scan starts and gets torn down after the scan completes?

    Thanks
    Vinoth
  • 3. Re: Quick question on DiskOrderedCursorConfig#setInternalMemoryLimit
    greybird Expert
    Currently Being Moderated
    And one final question -- this memory gets allocated when the scan starts and gets torn down after the scan completes?
    Any allocated memory is freed when the cursor is closed, but the use of memory during the scan is much more complex -- there isn't just a single block of memory allocated at the beginning and freed at the end. For more info please take a look at the implementation:

    src/com/sleepycat/je/dbi/DiskOrderedCursorImpl.java
    src/com/sleepycat/je/dbi/SortedLSNTreeWalker.java

    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
  • 4. Re: Quick question on DiskOrderedCursorConfig#setInternalMemoryLimit
    vinothchandar Newbie
    Currently Being Moderated
    Alright. will do. thanks for the pointers.

    Thanks
    Vinoth
  • 5. Re: Quick question on DiskOrderedCursorConfig#setInternalMemoryLimit
    vinothchandar Newbie
    Currently Being Moderated
    Hi,

    I am doing evaluation on JE 5.0.58 and converting the existing scans to using DiskOrderedCursor. (Note that sortedDuplicates is false. I have handled duplicates myself)

    Time and again, when I do the scan in background while taking live traffic (get, put), cursor.getNext(key,data, LockMode.UNCOMMITTED) returns null for the key.
    I cannot reproduce the problem without the live traffic. Is this expected at all?

    Thanks
    Vinoth

    My code for your reference.

    DiskOrderedCursorConfig docc = new DiskOrderedCursorConfig();
    docc.setInternalMemoryLimit(64 * 1024 * 1024);
    DiskOrderedCursor cursor = getBdbDatabase().openCursor(docc);
    while () {
    OperationStatus status = cursor.getNext(keyEntry,
    valueEntry,
    LockMode.READ_UNCOMMITTED);
    }
    cursor.close()
  • 6. Re: Quick question on DiskOrderedCursorConfig#setInternalMemoryLimit
    greybird Expert
    Currently Being Moderated
    Hi Vinoth,

    That behavior does not seem correct to me, or if it is correct at least it is not documented. When the returned key is null, what is the OperationStatus returned? Is it KEYEMPTY?

    --mark                                                                                                                                                                                                                                                                                                                                                                                                                       
  • 7. Re: Quick question on DiskOrderedCursorConfig#setInternalMemoryLimit
    vinothchandar Newbie
    Currently Being Moderated
    Hi,

    No. Actually it returns SUCCESS. For example,

    OperationStatus status = cursor.getNext(keyEntry,
    valueEntry,
    LockMode.READ_UNCOMMITTED);

    if(OperationStatus.NOTFOUND == status) {
    // we have reached the end of the cursor
    return false;
    }
    if (keyEntry.getData() == null){
    numNullResults++;
    System.out.println("Null key "+ numNullResults+" op status "+ status.toString());
    }


    prints.

    Null key 1 op status OperationStatus.SUCCESS
    Null key 2 op status OperationStatus.SUCCESS
    Null key 3 op status OperationStatus.SUCCESS
    Null key 4 op status OperationStatus.SUCCESS
    ...
    ...
  • 8. Re: Quick question on DiskOrderedCursorConfig#setInternalMemoryLimit
    greybird Expert
    Currently Being Moderated
    It's a JE bug then. I'll take a look soon and let you know at least whether skipping over the null keys is safe. Of course, it will need to be fixed as well and I've created internal ticket [#21667].

    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                       
  • 9. Re: Quick question on DiskOrderedCursorConfig#setInternalMemoryLimit
    vinothchandar Newbie
    Currently Being Moderated
    Hi Mark,

    Oh :( Sure. Thanks. Were you able to reproduce it or would a reproducible test code help?
    If yes, I will start writing up one.

    Thanks
    Vinoth
  • 10. Re: Quick question on DiskOrderedCursorConfig#setInternalMemoryLimit
    greybird Expert
    Currently Being Moderated
    would a reproducible test code help?
    Absolutely.
    --mark                                                                                                                                                                                                   
  • 11. Re: Quick question on DiskOrderedCursorConfig#setInternalMemoryLimit
    vinothchandar Newbie
    Currently Being Moderated
    Hi mark,

    I tried to write a standalone program to reproduce this, basically have a DiskOrderedScan going on and parallely throw traffic at the DB.
    However, I am unable to reproduce the problem. The only difference being the data itself.
    When I hit the problem, the data I have is converted from 4.0.92 using 4.1.20 PreUpgrade utility.

    I am wondering if there is some debugging I can turn on and give you information as I see it using real data. Anyways, I have emailed you the test code.

    Thanks
    Vinoth
  • 12. Re: Quick question on DiskOrderedCursorConfig#setInternalMemoryLimit
    vinothchandar Newbie
    Currently Being Moderated
    One more observation, I modified my code such that it just skips over null keys. the number of items thus scanned is less than the actual number of items in the db (based on a normal scan). I got a total of 1397 nulls in a DB with 74M entries (0.001%). Hence, I don't think its safe to skip the nulls and keep moving on.

    This aside, the memory isolation the DiskOrderedCursor provides seems to fix our GC woes while doing daily scan jobs. Hence, I am very very interested in working with you to get this fixed. Please let me know if you need any more information on this.

    Thanks
    Vinoth
  • 13. Re: Quick question on DiskOrderedCursorConfig#setInternalMemoryLimit
    vinothchandar Newbie
    Currently Being Moderated
    Hi Mark,

    Sorry to be a pest. Please let me know if you have any more guesses about what's going on

    Thanks
    Vinoth
  • 14. Re: Quick question on DiskOrderedCursorConfig#setInternalMemoryLimit
    greybird Expert
    Currently Being Moderated
    No, I don't. but I was planning to send you email in the next day or two, to give you an instrumented JE jar, assuming you can reproduce the problem reliably in your app.

    --mark                                                                                                                                                                                                                                                                                                                                                                           
1 2 Previous Next

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points