This discussion is archived
1 2 Previous Next 27 Replies Latest reply: Aug 24, 2012 9:47 AM by vinothchandar Go to original post RSS
  • 15. Re: Quick question on DiskOrderedCursorConfig#setInternalMemoryLimit
    vinothchandar Newbie
    Currently Being Moderated
    Sure. That would be great.

    Thanks
    Vinoth
  • 16. Re: Quick question on DiskOrderedCursorConfig#setInternalMemoryLimit
    greybird Expert
    Currently Being Moderated
    However, I am unable to reproduce the problem. The only difference being the data itself. When I hit the problem, the data I have is converted from 4.0.92 using 4.1.20 PreUpgrade utility.
    Vinoth, is the problematic data in a duplicates DB or a non-duplicates DB?
    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
  • 17. Re: Quick question on DiskOrderedCursorConfig#setInternalMemoryLimit
    vinothchandar Newbie
    Currently Being Moderated
    Hi Mark,

    The database has no duplicates. Here is what I do.

    1. scan the existing DB with duplicates (4.0.92) and convert to a format without duplicates (4.0.92)
    2. On the converted data without duplicates, run the PreUpgrade tool for BDB 5
    3. Bring up environment with BDB5 and do a disk ordered scan.

    Thanks
    Vinoth
  • 18. Re: Quick question on DiskOrderedCursorConfig#setInternalMemoryLimit
    greybird Expert
    Currently Being Moderated
    1. scan the existing DB with duplicates (4.0.92) and convert to a format without duplicates (4.0.92)
    Just to be certain, for the new database, DatabaseConfig.setSortedDuplicates(true) is not called. Correct?

    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
  • 19. Re: Quick question on DiskOrderedCursorConfig#setInternalMemoryLimit
    943066 Newbie
    Currently Being Moderated
    Just a random thought, but have you tried this with READ_UNCOMMITTED turned off?

    Without looking at the internals in the slightest, it does seem feasible that a disk ordered cursor could end up on a key that ends up not existing after a transaction abort?

    Obviously, this isn't desired behaviour either way, but it may help narrow down on the problem.
  • 20. Re: Quick question on DiskOrderedCursorConfig#setInternalMemoryLimit
    greybird Expert
    Currently Being Moderated
    Just a random thought, but have you tried this with READ_UNCOMMITTED turned off?

    Without looking at the internals in the slightest, it does seem feasible that a disk ordered cursor could end up on a key that ends up not existing after a transaction abort?
    Thanks for the thought. We are working on this off-forum and I have a potential fix, I'll report back here when it's final.

    For the record, a DiskOrderedCursor only operates in READ_UNCOMMITTED mode, so that can't be turned off.

    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
  • 21. Re: Quick question on DiskOrderedCursorConfig#setInternalMemoryLimit
    vinothchandar Newbie
    Currently Being Moderated
    yes. I am getting rid of duplicates in the new data. I am getting slammed with lot of production issues today. Will definitely do the testing on the experimental jar and get back to you in a day or two.

    Thanks for all the help , Mark

    Thanks
    Vinoth
  • 22. Re: Quick question on DiskOrderedCursorConfig#setInternalMemoryLimit
    greybird Expert
    Currently Being Moderated
    I am getting rid of duplicates in the new data
    The problem is, that's an ambiguous statement. It is possible to call DatabaseConfig.setSortedDuplicates(true) and still have only one record per key (no duplicates). That's why I need to know for certain that you're not calling DatabaseConfig.setSortedDuplicates(true).

    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
  • 23. Re: Quick question on DiskOrderedCursorConfig#setInternalMemoryLimit
    vinothchandar Newbie
    Currently Being Moderated
    Well, I assumed it was clear since we would not need to turn it on otherwise.
    Anyways, let me be explicit. I am doing DatabaseConfig.setSortedDuplicates(false)

    https://github.com/voldemort/voldemort/commit/65e06be129070a97894645797c78a5c99d5a45dc#diff-6

    Thanks
    Vinoth
  • 24. Re: Quick question on DiskOrderedCursorConfig#setInternalMemoryLimit
    greybird Expert
    Currently Being Moderated
    Thanks.
    --mark                                                                                                                                                                                                                                                   
  • 25. Re: Quick question on DiskOrderedCursorConfig#setInternalMemoryLimit
    vinothchandar Newbie
    Currently Being Moderated
    Hi mark,

    I have some questions on the semantics offered by the DiskOrderedScan.
    If there are live updates happening during scan,

    -- the scan will first encounter the old LN first, check Btree Index and return it since its a valid live entry.
    -- Concurrent update logs a new LN
    -- DOScan hits the new LN at a later point, Btree index is valid again and returns new LN also

    How is this prevented? In general, does a DOScan starting at time 't' have the semantics that it returns live k-v pairs as of time t? If not, can you explicitly the semantics provided?
    Also, the lookup on the Btree index, does this happen on the memory buffer for the DOScan or do this bring in new index nodes into the regular BDB cache?
    Anyways, I will go through the files you pointed out and try to understand this better. I know this is a lot of detailed info. but could help people moving their regular cursors to DOCursors.

    PS : DO Scan is great!

    Thanks
    Vinoth
  • 26. Re: Quick question on DiskOrderedCursorConfig#setInternalMemoryLimit
    greybird Expert
    Currently Being Moderated
    Hi Vinoth,
    I have some questions on the semantics offered by the DiskOrderedScan.
    The high level description of this is in the [url http://docs.oracle.com/cd/E17277_02/html/java/com/sleepycat/je/Database.html#openCursor(com.sleepycat.je.DiskOrderedCursorConfig)]openCursor javadoc:
    >
    The records returned by the scan correspond to the state of the data at the beginning of the scan plus some, but not all, changes made by the application after the start of the scan. The user should not rely on the scan returning any changes made after the start of the scan. Specifically, if the record referred to by the ForwardCursor in a DiskOrderedScan is deleted after the ForwardCursor is positioned at that record, getCurrent() will still return the key and value of that record and OperationStatus.SUCCESS. As with a READ_UNCOMMITTED scan, changes made by all transactions, including uncommitted transactions, may be returned by the scan. Also, a record returned by the scan is not locked, and may be modified or deleted by the application after it is returned, including modification or deletion of the record at the cursor position. If a transactionally correct data set is required, the application must ensure that all transactions that write to the key range are committed before the beginning of the scan, and that during the scan no records in the key range of the scan are inserted, deleted, or modified.
    >

    You said:
    -- the scan will first encounter the old LN first, check Btree Index and return it since its a valid live entry.
    -- Concurrent update logs a new LN
    -- DOScan hits the new LN at a later point, Btree index is valid again and returns new LN also
    How is this prevented? In general, does a DOScan starting at time 't' have the semantics that it returns live k-v pairs as of time t? If not, can you explicitly the semantics provided?
    What you describe above isn't how it processes LNs. It doesn't scan the disk and check the Btree. It walks the Btree from the top down. As it does so, it accumulates LSNs (disk addresses) in a memory buffer. When the configured memory limited is exceeded, it sorts the LSNs in disk order, fetches them in that order, and they are returned via the cursor.
    Also, the lookup on the Btree index, does this happen on the memory buffer for the DOScan or do this bring in new index nodes into the regular BDB cache?
    The process starts with the root of the Btree in the JE cache and proceeds as described in [url http://docs.oracle.com/cd/E17277_02/html/java/com/sleepycat/je/DiskOrderedCursor.html]DiskOrderedCursor:
    >
    When the DiskOrderedCursor is first created, it is "seeded" with in-memory internal B-Tree nodes. This seeding causes the root and all seed INs to be latched for read, for use by the cursor. Seeded B-Tree nodes help the efficiency of the cursor, but penalize any concurrent update operations which require write locks on those nodes. DiskOrderedCursorConfig.setMaxSeedMillisecs(long) can be used to limit the time spent on the seeding process in exchange for reduced performance during the cursor walk.
    >

    Thereafter the traversal uses the Btree nodes on disk rather than in cache, as I described further above.

    Does this answer your question?
    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
  • 27. Re: Quick question on DiskOrderedCursorConfig#setInternalMemoryLimit
    vinothchandar Newbie
    Currently Being Moderated
    Thanks for the explanation. Clarifies a lot.

    Thanks
    Vinoth
1 2 Previous Next

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points