Just a random thought, but have you tried this with READ_UNCOMMITTED turned off?Thanks for the thought. We are working on this off-forum and I have a potential fix, I'll report back here when it's final.
Without looking at the internals in the slightest, it does seem feasible that a disk ordered cursor could end up on a key that ends up not existing after a transaction abort?
I am getting rid of duplicates in the new dataThe problem is, that's an ambiguous statement. It is possible to call DatabaseConfig.setSortedDuplicates(true) and still have only one record per key (no duplicates). That's why I need to know for certain that you're not calling DatabaseConfig.setSortedDuplicates(true).
I have some questions on the semantics offered by the DiskOrderedScan.The high level description of this is in the [url http://docs.oracle.com/cd/E17277_02/html/java/com/sleepycat/je/Database.html#openCursor(com.sleepycat.je.DiskOrderedCursorConfig)]openCursor javadoc:
-- the scan will first encounter the old LN first, check Btree Index and return it since its a valid live entry.What you describe above isn't how it processes LNs. It doesn't scan the disk and check the Btree. It walks the Btree from the top down. As it does so, it accumulates LSNs (disk addresses) in a memory buffer. When the configured memory limited is exceeded, it sorts the LSNs in disk order, fetches them in that order, and they are returned via the cursor.
-- Concurrent update logs a new LN
-- DOScan hits the new LN at a later point, Btree index is valid again and returns new LN also
How is this prevented? In general, does a DOScan starting at time 't' have the semantics that it returns live k-v pairs as of time t? If not, can you explicitly the semantics provided?
Also, the lookup on the Btree index, does this happen on the memory buffer for the DOScan or do this bring in new index nodes into the regular BDB cache?The process starts with the root of the Btree in the JE cache and proceeds as described in [url http://docs.oracle.com/cd/E17277_02/html/java/com/sleepycat/je/DiskOrderedCursor.html]DiskOrderedCursor: