The only other slightly exotic thing I'm using is a custom partial BTree comparator, however, it quite happily loaded/updated literally tens of millions of records for hours before the FileNotFound error cropped up, so it seems unlikely that would be the cause.Could you please post the source code for this comparator? It is the first suspect. An incorrect comparator could cause this error -- in fact an incorrect comparator could create any number of problems. And it could happen after loading a large number of records.
Thanks heaps to Mark for working through this with me.You're welcome. Thanks for following up and explaining it for the benefit of others. And I'm very glad it wasn't a JE bug!
My solution is to switch to using a secondary database for providing differentiated "uniqueness" vs "ordering".An index for uniqueness may be a good solution. But as you said in email, it adds significant overhead (memory and disk). This overhead can be minimized by keeping your keys (primary and secondary) as small as possible, and enabling key prefixing.
I'd also like to point out that adding a secondary isn't always the best choice. For example, if the number of keys with the same C1 value is fairly small, another way of checking for uniqueness (when inserting) is to iterate over them, looking for a match on C1:C3. The cost of this iteration may be less than the cost of maintaining a uniqueness index. To make this work, you'll have to use Serializable isolation during the iteration, to prevent another thread from inserting a key in that range.Yup, I've considered this as an alternative implementation. The number of possible keys sharing C1 is actually relatively small (<350) so it may end up being quicker to use this approach.
By the way, does this operate on literal prefixes (ie: at a byte array level) or does it consider the comparator? In my case, the literal prefix and effective prefix differ.It's just the literal prefix, sorry. To handle anything else there would need to be another callback that splits/combines a key into prefix and suffix, and we don't have such a callback.