We recently rewrote the BDB storage layer for Voldemort (as advised by the JE team) and seeing good improvements.. Idea is to move as much data as possible off the heap, and hence we attempted to set lazy_migration to off.As you know I believe lazy migration should always be off. It has no advantage and several disadvantages.
1. If I turn lazy_migration on however, backlog disappears right away. So wondering if there is a known cleaning issue in 4.1.x.I suspect it is not a cleaning issue, it is simply that JE 4.1 is much less efficient -- there is much more metadata writing -- and therefore more cleaner threads are needed to keep up.
2. Setting CLEANER_BACKGROUND_PROACTIVE_MIGRATION on with lazy_migration=off, also helps. Some help on what exactly happens (esp its impact on foreground operations) when this is turned on would help.Same thing. With proactive migration, the evictor threads and app threads are assisting with cleaning during eviction. It seems very likely that you have a lot of eviction.
3. BDB5 does not seem to have this issue. Again, it would be nice to know what improvements one can expect in BDB5 in terms of log cleaning, to make sense of this.JE 5 writes much less metadata (Btree internal nodes) so less cleaning is necessary and less cleaner threads are necessary to keep up.
Some advice/clarification appreciated.Perhaps with a low write rate performance is not a big consideration and you'd prefer to use lazy migration in order for the cleaner to keep up. But in general it is much more efficient to increase the number of cleaner threads and turn off lazy migration.