In that case, the problem occurred only after an OutOfMemoryError and only when configuring direct NIO (LOG_USE_NIO and LOG_DIRECT_NIO), which is deprecated.
Do you see any exceptions before the problem occurs?
We have a fix for this. However, when this was reported earlier we could not find a case where it could occur without a fatal error (such as OutOfMemoryError) preceding it, so we did not apply the fix to the JE 3.3 release. The fact that the problem has occurred for you means that we misjudged the scope of the problem.
It looks like it can also occur under certain conditions after an environment using the shared cache is closed, and the remaining environments using the shared cache have no evictable nodes, and the cache overflows.
Please send me email and I'll give you a JE 3.3 jar file with a fix to try. I'm mark.hayes at the obvious (oracle.com).
No, I don't use direct NIO and I didn't see any exceptions before. My application was opening an environment with WriteNoSync = true which was not closed properly. As far as I understand, it tried to open an existing database and doing that it got internally into the infinite cycle. This environment shares cache with another one which is configured by default with WriteNoSync = false.
Actually, I've already found a workaround, I've set the lruOnly = true, and the application started successfully.
Thanks for the information and for being willing to try the fix I sent you.
I'm afraid that setting lruOnly = true is simply changing the timing of eviction a little bit. The problem is very much timing dependent. If you were to change the cache size slightly, that might also make the problem go away. But I'm afraid it is still there and could occur again.
For others who may be interested: This occurs when using EnvironmentConfig.setSharedCache(true) and is very timing dependent. We will be putting a fix for it into the next JE 3.3 release.