BerkeleyDB LOG_INTEGRITY errors

Greetings!

We run BerkeleyDB embedded in our java application. Everything runs fine normally. Trouble happens if we run two instances of the same application. The folders where database files are created are separate for both. But it seems that when one stops something deletes the database files of the running one and it crashes.

A little more detail on how we run things:

init.d script (1) --> Starts a python module as root (2) ---> Starts a Java program as a regular user (3) ---> Forks off java programs that use individual instances of BerkeleyDB (4)

So if we start (3) manually things seem to run fine even if we have two instances of 4. It is only when the full thing runs that the error happens - the first process runs fine, the second process starts and completes and then the first process crashes because the entire content of its database folder is gone.

One more observation - if we set up the HOME environment variable to "/root" before starting (3) above - things do not fail. We cannot do that for obvious reasons, but this is something we found by chance.

The stack trace is attached below. If anyone can throw some light on what may be causing this that will be very very helpful.

Regards,
Arnab.

Caused by: com.sleepycat.je.EnvironmentFailureException: (JE 5.0.103) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.103) db com.sleepycat.je.EnvironmentFailureException: (JE 5.0.103) db java.io.FileNotFoundException: db/00000001.jdb (No such file or directory) LOG_INTEGRITY: Log information is incorrect, problem is likely persistent. UNEXPECTED_EXCEPTION_FATAL: Unexpected internal Exception, unable to continue. Environment is invalid and must be closed.

        at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:246)

        at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1618)

        at com.sleepycat.je.dbi.CursorImpl.checkEnv(CursorImpl.java:2859)

        at com.sleepycat.je.Cursor.checkEnv(Cursor.java:4181)

        at com.sleepycat.je.Cursor.close(Cursor.java:517)

        at com.sleepycat.je.Database.putInternal(Database.java:1584)

        at com.sleepycat.je.Database.put(Database.java:1448)

        at com.veracode.webscan.scanner.BerkeleyDbStorage$Db.put(BerkeleyDbStorage.java:585)

        at com.veracode.webscan.scanner.BerkeleyDbStorage$Db.put(BerkeleyDbStorage.java:615)

        at com.veracode.webscan.scanner.BerkeleyDbStorage$Db.access$3400(BerkeleyDbStorage.java:568)

        at com.veracode.webscan.scanner.BerkeleyDbStorage$NodeProviderImpl.setProp(BerkeleyDbStorage.java:850)

        at com.veracode.webscan.scanner.BerkeleyDbStorage$Node.setProp(BerkeleyDbStorage.java:148)

        at com.veracode.dscancommon.util.BaseMutablePropsAsMutableProps.setProp(BaseMutablePropsAsMutableProps.java:62)

        ... 34 more

Caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.103) db com.sleepycat.je.EnvironmentFailureException: (JE 5.0.103) db java.io.FileNotFoundException: db/00000001.jdb (No such file or directory) LOG_INTEGRITY: Log information is incorrect, problem is likely persistent. UNEXPECTED_EXCEPTION_FATAL: Unexpected internal Exception, unable to continue. Environment is invalid and must be closed.

        at com.sleepycat.je.EnvironmentFailureException.unexpectedException(EnvironmentFailureException.java:351)

        at com.sleepycat.je.log.LogBufferPool.writeBufferToFile(LogBufferPool.java:443)

        at com.sleepycat.je.log.LogBufferPool.writeDirty(LogBufferPool.java:370)

        at com.sleepycat.je.log.LogManager.multiLog(LogManager.java:426)

        at com.sleepycat.je.log.LogManager.log(LogManager.java:350)

        at com.sleepycat.je.tree.IN.logInternal(IN.java:3547)

        at com.sleepycat.je.tree.IN.optionalLog(IN.java:3494)

        at com.sleepycat.je.tree.IN.splitInternal(IN.java:2737)

        at com.sleepycat.je.tree.IN.split(IN.java:2535)

        at com.sleepycat.je.tree.Tree.forceSplit(Tree.java:1992)

        at com.sleepycat.je.tree.Tree.searchSubTreeSplitsAllowed(Tree.java:1768)

        at com.sleepycat.je.tree.Tree.searchSplitsAllowed(Tree.java:1318)

        at com.sleepycat.je.tree.Tree.findBinForInsert(Tree.java:2246)

        at com.sleepycat.je.dbi.CursorImpl.putInternal(CursorImpl.java:875)

        at com.sleepycat.je.dbi.CursorImpl.put(CursorImpl.java:816)

        at com.sleepycat.je.Cursor.putAllowPhantoms(Cursor.java:2397)

        at com.sleepycat.je.Cursor.putNoNotify(Cursor.java:2352)

        at com.sleepycat.je.Cursor.putNotify(Cursor.java:2260)

        at com.sleepycat.je.Cursor.putNoDups(Cursor.java:2149)

        at com.sleepycat.je.Cursor.putInternal(Cursor.java:2117)

        at com.sleepycat.je.Database.putInternal(Database.java:1580)

        ... 41 more

Caused by: com.sleepycat.je.EnvironmentFailureException: (JE 5.0.103) db java.io.FileNotFoundException: db/00000001.jdb (No such file or directory) LOG_INTEGRITY: Log information is incorrect, problem is likely persistent.

        at com.sleepycat.je.log.FileManager$LogEndFileDescriptor.getWritableFile(FileManager.java:2981)

        at com.sleepycat.je.log.FileManager$LogEndFileDescriptor.access$200(FileManager.java:2682)

        at com.sleepycat.je.log.FileManager.writeLogBuffer(FileManager.java:1607)

        at com.sleepycat.je.log.LogBufferPool.writeBufferToFile(LogBufferPool.java:411)

        ... 60 more

Caused by: java.io.FileNotFoundException: db/00000001.jdb (No such file or directory)

        at java.io.RandomAccessFile.open(Native Method)

        at java.io.RandomAccessFile.<init>(Unknown Source)

        at java.io.RandomAccessFile.<init>(Unknown Source)

        at com.sleepycat.je.log.FileManager$DefaultRandomAccessFile.<init>(FileManager.java:3181)

        at com.sleepycat.je.log.FileManager$6.createFile(FileManager.java:3209)

        at com.sleepycat.je.log.FileManager.openFileHandle(FileManager.java:1278)

        at com.sleepycat.je.log.FileManager.makeFileHandle(FileManager.java:1245)

        at com.sleepycat.je.log.FileManager.access$1500(FileManager.java:76)

        at com.sleepycat.je.log.FileManager$LogEndFileDescriptor.getWritableFile(FileManager.java:2959)

        ... 63 more