Forum Stats

  • 3,770,456 Users
  • 2,253,116 Discussions
  • 7,875,463 Comments

Discussions

BerkeleyDB LOG_INTEGRITY errors

2641727
2641727 Member Posts: 3
edited Mar 28, 2014 11:50AM in Berkeley DB Java Edition

Greetings!

We run BerkeleyDB embedded in our java application. Everything runs fine normally. Trouble happens if we run two instances of the same application. The folders where database files are created are separate for both. But it seems that when one stops something deletes the database files of the running one and it crashes.

A little more detail on how we run things:

init.d script (1) --> Starts a python module as root (2) ---> Starts a Java program as a regular user (3) ---> Forks off java programs that use individual instances of BerkeleyDB (4)

So if we start (3) manually things seem to run fine even if we have two instances of 4. It is only when the full thing runs that the error happens - the first process runs fine, the second process starts and completes and then the first process crashes because the entire content of its database folder is gone.

One more observation - if we set up the HOME environment variable to "/root" before starting (3) above - things do not fail. We cannot do that for obvious reasons, but this is something we found by chance.

The stack trace is attached below. If anyone can throw some light on what may be causing this that will be very very helpful.


Regards,
Arnab.


Caused by: com.sleepycat.je.EnvironmentFailureException: (JE 5.0.103) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.103) db com.sleepycat.je.EnvironmentFailureException: (JE 5.0.103) db java.io.FileNotFoundException: db/00000001.jdb (No such file or directory) LOG_INTEGRITY: Log information is incorrect, problem is likely persistent. UNEXPECTED_EXCEPTION_FATAL: Unexpected internal Exception, unable to continue. Environment is invalid and must be closed.

        at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:246)

        at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1618)

        at com.sleepycat.je.dbi.CursorImpl.checkEnv(CursorImpl.java:2859)

        at com.sleepycat.je.Cursor.checkEnv(Cursor.java:4181)

        at com.sleepycat.je.Cursor.close(Cursor.java:517)

        at com.sleepycat.je.Database.putInternal(Database.java:1584)

        at com.sleepycat.je.Database.put(Database.java:1448)

        at com.veracode.webscan.scanner.BerkeleyDbStorage$Db.put(BerkeleyDbStorage.java:585)

        at com.veracode.webscan.scanner.BerkeleyDbStorage$Db.put(BerkeleyDbStorage.java:615)

        at com.veracode.webscan.scanner.BerkeleyDbStorage$Db.access$3400(BerkeleyDbStorage.java:568)

        at com.veracode.webscan.scanner.BerkeleyDbStorage$NodeProviderImpl.setProp(BerkeleyDbStorage.java:850)

        at com.veracode.webscan.scanner.BerkeleyDbStorage$Node.setProp(BerkeleyDbStorage.java:148)

        at com.veracode.dscancommon.util.BaseMutablePropsAsMutableProps.setProp(BaseMutablePropsAsMutableProps.java:62)

        ... 34 more

Caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 5.0.103) db com.sleepycat.je.EnvironmentFailureException: (JE 5.0.103) db java.io.FileNotFoundException: db/00000001.jdb (No such file or directory) LOG_INTEGRITY: Log information is incorrect, problem is likely persistent. UNEXPECTED_EXCEPTION_FATAL: Unexpected internal Exception, unable to continue. Environment is invalid and must be closed.

        at com.sleepycat.je.EnvironmentFailureException.unexpectedException(EnvironmentFailureException.java:351)

        at com.sleepycat.je.log.LogBufferPool.writeBufferToFile(LogBufferPool.java:443)

        at com.sleepycat.je.log.LogBufferPool.writeDirty(LogBufferPool.java:370)

        at com.sleepycat.je.log.LogManager.multiLog(LogManager.java:426)

        at com.sleepycat.je.log.LogManager.log(LogManager.java:350)

        at com.sleepycat.je.tree.IN.logInternal(IN.java:3547)

        at com.sleepycat.je.tree.IN.optionalLog(IN.java:3494)

        at com.sleepycat.je.tree.IN.splitInternal(IN.java:2737)

        at com.sleepycat.je.tree.IN.split(IN.java:2535)

        at com.sleepycat.je.tree.Tree.forceSplit(Tree.java:1992)

        at com.sleepycat.je.tree.Tree.searchSubTreeSplitsAllowed(Tree.java:1768)

        at com.sleepycat.je.tree.Tree.searchSplitsAllowed(Tree.java:1318)

        at com.sleepycat.je.tree.Tree.findBinForInsert(Tree.java:2246)

        at com.sleepycat.je.dbi.CursorImpl.putInternal(CursorImpl.java:875)

        at com.sleepycat.je.dbi.CursorImpl.put(CursorImpl.java:816)

        at com.sleepycat.je.Cursor.putAllowPhantoms(Cursor.java:2397)

        at com.sleepycat.je.Cursor.putNoNotify(Cursor.java:2352)

        at com.sleepycat.je.Cursor.putNotify(Cursor.java:2260)

        at com.sleepycat.je.Cursor.putNoDups(Cursor.java:2149)

        at com.sleepycat.je.Cursor.putInternal(Cursor.java:2117)

        at com.sleepycat.je.Database.putInternal(Database.java:1580)

        ... 41 more

Caused by: com.sleepycat.je.EnvironmentFailureException: (JE 5.0.103) db java.io.FileNotFoundException: db/00000001.jdb (No such file or directory) LOG_INTEGRITY: Log information is incorrect, problem is likely persistent.

        at com.sleepycat.je.log.FileManager$LogEndFileDescriptor.getWritableFile(FileManager.java:2981)

        at com.sleepycat.je.log.FileManager$LogEndFileDescriptor.access$200(FileManager.java:2682)

        at com.sleepycat.je.log.FileManager.writeLogBuffer(FileManager.java:1607)

        at com.sleepycat.je.log.LogBufferPool.writeBufferToFile(LogBufferPool.java:411)

        ... 60 more

Caused by: java.io.FileNotFoundException: db/00000001.jdb (No such file or directory)

        at java.io.RandomAccessFile.open(Native Method)

        at java.io.RandomAccessFile.<init>(Unknown Source)

        at java.io.RandomAccessFile.<init>(Unknown Source)

        at com.sleepycat.je.log.FileManager$DefaultRandomAccessFile.<init>(FileManager.java:3181)

        at com.sleepycat.je.log.FileManager$6.createFile(FileManager.java:3209)

        at com.sleepycat.je.log.FileManager.openFileHandle(FileManager.java:1278)

        at com.sleepycat.je.log.FileManager.makeFileHandle(FileManager.java:1245)

        at com.sleepycat.je.log.FileManager.access$1500(FileManager.java:76)

        at com.sleepycat.je.log.FileManager$LogEndFileDescriptor.getWritableFile(FileManager.java:2959)

        ... 63 more

Best Answer

  • Greybird-Oracle
    Greybird-Oracle Member Posts: 2,690
    Accepted Answer

    Based on your description I'm guessing that either your script/app or something external is deleting the files.  It is very unlikely that JE is deleting all the files in the environment directory.  We've never had a report of such a problem anyway, and it's difficult for me to imagine how it could happen.  For example, JE never cleans/deletes the two most recent files.

    I should also mention that we do not certify the product on Windows, so there is a possibility that you're running into a file locking issue that we don't know about.  Normally JE uses a lock file to ensure that only one JE process (JVM instance) can write to a given directory.  If you are accidentally allowing two JVMs to write to the same directory, and Windows file locking is misbehaving, that might cause the problem.  However, I think this is unlikely.

    --mark

Answers

  • Greybird-Oracle
    Greybird-Oracle Member Posts: 2,690
    Accepted Answer

    Based on your description I'm guessing that either your script/app or something external is deleting the files.  It is very unlikely that JE is deleting all the files in the environment directory.  We've never had a report of such a problem anyway, and it's difficult for me to imagine how it could happen.  For example, JE never cleans/deletes the two most recent files.

    I should also mention that we do not certify the product on Windows, so there is a possibility that you're running into a file locking issue that we don't know about.  Normally JE uses a lock file to ensure that only one JE process (JVM instance) can write to a given directory.  If you are accidentally allowing two JVMs to write to the same directory, and Windows file locking is misbehaving, that might cause the problem.  However, I think this is unlikely.

    --mark

  • 2641727
    2641727 Member Posts: 3

    Thanks a lot for your mail. I will see if something else is doing this. BTW, we are running this in a linux environment.

    Regards,
    Arnab.

  • 2641727
    2641727 Member Posts: 3

    Mark - you were right ... some old code (separate) was doing a cleaning job removing files which deleted these... thanks!

  • Good, glad you resolved it!

    --mark

This discussion has been closed.