This discussion is archived
6 Replies Latest reply: Mar 19, 2012 2:09 PM by greybird RSS

Cleaner thread prematurely removing log file

837802 Newbie
Currently Being Moderated
Dear JE community,

I´d like to post a stack trace. It seems like the cleaner thread had removed a log file that was still needed.

We are running JE 4.0.71, Solaris 10 x86, Java 1.6u20 HotSpot 64 bit, ZFS.

The database environment had to be closed with this exception:
com.sleepycat.je.EnvironmentFailureException: (JE 4.0.71) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 4.0.71) /klikit/database fetchTarget of 0x2c3/0x564bb4 parent IN=29622582 IN class=com.sleepycat.je.tree.DBIN lastFullVersion=0x2dd/0x38ae2f parent.getDirty()=true state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.

The root cause was:
Caused by: java.io.FileNotFoundException: /klikit/database/000002c3.jdb (No such file or directory)

Here is the full stack trace:

[#|2011-02-02T03:53:14.179+0100|SEVERE|sun-appserver2.1|javax.enterprise.system.container.web|_ThreadID=17;_ThreadName=httpWorkerThread-80-1;_RequestID=df4c9c76-f069-4098-be75-566b9f216008;|StandardWrapperValve[dispatcher]: PWC1406: Servlet.service() for servlet dispatcher threw exception
org.apache.velocity.exception.MethodInvocationException: Invocation of method 'getSportResultsForAssociation' in class lu.luggel.web.model.ContentManager threw exception lu.kpmg.core.db.store.StoreException: com.sleepycat.je.EnvironmentFailureException: (JE 4.0.71) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 4.0.71) /klikit/database fetchTarget of 0x2c3/0x564bb4 parent IN=29622582 IN class=com.sleepycat.je.tree.DBIN lastFullVersion=0x2dd/0x38ae2f parent.getDirty()=true state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed. at municipality/sportResults/sportResultsForAssociation.html[line 3, column 27]
     at org.apache.velocity.runtime.parser.node.ASTMethod.handleInvocationException(ASTMethod.java:337)
     at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:284)
     at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:252)
     at org.apache.velocity.runtime.parser.node.ASTReference.value(ASTReference.java:493)
     at org.apache.velocity.runtime.parser.node.ASTExpression.value(ASTExpression.java:71)
     at org.apache.velocity.runtime.parser.node.ASTSetDirective.render(ASTSetDirective.java:142)
     at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:336)
     at org.apache.velocity.runtime.directive.Parse.render(Parse.java:260)
     at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:175)
     at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:336)
     at org.apache.velocity.Template.merge(Template.java:328)
     at org.apache.velocity.Template.merge(Template.java:235)
     at org.springframework.web.servlet.view.velocity.VelocityLayoutView.renderScreenContent(VelocityLayoutView.java:180)
     at org.springframework.web.servlet.view.velocity.VelocityLayoutView.doRender(VelocityLayoutView.java:150)
     at org.springframework.web.servlet.view.velocity.VelocityView.renderMergedTemplateModel(VelocityView.java:291)
     at org.springframework.web.servlet.view.AbstractTemplateView.renderMergedOutputModel(AbstractTemplateView.java:167)
     at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:250)
     at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1060)
     at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:798)
     at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:716)
     at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:647)
     at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:552)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
     at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:427)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:333)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
     at com.sun.appserv.web.cache.filter.CachingFilter.doFilter(CachingFilter.java:291)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
     at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
     at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
     at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:313)
     at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:287)
     at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:218)
     at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
     at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
     at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
     at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:98)
     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:222)
     at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
     at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
     at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
     at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166)
     at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
     at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
     at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
     at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
     at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:291)
     at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:666)
     at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:597)
     at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:872)
     at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341)
     at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263)
     at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214)
     at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:264)
     at com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:117)
Caused by: lu.kpmg.core.db.store.StoreException: com.sleepycat.je.EnvironmentFailureException: (JE 4.0.71) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 4.0.71) /klikit/database fetchTarget of 0x2c3/0x564bb4 parent IN=29622582 IN class=com.sleepycat.je.tree.DBIN lastFullVersion=0x2dd/0x38ae2f parent.getDirty()=true state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.
     at lu.kpmg.core.db.store.Store.execute(Store.java:230)
     at lu.kpmg.core.db.store.RecordManager.forEach(RecordManager.java:164)
     at lu.kpmg.core.db.store.RecordManager.queryRange(RecordManager.java:158)
     at lu.kpmg.core.db.store.RecordManager.queryAll(RecordManager.java:147)
     at lu.luggel.web.model.ContentManager.getSportResultsForAssociation(ContentManager.java:436)
     at sun.reflect.GeneratedMethodAccessor160.invoke(Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:597)
     at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:389)
     at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:378)
     at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:270)
     ... 58 more
Caused by: com.sleepycat.je.EnvironmentFailureException: (JE 4.0.71) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 4.0.71) /klikit/database fetchTarget of 0x2c3/0x564bb4 parent IN=29622582 IN class=com.sleepycat.je.tree.DBIN lastFullVersion=0x2dd/0x38ae2f parent.getDirty()=true state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.
     at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:197)
     at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1392)
     at com.sleepycat.je.dbi.CursorImpl.checkEnv(CursorImpl.java:2813)
     at com.sleepycat.je.Cursor.checkEnv(Cursor.java:2846)
     at com.sleepycat.je.Cursor.close(Cursor.java:439)
     at lu.kpmg.core.db.store.Store.execute(Store.java:226)
     ... 68 more
Caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 4.0.71) /klikit/database fetchTarget of 0x2c3/0x564bb4 parent IN=29622582 IN class=com.sleepycat.je.tree.DBIN lastFullVersion=0x2dd/0x38ae2f parent.getDirty()=true state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.
     at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1241)
     at com.sleepycat.je.tree.BIN.fetchTarget(BIN.java:1300)
     at com.sleepycat.je.dbi.CursorImpl.fetchCurrent(CursorImpl.java:2362)
     at com.sleepycat.je.dbi.CursorImpl.fetchCurrent(CursorImpl.java:2389)
     at com.sleepycat.je.dbi.CursorImpl.getCurrentAlreadyLatched(CursorImpl.java:1410)
     at com.sleepycat.je.Cursor.searchInternal(Cursor.java:2174)
     at com.sleepycat.je.Cursor.searchAllowPhantoms(Cursor.java:2058)
     at com.sleepycat.je.Cursor.search(Cursor.java:1926)
     at com.sleepycat.je.SecondaryCursor.search(SecondaryCursor.java:1364)
     at com.sleepycat.je.SecondaryCursor.getSearchKeyRange(SecondaryCursor.java:1176)
     at lu.kpmg.core.db.store.Store.execute(Store.java:197)
     ... 68 more
Caused by: java.io.FileNotFoundException: /klikit/database/000002c3.jdb (No such file or directory)
     at java.io.RandomAccessFile.open(Native Method)
     at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
     at java.io.RandomAccessFile.<init>(RandomAccessFile.java:98)
     at com.sleepycat.je.log.FileManager$1.<init>(FileManager.java:992)
     at com.sleepycat.je.log.FileManager.openFileHandle(FileManager.java:991)
     at com.sleepycat.je.log.FileManager.getFileHandle(FileManager.java:887)
     at com.sleepycat.je.log.LogManager.getLogSource(LogManager.java:1073)
     at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:779)
     at com.sleepycat.je.log.LogManager.getLogEntryAllowInvisibleAtRecovery(LogManager.java:743)
     at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1225)
     ... 78 more



The exception occured several times on a day. The root cause was always related to the missing file "000002c3.jdb".

When we run the DBVerify-Tool on that database, it fails with the following exception:

Checking tree for key
Error encountered (continuing):
com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 4.1.7) d:\transfer\database_FileNotFound fetchTarget of 0x2c3/0x4d677e parent IN=29622782 IN class=com.sleepycat.je.tree.DBIN lastFullVersion=0x2de/0x8ccc00 parent.getDirty()=false state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.
Error Key 592d83a084 Y-\83\a0\84
Error Data 5a4e3398 ZN3\98
Error encountered (continuing):
com.sleepycat.je.EnvironmentFailureException: (JE 4.1.7) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 4.1.7) d:\transfer\database_FileNotFound fetchTarget of 0x2c3/0x4d677e parent IN=29622782 IN class=com.sleepycat.je.tree.DBIN lastFullVersion=0x2de/0x8ccc00 parent.getDirty()=false state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed. fetchTarget of 0x2de/0x78b409 parent IN=29622782 IN class=com.sleepycat.je.tree.DBIN lastFullVersion=0x2de/0x8ccc00 parent.getDirty()=false state=0
Error Key 592d83a084 Y-\83\a0\84
Error Data 5a4e341f ZN4\1f
Error encountered (continuing):
com.sleepycat.je.EnvironmentFailureException: (JE 4.1.7) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 4.1.7) d:\transfer\database_FileNotFound fetchTarget of 0x2c3/0x4d677e parent IN=29622782 IN class=com.sleepycat.je.tree.DBIN lastFullVersion=0x2de/0x8ccc00 parent.getDirty()=false state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed. fetchTarget of 0x2de/0x7a568e parent IN=29622781 IN class=com.sleepycat.je.tree.DIN lastFullVersion=0x2de/0x8cce1a parent.getDirty()=false state=0
Error Key 592d87a06842fed4e17b2afa Y-\87\a0hB\fe\d4\e1{*\fa
Error Data UNKNOWN
Error encountered (continuing):
com.sleepycat.je.EnvironmentFailureException: (JE 4.1.7) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 4.1.7) d:\transfer\database_FileNotFound fetchTarget of 0x2c3/0x4d677e parent IN=29622782 IN class=com.sleepycat.je.tree.DBIN lastFullVersion=0x2de/0x8ccc00 parent.getDirty()=false state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed. fetchTarget of 0x2de/0x79dae8 parent IN=16651289 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0x2de/0x91e179 parent.getDirty()=false state=0
Error Key 592d88a06842fed362b24baa Y-\88\a0hB\fe\d3b\b2K\aa
Error Data UNKNOWN
Error encountered (continuing):
...
...
com.sleepycat.je.EnvironmentFailureException: (JE 4.1.7) Environment must be clo
sed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invali
d because of previous exception: (JE 4.1.7) d:\transfer\database_FileNotFound fe
tchTarget of 0x2c3/0x4d677e parent IN=29622782 IN class=com.sleepycat.je.tree.DB
IN lastFullVersion=0x2de/0x8ccc00 parent.getDirty()=false state=0 LOG_FILE_NOT_F
OUND: Log file missing, log is likely invalid. Environment is invalid and must b
e closed.
at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:196)
at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1439)
at com.sleepycat.je.Database.checkEnv(Database.java:1778)
at com.sleepycat.je.Database.closeInternal(Database.java:377)
at com.sleepycat.je.Database.close(Database.java:314)
at com.sleepycat.je.util.DbVerify.verify(DbVerify.java:293)
at com.sleepycat.je.util.DbVerify.main(DbVerify.java:98)
Caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid bec
ause of previous exception: (JE 4.1.7) d:\transfer\database_FileNotFound fetchTa
rget of 0x2c3/0x4d677e parent IN=29622782 IN class=com.sleepycat.je.tree.DBIN la
stFullVersion=0x2de/0x8ccc00 parent.getDirty()=false state=0 LOG_FILE_NOT_FOUND:
Log file missing, log is likely invalid. Environment is invalid and must be clo
sed.
at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1337)
at com.sleepycat.je.tree.BIN.fetchTarget(BIN.java:1367)
at com.sleepycat.je.dbi.CursorImpl.fetchCurrent(CursorImpl.java:2499)
at com.sleepycat.je.dbi.CursorImpl.fetchCurrent(CursorImpl.java:2526)
at com.sleepycat.je.dbi.CursorImpl.getCurrentAlreadyLatched(CursorImpl.java:1545)
at com.sleepycat.je.dbi.CursorImpl.getNextWithKeyChangeStatus(CursorImpl.java:1692)
at com.sleepycat.je.dbi.CursorImpl.getNext(CursorImpl.java:1617)
at com.sleepycat.je.dbi.DatabaseImpl.walkDatabaseTree(DatabaseImpl.java:1473)
at com.sleepycat.je.dbi.DatabaseImpl.verify(DatabaseImpl.java:1420)
at com.sleepycat.je.util.DbVerify.verifyOneDbImpl(DbVerify.java:366)
at com.sleepycat.je.util.DbVerify.verify(DbVerify.java:285)
... 1 more
Caused by: java.io.FileNotFoundException: d:\transfer\database_FileNotFound\000002c3.jdb (The system cannot find the file specified)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(Unknown Source)
at java.io.RandomAccessFile.<init>(Unknown Source)
at com.sleepycat.je.log.FileManager$1.<init>(FileManager.java:995)
at com.sleepycat.je.log.FileManager.openFileHandle(FileManager.java:994)
at com.sleepycat.je.log.FileManager.getFileHandle(FileManager.java:890)
at com.sleepycat.je.log.LogManager.getLogSource(LogManager.java:1074)
at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:778)
at com.sleepycat.je.log.LogManager.getLogEntryAllowInvisibleAtRecovery(LogManager.java:742)
at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1320)
... 11 more
Exit status = false



It seems like the cleaner thread had removed the file "000002c3.jdb", while it was still needed.
As the database was running with the "je.cleaner.expunge=false" setting, the file was still kept as "000002c3.del".

So we renamed that file back to "000002c3.jdb".
Running the dbVerify tool on that database succeeded (Exit status = true).
The previously mentioned "FileNotFoundException" did also not occur any more.


There is an additional piece of information that might be helpful.
For the database with the missing file the dbverify tool had failed for a record with "Error Key 592d83a084".

Before having the "FileNotFoundException" we found a record in the database that has a corrupted secondary index. We found out that
the key 592d83a084 is pointing to this record.
Deleting or updating this record fails with the following exception:

Caused by: com.sleepycat.je.SecondaryIntegrityException: (JE 4.0.71) Secondary is corrupt: the primary record contains a key that is not present in the secondary
at com.sleepycat.je.SecondaryDatabase.deleteKey(SecondaryDatabase.java:937)
at com.sleepycat.je.SecondaryDatabase.updateSecondary(SecondaryDatabase.java:900)
at com.sleepycat.je.SecondaryTrigger.databaseUpdated(SecondaryTrigger.java:42)
at com.sleepycat.je.Database.notifyTriggers(Database.java:2004)
at com.sleepycat.je.Cursor.putNotify(Cursor.java:1692)
at com.sleepycat.je.Cursor.putInternal(Cursor.java:1616)
at com.sleepycat.je.Database.putInternal(Database.java:1178)
at com.sleepycat.je.Database.put(Database.java:1050)
at lu.kpmg.core.db.store.Store.store(Store.java:325)
... 51 more

I hope this helps to uncover the root cause of the problem and make JE even better.

Holger
  • 1. Re: Cleaner thread prematurely removing log file
    greybird Expert
    Currently Being Moderated
    Hi Holger,

    There are several fixes for bugs that could cause this problem in the JE 4.0 product.

    http://download.oracle.com/otndocs/products/berkeleydb/html/je/je-40117-changelog.html

    Reference numbers for bugs that could cause this problem are: [#17252], [#19422], [#18227]. These fixes were all made after the JE release you're using, JE 4.0.71.

    Since I can tell from the stack trace you posted that you are using duplicates, at least one of these bugs may apply to your application.

    With this type of bug, it is usually not possible to continue to use the JE environment where the problem occurred. I suggest that you revert to a backup that was made before this problem occurred and then upgrade to the latest version of JE.

    You could upgrade to JE 4.0.117, the latest release in the JE 4.0 line, but it would be much better to upgrade to JE 4.1.7 to get the benefit of other performance improvements and bug fixes that have been made in JE 4.1. See: http://download.oracle.com/docs/cd/E17277_02/html/changelog.html

    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
  • 2. Re: Cleaner thread prematurely removing log file
    837802 Newbie
    Currently Being Moderated
    Hi Mark,
    Thanks for your explanations. We will consider to upgrade to 4.1.7

    Holger
  • 3. Re: Cleaner thread prematurely removing log file
    924932 Newbie
    Currently Being Moderated
    I use je 4.1.10 yet I am seeing similar behavior. As this happens intermittently, I haven't figured out a proper steps to reproduce the issue in my application. This is the stack trace:
    Msg:(JE 4.1.10) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 4.1.10) nodePrimary(1):/www/a/logs/mcom/bdb/mcom-primary fetchTarget of 0xc2/0x787838 parent IN=4928 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0x18b/0x6b60d4 parent.getDirty()=false state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.
    com.sleepycat.je.EnvironmentFailureException: (JE 4.1.10) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 4.1.10) nodePrimary(1):/www/a/logs/mcom/bdb/mcom-primary fetchTarget of 0xc2/0x787838 parent IN=4928 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0x18b/0x6b60d4 parent.getDirty()=false state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.
    at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:196)
    at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1455)
    at com.sleepycat.je.Database.checkEnv(Database.java:1778)
    at com.sleepycat.je.Database.openCursor(Database.java:625)
    at com.sleepycat.je.SecondaryDatabase.openCursor(SecondaryDatabase.java:436)
    at com.sleepycat.je.SecondaryDatabase.openCursor(SecondaryDatabase.java:125)
    at com.sleepycat.persist.BasicIndex.cursor(BasicIndex.java:232)
    at com.sleepycat.persist.BasicIndex.cursor(BasicIndex.java:223)
    at com.sleepycat.persist.BasicIndex.entities(BasicIndex.java:186)
    at com.sleepycat.persist.SecondaryIndex.entities(SecondaryIndex.java:765)
  • 4. Re: Cleaner thread prematurely removing log file
    greybird Expert
    Currently Being Moderated
    When you say it happens intermittently, do you mean that it stops happening when you restart? Or is it a persistent problem and requires restore from backup to correct the problem?

    --mark                                                                                                                                                                                                                                                                                                                                                                                               
  • 5. Re: Cleaner thread prematurely removing log file
    924932 Newbie
    Currently Being Moderated
    The stack trace is part of a daily batch processing job, running concurrently. On most days the job runs correctly, only a couple of times have I seen this exception happen. But when i do see it, I need to restore from backup for the job to work again.
  • 6. Re: Cleaner thread prematurely removing log file
    greybird Expert
    Currently Being Moderated
    The only bug we know of that could cause that problem is [#20732], fixed in JE 4.1.17.
    http://download.oracle.com/otndocs/products/berkeleydb/html/je/je-4.1.17_changelog.html

    I suggest the following course of action:
    1) Upgrade to 4.1.17.
    2) If the problem occurs again, save the problem environment home directory (the .jdb files) and post back here again. If you can send me .jdb files, I can work with you to try to diagnose the problem.

    Thanks,
    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points