6 Replies Latest reply: Mar 19, 2012 4:09 PM by Greybird-Oracle RSS

    Cleaner thread prematurely removing log file

    837802
      Dear JE community,

      I´d like to post a stack trace. It seems like the cleaner thread had removed a log file that was still needed.

      We are running JE 4.0.71, Solaris 10 x86, Java 1.6u20 HotSpot 64 bit, ZFS.

      The database environment had to be closed with this exception:
      com.sleepycat.je.EnvironmentFailureException: (JE 4.0.71) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 4.0.71) /klikit/database fetchTarget of 0x2c3/0x564bb4 parent IN=29622582 IN class=com.sleepycat.je.tree.DBIN lastFullVersion=0x2dd/0x38ae2f parent.getDirty()=true state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.

      The root cause was:
      Caused by: java.io.FileNotFoundException: /klikit/database/000002c3.jdb (No such file or directory)

      Here is the full stack trace:

      [#|2011-02-02T03:53:14.179+0100|SEVERE|sun-appserver2.1|javax.enterprise.system.container.web|_ThreadID=17;_ThreadName=httpWorkerThread-80-1;_RequestID=df4c9c76-f069-4098-be75-566b9f216008;|StandardWrapperValve[dispatcher]: PWC1406: Servlet.service() for servlet dispatcher threw exception
      org.apache.velocity.exception.MethodInvocationException: Invocation of method 'getSportResultsForAssociation' in class lu.luggel.web.model.ContentManager threw exception lu.kpmg.core.db.store.StoreException: com.sleepycat.je.EnvironmentFailureException: (JE 4.0.71) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 4.0.71) /klikit/database fetchTarget of 0x2c3/0x564bb4 parent IN=29622582 IN class=com.sleepycat.je.tree.DBIN lastFullVersion=0x2dd/0x38ae2f parent.getDirty()=true state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed. at municipality/sportResults/sportResultsForAssociation.html[line 3, column 27]
           at org.apache.velocity.runtime.parser.node.ASTMethod.handleInvocationException(ASTMethod.java:337)
           at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:284)
           at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:252)
           at org.apache.velocity.runtime.parser.node.ASTReference.value(ASTReference.java:493)
           at org.apache.velocity.runtime.parser.node.ASTExpression.value(ASTExpression.java:71)
           at org.apache.velocity.runtime.parser.node.ASTSetDirective.render(ASTSetDirective.java:142)
           at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:336)
           at org.apache.velocity.runtime.directive.Parse.render(Parse.java:260)
           at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:175)
           at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:336)
           at org.apache.velocity.Template.merge(Template.java:328)
           at org.apache.velocity.Template.merge(Template.java:235)
           at org.springframework.web.servlet.view.velocity.VelocityLayoutView.renderScreenContent(VelocityLayoutView.java:180)
           at org.springframework.web.servlet.view.velocity.VelocityLayoutView.doRender(VelocityLayoutView.java:150)
           at org.springframework.web.servlet.view.velocity.VelocityView.renderMergedTemplateModel(VelocityView.java:291)
           at org.springframework.web.servlet.view.AbstractTemplateView.renderMergedOutputModel(AbstractTemplateView.java:167)
           at org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:250)
           at org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1060)
           at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:798)
           at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:716)
           at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:647)
           at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:552)
           at javax.servlet.http.HttpServlet.service(HttpServlet.java:734)
           at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
           at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:427)
           at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:333)
           at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
           at com.sun.appserv.web.cache.filter.CachingFilter.doFilter(CachingFilter.java:291)
           at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
           at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
           at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
           at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
           at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)
           at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)
           at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:313)
           at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:287)
           at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:218)
           at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
           at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
           at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
           at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:98)
           at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:222)
           at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
           at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
           at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
           at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
           at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166)
           at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)
           at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)
           at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)
           at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1093)
           at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:291)
           at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:666)
           at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:597)
           at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:872)
           at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341)
           at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263)
           at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214)
           at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:264)
           at com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:117)
      Caused by: lu.kpmg.core.db.store.StoreException: com.sleepycat.je.EnvironmentFailureException: (JE 4.0.71) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 4.0.71) /klikit/database fetchTarget of 0x2c3/0x564bb4 parent IN=29622582 IN class=com.sleepycat.je.tree.DBIN lastFullVersion=0x2dd/0x38ae2f parent.getDirty()=true state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.
           at lu.kpmg.core.db.store.Store.execute(Store.java:230)
           at lu.kpmg.core.db.store.RecordManager.forEach(RecordManager.java:164)
           at lu.kpmg.core.db.store.RecordManager.queryRange(RecordManager.java:158)
           at lu.kpmg.core.db.store.RecordManager.queryAll(RecordManager.java:147)
           at lu.luggel.web.model.ContentManager.getSportResultsForAssociation(ContentManager.java:436)
           at sun.reflect.GeneratedMethodAccessor160.invoke(Unknown Source)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
           at java.lang.reflect.Method.invoke(Method.java:597)
           at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:389)
           at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:378)
           at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:270)
           ... 58 more
      Caused by: com.sleepycat.je.EnvironmentFailureException: (JE 4.0.71) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 4.0.71) /klikit/database fetchTarget of 0x2c3/0x564bb4 parent IN=29622582 IN class=com.sleepycat.je.tree.DBIN lastFullVersion=0x2dd/0x38ae2f parent.getDirty()=true state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.
           at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:197)
           at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1392)
           at com.sleepycat.je.dbi.CursorImpl.checkEnv(CursorImpl.java:2813)
           at com.sleepycat.je.Cursor.checkEnv(Cursor.java:2846)
           at com.sleepycat.je.Cursor.close(Cursor.java:439)
           at lu.kpmg.core.db.store.Store.execute(Store.java:226)
           ... 68 more
      Caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 4.0.71) /klikit/database fetchTarget of 0x2c3/0x564bb4 parent IN=29622582 IN class=com.sleepycat.je.tree.DBIN lastFullVersion=0x2dd/0x38ae2f parent.getDirty()=true state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.
           at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1241)
           at com.sleepycat.je.tree.BIN.fetchTarget(BIN.java:1300)
           at com.sleepycat.je.dbi.CursorImpl.fetchCurrent(CursorImpl.java:2362)
           at com.sleepycat.je.dbi.CursorImpl.fetchCurrent(CursorImpl.java:2389)
           at com.sleepycat.je.dbi.CursorImpl.getCurrentAlreadyLatched(CursorImpl.java:1410)
           at com.sleepycat.je.Cursor.searchInternal(Cursor.java:2174)
           at com.sleepycat.je.Cursor.searchAllowPhantoms(Cursor.java:2058)
           at com.sleepycat.je.Cursor.search(Cursor.java:1926)
           at com.sleepycat.je.SecondaryCursor.search(SecondaryCursor.java:1364)
           at com.sleepycat.je.SecondaryCursor.getSearchKeyRange(SecondaryCursor.java:1176)
           at lu.kpmg.core.db.store.Store.execute(Store.java:197)
           ... 68 more
      Caused by: java.io.FileNotFoundException: /klikit/database/000002c3.jdb (No such file or directory)
           at java.io.RandomAccessFile.open(Native Method)
           at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
           at java.io.RandomAccessFile.<init>(RandomAccessFile.java:98)
           at com.sleepycat.je.log.FileManager$1.<init>(FileManager.java:992)
           at com.sleepycat.je.log.FileManager.openFileHandle(FileManager.java:991)
           at com.sleepycat.je.log.FileManager.getFileHandle(FileManager.java:887)
           at com.sleepycat.je.log.LogManager.getLogSource(LogManager.java:1073)
           at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:779)
           at com.sleepycat.je.log.LogManager.getLogEntryAllowInvisibleAtRecovery(LogManager.java:743)
           at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1225)
           ... 78 more



      The exception occured several times on a day. The root cause was always related to the missing file "000002c3.jdb".

      When we run the DBVerify-Tool on that database, it fails with the following exception:

      Checking tree for key
      Error encountered (continuing):
      com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 4.1.7) d:\transfer\database_FileNotFound fetchTarget of 0x2c3/0x4d677e parent IN=29622782 IN class=com.sleepycat.je.tree.DBIN lastFullVersion=0x2de/0x8ccc00 parent.getDirty()=false state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.
      Error Key 592d83a084 Y-\83\a0\84
      Error Data 5a4e3398 ZN3\98
      Error encountered (continuing):
      com.sleepycat.je.EnvironmentFailureException: (JE 4.1.7) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 4.1.7) d:\transfer\database_FileNotFound fetchTarget of 0x2c3/0x4d677e parent IN=29622782 IN class=com.sleepycat.je.tree.DBIN lastFullVersion=0x2de/0x8ccc00 parent.getDirty()=false state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed. fetchTarget of 0x2de/0x78b409 parent IN=29622782 IN class=com.sleepycat.je.tree.DBIN lastFullVersion=0x2de/0x8ccc00 parent.getDirty()=false state=0
      Error Key 592d83a084 Y-\83\a0\84
      Error Data 5a4e341f ZN4\1f
      Error encountered (continuing):
      com.sleepycat.je.EnvironmentFailureException: (JE 4.1.7) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 4.1.7) d:\transfer\database_FileNotFound fetchTarget of 0x2c3/0x4d677e parent IN=29622782 IN class=com.sleepycat.je.tree.DBIN lastFullVersion=0x2de/0x8ccc00 parent.getDirty()=false state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed. fetchTarget of 0x2de/0x7a568e parent IN=29622781 IN class=com.sleepycat.je.tree.DIN lastFullVersion=0x2de/0x8cce1a parent.getDirty()=false state=0
      Error Key 592d87a06842fed4e17b2afa Y-\87\a0hB\fe\d4\e1{*\fa
      Error Data UNKNOWN
      Error encountered (continuing):
      com.sleepycat.je.EnvironmentFailureException: (JE 4.1.7) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 4.1.7) d:\transfer\database_FileNotFound fetchTarget of 0x2c3/0x4d677e parent IN=29622782 IN class=com.sleepycat.je.tree.DBIN lastFullVersion=0x2de/0x8ccc00 parent.getDirty()=false state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed. fetchTarget of 0x2de/0x79dae8 parent IN=16651289 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0x2de/0x91e179 parent.getDirty()=false state=0
      Error Key 592d88a06842fed362b24baa Y-\88\a0hB\fe\d3b\b2K\aa
      Error Data UNKNOWN
      Error encountered (continuing):
      ...
      ...
      com.sleepycat.je.EnvironmentFailureException: (JE 4.1.7) Environment must be clo
      sed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invali
      d because of previous exception: (JE 4.1.7) d:\transfer\database_FileNotFound fe
      tchTarget of 0x2c3/0x4d677e parent IN=29622782 IN class=com.sleepycat.je.tree.DB
      IN lastFullVersion=0x2de/0x8ccc00 parent.getDirty()=false state=0 LOG_FILE_NOT_F
      OUND: Log file missing, log is likely invalid. Environment is invalid and must b
      e closed.
      at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:196)
      at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1439)
      at com.sleepycat.je.Database.checkEnv(Database.java:1778)
      at com.sleepycat.je.Database.closeInternal(Database.java:377)
      at com.sleepycat.je.Database.close(Database.java:314)
      at com.sleepycat.je.util.DbVerify.verify(DbVerify.java:293)
      at com.sleepycat.je.util.DbVerify.main(DbVerify.java:98)
      Caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid bec
      ause of previous exception: (JE 4.1.7) d:\transfer\database_FileNotFound fetchTa
      rget of 0x2c3/0x4d677e parent IN=29622782 IN class=com.sleepycat.je.tree.DBIN la
      stFullVersion=0x2de/0x8ccc00 parent.getDirty()=false state=0 LOG_FILE_NOT_FOUND:
      Log file missing, log is likely invalid. Environment is invalid and must be clo
      sed.
      at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1337)
      at com.sleepycat.je.tree.BIN.fetchTarget(BIN.java:1367)
      at com.sleepycat.je.dbi.CursorImpl.fetchCurrent(CursorImpl.java:2499)
      at com.sleepycat.je.dbi.CursorImpl.fetchCurrent(CursorImpl.java:2526)
      at com.sleepycat.je.dbi.CursorImpl.getCurrentAlreadyLatched(CursorImpl.java:1545)
      at com.sleepycat.je.dbi.CursorImpl.getNextWithKeyChangeStatus(CursorImpl.java:1692)
      at com.sleepycat.je.dbi.CursorImpl.getNext(CursorImpl.java:1617)
      at com.sleepycat.je.dbi.DatabaseImpl.walkDatabaseTree(DatabaseImpl.java:1473)
      at com.sleepycat.je.dbi.DatabaseImpl.verify(DatabaseImpl.java:1420)
      at com.sleepycat.je.util.DbVerify.verifyOneDbImpl(DbVerify.java:366)
      at com.sleepycat.je.util.DbVerify.verify(DbVerify.java:285)
      ... 1 more
      Caused by: java.io.FileNotFoundException: d:\transfer\database_FileNotFound\000002c3.jdb (The system cannot find the file specified)
      at java.io.RandomAccessFile.open(Native Method)
      at java.io.RandomAccessFile.<init>(Unknown Source)
      at java.io.RandomAccessFile.<init>(Unknown Source)
      at com.sleepycat.je.log.FileManager$1.<init>(FileManager.java:995)
      at com.sleepycat.je.log.FileManager.openFileHandle(FileManager.java:994)
      at com.sleepycat.je.log.FileManager.getFileHandle(FileManager.java:890)
      at com.sleepycat.je.log.LogManager.getLogSource(LogManager.java:1074)
      at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:778)
      at com.sleepycat.je.log.LogManager.getLogEntryAllowInvisibleAtRecovery(LogManager.java:742)
      at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1320)
      ... 11 more
      Exit status = false



      It seems like the cleaner thread had removed the file "000002c3.jdb", while it was still needed.
      As the database was running with the "je.cleaner.expunge=false" setting, the file was still kept as "000002c3.del".

      So we renamed that file back to "000002c3.jdb".
      Running the dbVerify tool on that database succeeded (Exit status = true).
      The previously mentioned "FileNotFoundException" did also not occur any more.


      There is an additional piece of information that might be helpful.
      For the database with the missing file the dbverify tool had failed for a record with "Error Key 592d83a084".

      Before having the "FileNotFoundException" we found a record in the database that has a corrupted secondary index. We found out that
      the key 592d83a084 is pointing to this record.
      Deleting or updating this record fails with the following exception:

      Caused by: com.sleepycat.je.SecondaryIntegrityException: (JE 4.0.71) Secondary is corrupt: the primary record contains a key that is not present in the secondary
      at com.sleepycat.je.SecondaryDatabase.deleteKey(SecondaryDatabase.java:937)
      at com.sleepycat.je.SecondaryDatabase.updateSecondary(SecondaryDatabase.java:900)
      at com.sleepycat.je.SecondaryTrigger.databaseUpdated(SecondaryTrigger.java:42)
      at com.sleepycat.je.Database.notifyTriggers(Database.java:2004)
      at com.sleepycat.je.Cursor.putNotify(Cursor.java:1692)
      at com.sleepycat.je.Cursor.putInternal(Cursor.java:1616)
      at com.sleepycat.je.Database.putInternal(Database.java:1178)
      at com.sleepycat.je.Database.put(Database.java:1050)
      at lu.kpmg.core.db.store.Store.store(Store.java:325)
      ... 51 more

      I hope this helps to uncover the root cause of the problem and make JE even better.

      Holger
        • 1. Re: Cleaner thread prematurely removing log file
          Greybird-Oracle
          Hi Holger,

          There are several fixes for bugs that could cause this problem in the JE 4.0 product.

          http://download.oracle.com/otndocs/products/berkeleydb/html/je/je-40117-changelog.html

          Reference numbers for bugs that could cause this problem are: [#17252], [#19422], [#18227]. These fixes were all made after the JE release you're using, JE 4.0.71.

          Since I can tell from the stack trace you posted that you are using duplicates, at least one of these bugs may apply to your application.

          With this type of bug, it is usually not possible to continue to use the JE environment where the problem occurred. I suggest that you revert to a backup that was made before this problem occurred and then upgrade to the latest version of JE.

          You could upgrade to JE 4.0.117, the latest release in the JE 4.0 line, but it would be much better to upgrade to JE 4.1.7 to get the benefit of other performance improvements and bug fixes that have been made in JE 4.1. See: http://download.oracle.com/docs/cd/E17277_02/html/changelog.html

          --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
          • 2. Re: Cleaner thread prematurely removing log file
            837802
            Hi Mark,
            Thanks for your explanations. We will consider to upgrade to 4.1.7

            Holger
            • 3. Re: Cleaner thread prematurely removing log file
              924932
              I use je 4.1.10 yet I am seeing similar behavior. As this happens intermittently, I haven't figured out a proper steps to reproduce the issue in my application. This is the stack trace:
              Msg:(JE 4.1.10) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 4.1.10) nodePrimary(1):/www/a/logs/mcom/bdb/mcom-primary fetchTarget of 0xc2/0x787838 parent IN=4928 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0x18b/0x6b60d4 parent.getDirty()=false state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.
              com.sleepycat.je.EnvironmentFailureException: (JE 4.1.10) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 4.1.10) nodePrimary(1):/www/a/logs/mcom/bdb/mcom-primary fetchTarget of 0xc2/0x787838 parent IN=4928 IN class=com.sleepycat.je.tree.BIN lastFullVersion=0x18b/0x6b60d4 parent.getDirty()=false state=0 LOG_FILE_NOT_FOUND: Log file missing, log is likely invalid. Environment is invalid and must be closed.
              at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:196)
              at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1455)
              at com.sleepycat.je.Database.checkEnv(Database.java:1778)
              at com.sleepycat.je.Database.openCursor(Database.java:625)
              at com.sleepycat.je.SecondaryDatabase.openCursor(SecondaryDatabase.java:436)
              at com.sleepycat.je.SecondaryDatabase.openCursor(SecondaryDatabase.java:125)
              at com.sleepycat.persist.BasicIndex.cursor(BasicIndex.java:232)
              at com.sleepycat.persist.BasicIndex.cursor(BasicIndex.java:223)
              at com.sleepycat.persist.BasicIndex.entities(BasicIndex.java:186)
              at com.sleepycat.persist.SecondaryIndex.entities(SecondaryIndex.java:765)
              • 4. Re: Cleaner thread prematurely removing log file
                Greybird-Oracle
                When you say it happens intermittently, do you mean that it stops happening when you restart? Or is it a persistent problem and requires restore from backup to correct the problem?

                --mark                                                                                                                                                                                                                                                                                                                                                                                               
                • 5. Re: Cleaner thread prematurely removing log file
                  924932
                  The stack trace is part of a daily batch processing job, running concurrently. On most days the job runs correctly, only a couple of times have I seen this exception happen. But when i do see it, I need to restore from backup for the job to work again.
                  • 6. Re: Cleaner thread prematurely removing log file
                    Greybird-Oracle
                    The only bug we know of that could cause that problem is [#20732], fixed in JE 4.1.17.
                    http://download.oracle.com/otndocs/products/berkeleydb/html/je/je-4.1.17_changelog.html

                    I suggest the following course of action:
                    1) Upgrade to 4.1.17.
                    2) If the problem occurs again, save the problem environment home directory (the .jdb files) and post back here again. If you can send me .jdb files, I can work with you to try to diagnose the problem.

                    Thanks,
                    --mark