7 Replies Latest reply: Mar 19, 2007 8:54 AM by greybird RSS

    help! exception in Evictor.java, db becomes unsable after that

    538688
      Hi,

      I am using BDBJE inside a project. The application runs normally at the beginning (BDB being used at all time). It seems when the heap max is hit (1G in my case) and garbage collection starts to run more often, then the following db error seems to surface , after which any db operation gives the same error.

      I am running Sun jdk 1.5 and uses deferred write on the db. The application creates hundreds of dbs , then insert/delete entries from each db, and also delete a db entirely when that db becomes empty.

      I can re-produce this fairly easily using the application.

      Thanks for any help!

      java.io.IOException: (JE 3.0.12) Database blogads.com id=404 IN type=BIN/2 id=3335404 not expected on INList
           at com.sleepycat.je.evictor.Evictor.selectIN(Evictor.java:469)
           at com.sleepycat.je.evictor.Evictor.evictBatch(Evictor.java:330)
           at com.sleepycat.je.evictor.Evictor.doEvict(Evictor.java:242)
           at com.sleepycat.je.evictor.Evictor.doCriticalEviction(Evictor.java:266)
           at com.sleepycat.je.dbi.CursorImpl.close(CursorImpl.java:666)
           at com.sleepycat.je.Cursor.close(Cursor.java:243)
           at com.sleepycat.je.Database.putInternal(Database.java:571)
           at com.sleepycat.je.Database.putNoOverwrite(Database.java:530)
        • 1. Re: help! exception in Evictor.java, db becomes unsable after that
          Charles Lamb
          Hi user535685,

          Could you please let us know what version of JE you're using?

          Thanks.

          Charles Lamb
          • 2. Re: help! exception in Evictor.java, db becomes unsable after that
            Linda Lee
            Sorry, we see the JE version in the error message. It's JE 3.0.12.

            Is it possible for you to give us a test case?

            Thanks,

            Linda Lee
            • 3. Re: help! exception in Evictor.java, db becomes unsable after that
              greybird
              Hi,

              If you don't have a test case, but you can reproduce this problem, please email me and we can work together to troubleshoot the problem. I'll send you an instrumented jar file that will give us more information. Please email me: mark.hayes at the obvious dot com.

              Thanks for reporting this problem. This is the first time we've seen this.

              Mark
              • 4. Re: help! exception in Evictor.java, db becomes unsable after that
                greybird
                Question for you: Did you include the full stack trace, exactly as it appeared? The reason I ask is that the stack trace in your message is an IOException, but I believe it should be a DatabaseException. I'm wondering if an IOException also occurs, before or after the DatabaseException. If so, please send the complete stack trace.

                By code inspection I am not able to find a code path by which this problem could occur. I'll keep looking at it and get back to you if I find anything.

                Mark
                • 5. Re: help! exception in Evictor.java, db becomes unsable after that
                  538688
                  Hi, thanks for such prompt response.

                  I didn't included the full stack trace, I included the part that is bdb related, the other part of the stack trace are my application stuff that calls BDB and should not matter.

                  In my application, I actually convert DatabaseException from dbd into an IOException to simply exception handling, so even though the stack trace is showing IOException, it is really a DatabaseException. There is no prior IO exception in the application.

                  I will follow up with Mark Hayes and get the instrumented jar file as this is probably the preferred way from my side. Even though I can reproduce this error , it takes running the application for a long time and i am not sure I can release the application code as a test case.

                  Thanks again for all the help.
                  • 6. Re: help! exception in Evictor.java, db becomes unsable after that
                    greybird
                    Hello all,

                    I wanted to let everyone know about the resolution to this problem, in case you encounter it.

                    I'd like to thank the poster, user535685, for reporting this problem and working with us to test the fix.

                    The problem occurs under the following conditions:

                    + A DeferredWrite Database is used (DatabaseConfig.setDeferredWrite(true) is called).
                    + Some information is stored in the Database.
                    + The Database is closed without ever calling Database.sync method.

                    After closing the Database, the exception reported in this thread may occur while performing other activities, causing the Environment to become unusable.

                    If you encounter this problem, or if you use DeferredWrite databases and you would like to avoid this problem, please send me email and I'll send you a fix. My email is mark.hayes at the obvious dot com.

                    I've also included a diff below that can be used to patch the 3.2 source code to apply the fix.
                    Index: src/com/sleepycat/je/dbi/DatabaseImpl.java
                    ===================================================================
                    RCS file: /a/CVSROOT/je/src/com/sleepycat/je/dbi/DatabaseImpl.java,v
                    retrieving revision 1.157
                    diff -c -r1.157 DatabaseImpl.java
                    *** src/com/sleepycat/je/dbi/DatabaseImpl.java     13 Dec 2006 18:55:34 -0000     1.157
                    --- src/com/sleepycat/je/dbi/DatabaseImpl.java     12 Jan 2007 02:47:05 -0000
                    ***************
                    *** 543,604 ****
                                   * out the root.
                                   */
                                  long rootLsn = tree.getRootLsn();
                    -             if (rootLsn == DbLsn.NULL_LSN) {
                     
                    !                 /*
                    !                  * There's nothing in this database. (It might be the abort of
                    !                  * a truncation, where we are trying to clean up the new, blank
                    !                  * database. Do delete the MapLN.
                    !                  */
                    !                 envImpl.getDbMapTree().deleteMapLN(id);
                    !
                    !             } else {
                    !
                    !                 UtilizationTracker snapshot = new UtilizationTracker(envImpl);
                    !
                    !                 /*
                    !                  * Start by recording the lsn of the root IN as obsolete.  A
                    !                  * zero size is passed for the last parameter because it is too
                    !                  * expensive to fetch the node.
                    !                  */
                                      snapshot.countObsoleteNodeInexact
                                          (rootLsn, LogEntryType.LOG_IN, 0);
                    -
                    -                 /* Fetch LNs to count LN sizes only if so configured. */
                    -                 boolean fetchLNSize =
                    -                     envImpl.getCleaner().getFetchObsoleteSize();
                    -
                    -                 /* Use the tree walker to visit every child lsn in the tree. */
                    -                 ObsoleteProcessor obsoleteProcessor =
                    -                     new ObsoleteProcessor(snapshot);
                    -                 SortedLSNTreeWalker walker = new ObsoleteTreeWalker
                    -                     (this, rootLsn, fetchLNSize, obsoleteProcessor);
                    -
                    -                 /*
                    -                  * Delete MapLN before the walk. Note that the processing of
                    -                  * the naming tree means this MapLN is never actually
                    -                  * accessible from the current tree, but deleting the MapLN
                    -                  * will do two things:
                    -                  * (a) mark it properly obsolete
                    -                  * (b) null out the database tree, leaving the INList the only
                    -                  * reference to the INs.
                    -                  */
                    -                 envImpl.getDbMapTree().deleteMapLN(id);
                    -
                    -                 /*
                    -                  * At this point, it's possible for the evictor to find an IN
                    -                  * for this database on the INList. It should be ignored.
                    -                  */
                    -                 walker.walk();
                    -
                    -                 /*
                    -                  * Count obsolete nodes for a deleted database at transaction
                    -                  * end time.  Write out the modified file summaries for
                    -                  * recovery.
                    -                  */
                    -                 envImpl.getUtilizationProfile().countAndLogSummaries
                    -                     (snapshot.getTrackedFiles());
                                  }
                              } finally {
                                  deleteState = DELETED;
                              }
                    --- 543,596 ----
                                   * out the root.
                                   */
                                  long rootLsn = tree.getRootLsn();
                     
                    !
                    !             /*
                    !              * Use a snapshot tracker that is accumulated under the log write
                    !              * latch when we're doing counting.  Start by recording the LSN of
                    !              * the root IN as obsolete.  A zero size is passed for the last
                    !              * parameter because it is too expensive to fetch the node.
                    !              */
                    !             UtilizationTracker snapshot = new UtilizationTracker(envImpl);
                    !             if (rootLsn != DbLsn.NULL_LSN) {
                                      snapshot.countObsoleteNodeInexact
                                          (rootLsn, LogEntryType.LOG_IN, 0);
                                  }
                    +
                    +             /* Fetch LNs to count LN sizes only if so configured. */
                    +             boolean fetchLNSize =
                    +                 envImpl.getCleaner().getFetchObsoleteSize();
                    +
                    +             /* Use the tree walker to visit every child lsn in the tree. */
                    +             ObsoleteProcessor obsoleteProcessor =
                    +                 new ObsoleteProcessor(snapshot);
                    +             SortedLSNTreeWalker walker = new ObsoleteTreeWalker
                    +                 (this, rootLsn, fetchLNSize, obsoleteProcessor);
                    +
                    +             /*
                    +              * Delete MapLN before the walk. Note that the processing of
                    +              * the naming tree means this MapLN is never actually
                    +              * accessible from the current tree, but deleting the MapLN
                    +              * will do two things:
                    +              * (a) mark it properly obsolete
                    +              * (b) null out the database tree, leaving the INList the only
                    +              * reference to the INs.
                    +              */
                    +             envImpl.getDbMapTree().deleteMapLN(id);
                    +
                    +             /*
                    +              * At this point, it's possible for the evictor to find an IN
                    +              * for this database on the INList. It should be ignored.
                    +              */
                    +             walker.walk();
                    +
                    +             /*
                    +              * Count obsolete nodes for a deleted database at transaction
                    +              * end time.  Write out the modified file summaries for
                    +              * recovery.
                    +              */
                    +             envImpl.getUtilizationProfile().countAndLogSummaries
                    +                 (snapshot.getTrackedFiles());
                              } finally {
                                  deleteState = DELETED;
                              }
                    Mark
                    • 7. Re: help! exception in Evictor.java, db becomes unsable after that
                      greybird
                      Hello all,

                      For the record, the fix for this problem is included in JE 3.2.21:

                      Berkeley DB Java Edition 3.2.21 is now available

                      Mark