This discussion is archived
7 Replies Latest reply: Mar 19, 2007 6:54 AM by greybird RSS

help! exception in Evictor.java, db becomes unsable after that

538688 Newbie
Currently Being Moderated
Hi,

I am using BDBJE inside a project. The application runs normally at the beginning (BDB being used at all time). It seems when the heap max is hit (1G in my case) and garbage collection starts to run more often, then the following db error seems to surface , after which any db operation gives the same error.

I am running Sun jdk 1.5 and uses deferred write on the db. The application creates hundreds of dbs , then insert/delete entries from each db, and also delete a db entirely when that db becomes empty.

I can re-produce this fairly easily using the application.

Thanks for any help!

java.io.IOException: (JE 3.0.12) Database blogads.com id=404 IN type=BIN/2 id=3335404 not expected on INList
     at com.sleepycat.je.evictor.Evictor.selectIN(Evictor.java:469)
     at com.sleepycat.je.evictor.Evictor.evictBatch(Evictor.java:330)
     at com.sleepycat.je.evictor.Evictor.doEvict(Evictor.java:242)
     at com.sleepycat.je.evictor.Evictor.doCriticalEviction(Evictor.java:266)
     at com.sleepycat.je.dbi.CursorImpl.close(CursorImpl.java:666)
     at com.sleepycat.je.Cursor.close(Cursor.java:243)
     at com.sleepycat.je.Database.putInternal(Database.java:571)
     at com.sleepycat.je.Database.putNoOverwrite(Database.java:530)
  • 1. Re: help! exception in Evictor.java, db becomes unsable after that
    Charles Lamb Pro
    Currently Being Moderated
    Hi user535685,

    Could you please let us know what version of JE you're using?

    Thanks.

    Charles Lamb
  • 2. Re: help! exception in Evictor.java, db becomes unsable after that
    Linda Lee Journeyer
    Currently Being Moderated
    Sorry, we see the JE version in the error message. It's JE 3.0.12.

    Is it possible for you to give us a test case?

    Thanks,

    Linda Lee
  • 3. Re: help! exception in Evictor.java, db becomes unsable after that
    greybird Expert
    Currently Being Moderated
    Hi,

    If you don't have a test case, but you can reproduce this problem, please email me and we can work together to troubleshoot the problem. I'll send you an instrumented jar file that will give us more information. Please email me: mark.hayes at the obvious dot com.

    Thanks for reporting this problem. This is the first time we've seen this.

    Mark
  • 4. Re: help! exception in Evictor.java, db becomes unsable after that
    greybird Expert
    Currently Being Moderated
    Question for you: Did you include the full stack trace, exactly as it appeared? The reason I ask is that the stack trace in your message is an IOException, but I believe it should be a DatabaseException. I'm wondering if an IOException also occurs, before or after the DatabaseException. If so, please send the complete stack trace.

    By code inspection I am not able to find a code path by which this problem could occur. I'll keep looking at it and get back to you if I find anything.

    Mark
  • 5. Re: help! exception in Evictor.java, db becomes unsable after that
    538688 Newbie
    Currently Being Moderated
    Hi, thanks for such prompt response.

    I didn't included the full stack trace, I included the part that is bdb related, the other part of the stack trace are my application stuff that calls BDB and should not matter.

    In my application, I actually convert DatabaseException from dbd into an IOException to simply exception handling, so even though the stack trace is showing IOException, it is really a DatabaseException. There is no prior IO exception in the application.

    I will follow up with Mark Hayes and get the instrumented jar file as this is probably the preferred way from my side. Even though I can reproduce this error , it takes running the application for a long time and i am not sure I can release the application code as a test case.

    Thanks again for all the help.
  • 6. Re: help! exception in Evictor.java, db becomes unsable after that
    greybird Expert
    Currently Being Moderated
    Hello all,

    I wanted to let everyone know about the resolution to this problem, in case you encounter it.

    I'd like to thank the poster, user535685, for reporting this problem and working with us to test the fix.

    The problem occurs under the following conditions:

    + A DeferredWrite Database is used (DatabaseConfig.setDeferredWrite(true) is called).
    + Some information is stored in the Database.
    + The Database is closed without ever calling Database.sync method.

    After closing the Database, the exception reported in this thread may occur while performing other activities, causing the Environment to become unusable.

    If you encounter this problem, or if you use DeferredWrite databases and you would like to avoid this problem, please send me email and I'll send you a fix. My email is mark.hayes at the obvious dot com.

    I've also included a diff below that can be used to patch the 3.2 source code to apply the fix.
    Index: src/com/sleepycat/je/dbi/DatabaseImpl.java
    ===================================================================
    RCS file: /a/CVSROOT/je/src/com/sleepycat/je/dbi/DatabaseImpl.java,v
    retrieving revision 1.157
    diff -c -r1.157 DatabaseImpl.java
    *** src/com/sleepycat/je/dbi/DatabaseImpl.java     13 Dec 2006 18:55:34 -0000     1.157
    --- src/com/sleepycat/je/dbi/DatabaseImpl.java     12 Jan 2007 02:47:05 -0000
    ***************
    *** 543,604 ****
                   * out the root.
                   */
                  long rootLsn = tree.getRootLsn();
    -             if (rootLsn == DbLsn.NULL_LSN) {
     
    !                 /*
    !                  * There's nothing in this database. (It might be the abort of
    !                  * a truncation, where we are trying to clean up the new, blank
    !                  * database. Do delete the MapLN.
    !                  */
    !                 envImpl.getDbMapTree().deleteMapLN(id);
    !
    !             } else {
    !
    !                 UtilizationTracker snapshot = new UtilizationTracker(envImpl);
    !
    !                 /*
    !                  * Start by recording the lsn of the root IN as obsolete.  A
    !                  * zero size is passed for the last parameter because it is too
    !                  * expensive to fetch the node.
    !                  */
                      snapshot.countObsoleteNodeInexact
                          (rootLsn, LogEntryType.LOG_IN, 0);
    -
    -                 /* Fetch LNs to count LN sizes only if so configured. */
    -                 boolean fetchLNSize =
    -                     envImpl.getCleaner().getFetchObsoleteSize();
    -
    -                 /* Use the tree walker to visit every child lsn in the tree. */
    -                 ObsoleteProcessor obsoleteProcessor =
    -                     new ObsoleteProcessor(snapshot);
    -                 SortedLSNTreeWalker walker = new ObsoleteTreeWalker
    -                     (this, rootLsn, fetchLNSize, obsoleteProcessor);
    -
    -                 /*
    -                  * Delete MapLN before the walk. Note that the processing of
    -                  * the naming tree means this MapLN is never actually
    -                  * accessible from the current tree, but deleting the MapLN
    -                  * will do two things:
    -                  * (a) mark it properly obsolete
    -                  * (b) null out the database tree, leaving the INList the only
    -                  * reference to the INs.
    -                  */
    -                 envImpl.getDbMapTree().deleteMapLN(id);
    -
    -                 /*
    -                  * At this point, it's possible for the evictor to find an IN
    -                  * for this database on the INList. It should be ignored.
    -                  */
    -                 walker.walk();
    -
    -                 /*
    -                  * Count obsolete nodes for a deleted database at transaction
    -                  * end time.  Write out the modified file summaries for
    -                  * recovery.
    -                  */
    -                 envImpl.getUtilizationProfile().countAndLogSummaries
    -                     (snapshot.getTrackedFiles());
                  }
              } finally {
                  deleteState = DELETED;
              }
    --- 543,596 ----
                   * out the root.
                   */
                  long rootLsn = tree.getRootLsn();
     
    !
    !             /*
    !              * Use a snapshot tracker that is accumulated under the log write
    !              * latch when we're doing counting.  Start by recording the LSN of
    !              * the root IN as obsolete.  A zero size is passed for the last
    !              * parameter because it is too expensive to fetch the node.
    !              */
    !             UtilizationTracker snapshot = new UtilizationTracker(envImpl);
    !             if (rootLsn != DbLsn.NULL_LSN) {
                      snapshot.countObsoleteNodeInexact
                          (rootLsn, LogEntryType.LOG_IN, 0);
                  }
    +
    +             /* Fetch LNs to count LN sizes only if so configured. */
    +             boolean fetchLNSize =
    +                 envImpl.getCleaner().getFetchObsoleteSize();
    +
    +             /* Use the tree walker to visit every child lsn in the tree. */
    +             ObsoleteProcessor obsoleteProcessor =
    +                 new ObsoleteProcessor(snapshot);
    +             SortedLSNTreeWalker walker = new ObsoleteTreeWalker
    +                 (this, rootLsn, fetchLNSize, obsoleteProcessor);
    +
    +             /*
    +              * Delete MapLN before the walk. Note that the processing of
    +              * the naming tree means this MapLN is never actually
    +              * accessible from the current tree, but deleting the MapLN
    +              * will do two things:
    +              * (a) mark it properly obsolete
    +              * (b) null out the database tree, leaving the INList the only
    +              * reference to the INs.
    +              */
    +             envImpl.getDbMapTree().deleteMapLN(id);
    +
    +             /*
    +              * At this point, it's possible for the evictor to find an IN
    +              * for this database on the INList. It should be ignored.
    +              */
    +             walker.walk();
    +
    +             /*
    +              * Count obsolete nodes for a deleted database at transaction
    +              * end time.  Write out the modified file summaries for
    +              * recovery.
    +              */
    +             envImpl.getUtilizationProfile().countAndLogSummaries
    +                 (snapshot.getTrackedFiles());
              } finally {
                  deleteState = DELETED;
              }
    Mark
  • 7. Re: help! exception in Evictor.java, db becomes unsable after that
    greybird Expert
    Currently Being Moderated
    Hello all,

    For the record, the fix for this problem is included in JE 3.2.21:

    Berkeley DB Java Edition 3.2.21 is now available

    Mark