Forum Stats

  • 3,769,977 Users
  • 2,253,041 Discussions
  • 7,875,257 Comments

Discussions

Problem with Lucene Incremental indexing on Berkeley DB JE

527078
527078 Member Posts: 33
edited Aug 16, 2006 9:10AM in Berkeley DB Java Edition
Hello, dear developers of JE

I have implemented Lucene Incremental indexing on Berkeley DB JE on live index. The problem is updated values not get affected in search results without restarting the JBoss server.

I need additions and deletions of documents to the index, and that should be immidiately get affect in search results without restart the JBoss server.

One more major problem is while i'm deleting the documents from index, the database size is growing.

Please suggest me the better solution to the above problems.

Thanks,

Katta

Comments

  • Charles Lamb
    Charles Lamb Member Posts: 836
    Hello Katta,

    Are you using transactions, and if so, are you committing the transactions?

    Regards,

    Charles Lamb
  • 527078
    527078 Member Posts: 33
    Hi Charles,

    I'm using transactions & committing the transactions. Please go through the following code.

    EnvironmentConfig envConfig = new EnvironmentConfig();
    envConfig.setTransactional(true);
    envConfig.setAllowCreate(true);

    DatabaseConfig dbConfig = new DatabaseConfig();
    dbConfig.setAllowCreate(true);
    dbConfig.setTransactional(true);

    Environment env = new Environment(dbHome, envConfig);
    Transaction txn = null;
    Database index, blocks;

    try {
    txn = env.beginTransaction(null, null);
    index = env.openDatabase(txn, "__index__", dbConfig);
    blocks = env.openDatabase(txn, "__blocks__", dbConfig);
    } catch (DatabaseException e) {
    if (txn != null)
    {
    txn.abort();
    txn = null;
    }
    throw e;
    } finally {
    if (txn != null)
    txn.commit();
    txn = null;
    }
    try{
    txn = env.beginTransaction(null, null);
    indexDir = new JEDirectory(txn, index, blocks);
    System.out.println("INDEXING INFO: Start Indexing updated content.");

    IndexWriter indexWriter = new IndexWriter(indexDir,new StandardAnalyzer(), false);
    indexWriter.setUseCompoundFile(false);
    // configure the writer
    indexWriter.setMergeFactor(1000);
    indexWriter.setMaxMergeDocs(9999999);
    indexWriter.setMaxBufferedDocs(1000);

    // call the updateLuceneIndex method
    updateLuceneIndex(indexWriter );

    indexWriter.optimize();
    indexWriter.close();

    System.out.println("INDEXING INFO: Optimizing Index finished......");
    } catch (IOException e)
    {
    txn.abort();
    txn = null;
    e.printStackTrace();
    throw e;
    } finally
    {
    if (txn != null)
    txn.commit();

    index.close();
    blocks.close();
    env.close();
    }


    Thanks,

    Katta.
  • Charles Lamb
    Charles Lamb Member Posts: 836
    Hi Katta,

    Can you please do a DbDump of the database before and after to see if anything has been added to it when you run this? You can do this with

    java com.sleepycat.je.util.DbDump -h <envdir> -s <dbname>

    Thanks.

    Charles Lamb
  • 527078
    527078 Member Posts: 33
    Hi Charles ,

    Even if i run DbDump before and after lucene incremental update there is no change in the db size.I did the following way.

    Step:

    1. java com.sleepycat.je.util.DbDump -h D:\berkely\kindex -s __blocks__

    2. java com.sleepycat.je.util.DbDump -h D:\berkely\kindex -s __index__

    3. java com.tg.lucene.UpdateLuceneIndex D:\berkely\dbEnvHome update

    4. java com.sleepycat.je.util.DbDump -h D:\berkely\kindex -s __blocks__

    5.java com.sleepycat.je.util.DbDump -h D:\berkely\kindex -s __index__

    Please can you explain what DbDump does. How it reduce the size. I indexed Country data(list of countries in my table), it took 360KB. After that in update, i deleted 2 countries data from that db, and optimized it. But the size grown to 770KB. While deleting data why it's size is growing instead of shrinking.


    Thanks,

    Katta.
  • Hi Katta, JE has a log based storage system, so deletes will write to the log. You may want to read the following FAQ's for more information on why you're seeing the log grow during deletes: http://forums.oracle.com/forums/thread.jspa?messageID=1411144&tstart=0#1411144.
  • 527078
    527078 Member Posts: 33
    Hi,

    The suggested link on JE log & FAQ taking me to the same page.Still i'm unable to get the solution for JE log grow during deletes.

    Thanx,

    Katta.
  • 512799
    512799 Member Posts: 494
    Try this link:

    http://www.oracle.com/technology/products/berkeley-db/faq/je_faq.html#14

    Ron
This discussion has been closed.