This discussion is archived
3 Replies Latest reply: Nov 19, 2007 5:44 PM by greybird RSS

DatabaseException

605179 Newbie
Currently Being Moderated
Hi,
my application creates a large amount of data stored on .jdb files (for example 6 Giga) on a long running time (for example 12 hours).
While my application is running the following DataBaseException is thrown:

<Checkpointer name="Checkpointer"/> caught exception: com.sleepycat.je.DatabaseException: (JE 3.2.21) can't find database 5343358
com.sleepycat.je.DatabaseException: (JE 3.2.21) can't find database 5343358
at com.sleepycat.je.dbi.DbTree.modifyDbRoot(DbTree.java:270)
at com.sleepycat.je.recovery.Checkpointer.flushIN(Checkpointer.java:746)
at com.sleepycat.je.recovery.Checkpointer.flushDirtyNodes(Checkpointer.java:669)
at com.sleepycat.je.recovery.Checkpointer.doCheckpoint(Checkpointer.java:447)
at com.sleepycat.je.recovery.Checkpointer.onWakeup(Checkpointer.java:216)
at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:191)
at java.lang.Thread.run(Thread.java:619)
Continuing


This Exception was thrown several times and it can't be caught by my application so i can't understand the reason why some databases were deleted and which of my application calls methods caused the Exception.
I thought that the cleaner thread could delete the .jdb files that were inactive for a long time so i tried to set the je.cleaner.minUtilization property to 0 value but the problem persists.

Can someone help me?

Thank you very much,
elena.
  • 1. Re: DatabaseException
    greybird Expert
    Currently Being Moderated
    Hi Elena,

    I don't think this is a bug in your application, it is probably a bug in JE. I need to ask you some questions to help us to diagnose the problem so that we can give you a workaround or a bug fix.

    1) Are you calling Environment.removeDatabase or truncateDatabase? If so, please describe how you are using these methods -- when they are called, and how often.

    2) Are you using deferred write -- DatabaseConfig.setDeferredWrite(true)?

    3) Please send the configuration for the environment you are using -- calls to EnvironmentConfig and any settings in je.properties.

    4) Please describe a little about how your application uses JE. For example, what is the mix of inserts, updates and deletes? How many threads are accessing JE? Etc.

    Thanks,
    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
  • 2. Re: DatabaseException
    605179 Newbie
    Currently Being Moderated
    Hi Mark!
    In the following method "createEnv" i create my enviroment, in red the code about enviroment config:

    public static void createEnv(int chacePercent, Object dataType)throws DataBaseException, ClassNotFoundException{
              
              try
              {

                   //create the directory that will contain the BerkeleyDB logs
                   envDir = new File(LOGSDIR);
                   envDir.mkdir();
    envDir.deleteOnExit();
         
                   //Create the BerkeleyDB Environment Configuration:
                   EnvironmentConfig envConfig = new EnvironmentConfig();
                   envConfig.setTransactional(false);
                   envConfig.setAllowCreate(true);

                   /* set the Cache percent (of the VM) used by BerkelyDB (Default value = 60).
                   * Cache percent must be >= 0 and <= 90 */
                   if (chacePercent >= 0 && chacePercent <= 90){
                        envConfig.setCachePercent(chacePercent);
                   }     
                   
         //Create the BerkeleyDB Environment
                   env = new Environment(envDir, envConfig);

                   //Create a configuration for databases:
                   dbConfig = new DatabaseConfig();
                   dbConfig.setTransactional(false);
                   dbConfig.setAllowCreate(true);
                   dbConfig.setSortedDuplicates(false);
         
                   dbConfigDup = new DatabaseConfig();
                   dbConfigDup.setTransactional(false);
                   dbConfigDup.setAllowCreate(true);
                   dbConfigDup.setSortedDuplicates(true);
         
                   /*
                   * the database record is composed of an String
                   * key and and instance of the Sequence class as data.
                   *
                   * A class catalog database is needed for storing class descriptions
                   * for the serial binding used below. This avoids storing class
                   * descriptions redundantly in each record.
                   */
                   catalogConfig = new DatabaseConfig();
                   catalogConfig.setTransactional(false);
                   catalogConfig.setAllowCreate(true);
                   catalogDb = env.openDatabase(null,"catalogDb",catalogConfig);
                   catalog = new StoredClassCatalog(catalogDb);
         
                   /*
                   * Create a serial binding for dataType data objects. Serial bindings
                   * can be used to store any Serializable object.
                   */
                   dataBinding = new SerialBinding(catalog, dataType.getClass());
                        }
              catch (Exception e)
              {
                             throw new DataBaseException("Exception in BerkelyEnv.createEnv() -"+e.getMessage(), e.getStackTrace());
              }

         }

    Yes i call the method Enviroment.removeDatabase in the following ricorsive procedure schema:
    the recorsive procedure has a parameter that represent a database. For semplicity call this database db1.
    Over db1 the frequent items are calculated (items that occur in the database a number of times over a given threshold)
    If the number of frequent items is zero then db1 is deleted else for each frequent item: a new database is built and the procedure is called ricorsively over db1.
    When the recorsive procedure is terminated db1 is deleted.
    This schema is repeated for two times.

    The recursive procedure name is "projectDBminsup" and follows:

    public void projectDBminsup(DataBase bdb, String dbName) throws SpmtccException{
         
         Sequence newPrefix;
         try
         {

         bdb.open(dbName);

         //get frequent items of the current database of sequences
         HashMap itemsfreq = getFrequentItems(bdb);
    bdb.close();
    //if the number of items is not null proceed
         if (!itemsfreq.isEmpty()) //frequent items exist
         {
         //obtain frequent items and their support
         Object[] items = itemsfreq.keySet().toArray();
         Object[] supports = itemsfreq.values().toArray();
         
         //divide and conquer phase: project every frequent item and obtain
         //smallers database on which the algorithm recurs
         DataBase projDB; //DataBase has a field "db" of type com.sleepycat.je.Database and methods which works over db using BerkelyDb methods.
         int itemValue;
         int itemSupp;
         int key;
         String exProjDB_prefix;
         String inProjDB_prefix;
         
         
         for (int i = 0; i < items.length; i++)
         {
    itemValue = ((Integer) items).intValue();
    itemSupp = ((Integer) supports[i]).intValue();
         exProjDB_prefix = dbName + "(" + itemValue + ")";
         projDB = new DataBase();
         projDB.open(exProjDB_prefix); //call projDB.db = Enviroment.openDatabase(exProjDB_prefix);
         //database projDB will be modified by extrasetGrowth method:
         newPrefix = this.minExtrasetGrowth(bdb,dbName,projDB,exProjDB_prefix,itemValue,itemSupp);
         if (newPrefix!=null) //database projDB is not empty
         {
              newPrefix.setSupport(projDB.getSize());
              //use the sequence size as key
                   key = newPrefix.getNumItems();
                   //the elements in res are ordered by key
              res.add(key, newPrefix); // res is a global variable
              numSeq++;
              projDB.close();
              //recursive call
         projectDBminsup(projDB,exProjDB_prefix);
              projDB.delete(exProjDB_prefix);// call Enviroment.removeDatabase(null, exProjDB_prefix);
                   
         }
         else
         {
              projDB.close();
              projDB.delete(exProjDB_prefix);// call Enviroment.removeDatabase(null, exProjDB_prefix);
         }
         
         if (dbName.endsWith(")"))
         {
              inProjDB_prefix = dbName.substring(0, dbName.length()-1)+ "," + itemValue + ")";
         }
         else
         {
              inProjDB_prefix = dbName+ "("+ itemValue + ")";
         }
         projDB = new DataBase();
         projDB.open(inProjDB_prefix);
         //database projDB will be modified by extrasetGrowth method
         newPrefix = this.minIntrasetGrowth(bdb,dbName,projDB,inProjDB_prefix,itemValue,itemSupp);
         if (newPrefix!=null) //database projDB is not empty
         {
              newPrefix.setSupport(projDB.getSize());
              //use the sequence size as key
                   key = newPrefix.getNumItems();
                   //the elements in res are ordered by key
         res.add(key, newPrefix); // res is a global variable
              numSeq++;
         projDB.close();
              //recursive call
              projectDBminsup(projDB,inProjDB_prefix);
              projDB.delete(inProjDB_prefix);
         }
         else
    {
    projDB.close();
         projDB.delete(inProjDB_prefix);
    }
         }//end loop on frequent items
    }//end else (items freq exist)
         }
         catch (DataBaseException e)
         {
              throw new SpmtccException(e.getMessage(), e.getStackTrace());
         }
    catch (Exception e)
         {
              throw new SpmtccException(e.getMessage(), e.getStackTrace());
         }
    }
  • 3. Re: DatabaseException
    greybird Expert
    Currently Being Moderated
    Hi Elena,

    Thanks for sending this additional information.

    I suspect this is a bug in JE that has to do with an interaction between JE built-in background threads -- the checkpointer and the cleaner -- and the removeDatabase method. It is going to take us some time to resolve this. We have not reproduced it yet. I suspect it will take us several weeks to resovle, but please understand that is an estimate.

    When the JE background thread -- checkpointer or cleaner -- gets an exception, it will continue later. So checkpoints and cleaning will not be prevented unless the exception occurs repeatedly.

    Could you please send me an email at mark.hayes at oracle dot com. When I have more information on this problem, I'll email you directly.

    Thanks for reporting this and I apologize for the inconvenience.

    --mark