3 Replies Latest reply: Nov 19, 2007 7:44 PM by Greybird-Oracle RSS

    DatabaseException

    605179
      Hi,
      my application creates a large amount of data stored on .jdb files (for example 6 Giga) on a long running time (for example 12 hours).
      While my application is running the following DataBaseException is thrown:

      <Checkpointer name="Checkpointer"/> caught exception: com.sleepycat.je.DatabaseException: (JE 3.2.21) can't find database 5343358
      com.sleepycat.je.DatabaseException: (JE 3.2.21) can't find database 5343358
      at com.sleepycat.je.dbi.DbTree.modifyDbRoot(DbTree.java:270)
      at com.sleepycat.je.recovery.Checkpointer.flushIN(Checkpointer.java:746)
      at com.sleepycat.je.recovery.Checkpointer.flushDirtyNodes(Checkpointer.java:669)
      at com.sleepycat.je.recovery.Checkpointer.doCheckpoint(Checkpointer.java:447)
      at com.sleepycat.je.recovery.Checkpointer.onWakeup(Checkpointer.java:216)
      at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:191)
      at java.lang.Thread.run(Thread.java:619)
      Continuing


      This Exception was thrown several times and it can't be caught by my application so i can't understand the reason why some databases were deleted and which of my application calls methods caused the Exception.
      I thought that the cleaner thread could delete the .jdb files that were inactive for a long time so i tried to set the je.cleaner.minUtilization property to 0 value but the problem persists.

      Can someone help me?

      Thank you very much,
      elena.
        • 1. Re: DatabaseException
          Greybird-Oracle
          Hi Elena,

          I don't think this is a bug in your application, it is probably a bug in JE. I need to ask you some questions to help us to diagnose the problem so that we can give you a workaround or a bug fix.

          1) Are you calling Environment.removeDatabase or truncateDatabase? If so, please describe how you are using these methods -- when they are called, and how often.

          2) Are you using deferred write -- DatabaseConfig.setDeferredWrite(true)?

          3) Please send the configuration for the environment you are using -- calls to EnvironmentConfig and any settings in je.properties.

          4) Please describe a little about how your application uses JE. For example, what is the mix of inserts, updates and deletes? How many threads are accessing JE? Etc.

          Thanks,
          --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
          • 2. Re: DatabaseException
            605179
            Hi Mark!
            In the following method "createEnv" i create my enviroment, in red the code about enviroment config:

            public static void createEnv(int chacePercent, Object dataType)throws DataBaseException, ClassNotFoundException{
                      
                      try
                      {

                           //create the directory that will contain the BerkeleyDB logs
                           envDir = new File(LOGSDIR);
                           envDir.mkdir();
            envDir.deleteOnExit();
                 
                           //Create the BerkeleyDB Environment Configuration:
                           EnvironmentConfig envConfig = new EnvironmentConfig();
                           envConfig.setTransactional(false);
                           envConfig.setAllowCreate(true);

                           /* set the Cache percent (of the VM) used by BerkelyDB (Default value = 60).
                           * Cache percent must be >= 0 and <= 90 */
                           if (chacePercent >= 0 && chacePercent <= 90){
                                envConfig.setCachePercent(chacePercent);
                           }     
                           
                 //Create the BerkeleyDB Environment
                           env = new Environment(envDir, envConfig);

                           //Create a configuration for databases:
                           dbConfig = new DatabaseConfig();
                           dbConfig.setTransactional(false);
                           dbConfig.setAllowCreate(true);
                           dbConfig.setSortedDuplicates(false);
                 
                           dbConfigDup = new DatabaseConfig();
                           dbConfigDup.setTransactional(false);
                           dbConfigDup.setAllowCreate(true);
                           dbConfigDup.setSortedDuplicates(true);
                 
                           /*
                           * the database record is composed of an String
                           * key and and instance of the Sequence class as data.
                           *
                           * A class catalog database is needed for storing class descriptions
                           * for the serial binding used below. This avoids storing class
                           * descriptions redundantly in each record.
                           */
                           catalogConfig = new DatabaseConfig();
                           catalogConfig.setTransactional(false);
                           catalogConfig.setAllowCreate(true);
                           catalogDb = env.openDatabase(null,"catalogDb",catalogConfig);
                           catalog = new StoredClassCatalog(catalogDb);
                 
                           /*
                           * Create a serial binding for dataType data objects. Serial bindings
                           * can be used to store any Serializable object.
                           */
                           dataBinding = new SerialBinding(catalog, dataType.getClass());
                                }
                      catch (Exception e)
                      {
                                     throw new DataBaseException("Exception in BerkelyEnv.createEnv() -"+e.getMessage(), e.getStackTrace());
                      }

                 }

            Yes i call the method Enviroment.removeDatabase in the following ricorsive procedure schema:
            the recorsive procedure has a parameter that represent a database. For semplicity call this database db1.
            Over db1 the frequent items are calculated (items that occur in the database a number of times over a given threshold)
            If the number of frequent items is zero then db1 is deleted else for each frequent item: a new database is built and the procedure is called ricorsively over db1.
            When the recorsive procedure is terminated db1 is deleted.
            This schema is repeated for two times.

            The recursive procedure name is "projectDBminsup" and follows:

            public void projectDBminsup(DataBase bdb, String dbName) throws SpmtccException{
                 
                 Sequence newPrefix;
                 try
                 {

                 bdb.open(dbName);

                 //get frequent items of the current database of sequences
                 HashMap itemsfreq = getFrequentItems(bdb);
            bdb.close();
            //if the number of items is not null proceed
                 if (!itemsfreq.isEmpty()) //frequent items exist
                 {
                 //obtain frequent items and their support
                 Object[] items = itemsfreq.keySet().toArray();
                 Object[] supports = itemsfreq.values().toArray();
                 
                 //divide and conquer phase: project every frequent item and obtain
                 //smallers database on which the algorithm recurs
                 DataBase projDB; //DataBase has a field "db" of type com.sleepycat.je.Database and methods which works over db using BerkelyDb methods.
                 int itemValue;
                 int itemSupp;
                 int key;
                 String exProjDB_prefix;
                 String inProjDB_prefix;
                 
                 
                 for (int i = 0; i < items.length; i++)
                 {
            itemValue = ((Integer) items).intValue();
            itemSupp = ((Integer) supports[i]).intValue();
                 exProjDB_prefix = dbName + "(" + itemValue + ")";
                 projDB = new DataBase();
                 projDB.open(exProjDB_prefix); //call projDB.db = Enviroment.openDatabase(exProjDB_prefix);
                 //database projDB will be modified by extrasetGrowth method:
                 newPrefix = this.minExtrasetGrowth(bdb,dbName,projDB,exProjDB_prefix,itemValue,itemSupp);
                 if (newPrefix!=null) //database projDB is not empty
                 {
                      newPrefix.setSupport(projDB.getSize());
                      //use the sequence size as key
                           key = newPrefix.getNumItems();
                           //the elements in res are ordered by key
                      res.add(key, newPrefix); // res is a global variable
                      numSeq++;
                      projDB.close();
                      //recursive call
                 projectDBminsup(projDB,exProjDB_prefix);
                      projDB.delete(exProjDB_prefix);// call Enviroment.removeDatabase(null, exProjDB_prefix);
                           
                 }
                 else
                 {
                      projDB.close();
                      projDB.delete(exProjDB_prefix);// call Enviroment.removeDatabase(null, exProjDB_prefix);
                 }
                 
                 if (dbName.endsWith(")"))
                 {
                      inProjDB_prefix = dbName.substring(0, dbName.length()-1)+ "," + itemValue + ")";
                 }
                 else
                 {
                      inProjDB_prefix = dbName+ "("+ itemValue + ")";
                 }
                 projDB = new DataBase();
                 projDB.open(inProjDB_prefix);
                 //database projDB will be modified by extrasetGrowth method
                 newPrefix = this.minIntrasetGrowth(bdb,dbName,projDB,inProjDB_prefix,itemValue,itemSupp);
                 if (newPrefix!=null) //database projDB is not empty
                 {
                      newPrefix.setSupport(projDB.getSize());
                      //use the sequence size as key
                           key = newPrefix.getNumItems();
                           //the elements in res are ordered by key
                 res.add(key, newPrefix); // res is a global variable
                      numSeq++;
                 projDB.close();
                      //recursive call
                      projectDBminsup(projDB,inProjDB_prefix);
                      projDB.delete(inProjDB_prefix);
                 }
                 else
            {
            projDB.close();
                 projDB.delete(inProjDB_prefix);
            }
                 }//end loop on frequent items
            }//end else (items freq exist)
                 }
                 catch (DataBaseException e)
                 {
                      throw new SpmtccException(e.getMessage(), e.getStackTrace());
                 }
            catch (Exception e)
                 {
                      throw new SpmtccException(e.getMessage(), e.getStackTrace());
                 }
            }
            • 3. Re: DatabaseException
              Greybird-Oracle
              Hi Elena,

              Thanks for sending this additional information.

              I suspect this is a bug in JE that has to do with an interaction between JE built-in background threads -- the checkpointer and the cleaner -- and the removeDatabase method. It is going to take us some time to resolve this. We have not reproduced it yet. I suspect it will take us several weeks to resovle, but please understand that is an estimate.

              When the JE background thread -- checkpointer or cleaner -- gets an exception, it will continue later. So checkpoints and cleaning will not be prevented unless the exception occurs repeatedly.

              Could you please send me an email at mark.hayes at oracle dot com. When I have more information on this problem, I'll email you directly.

              Thanks for reporting this and I apologize for the inconvenience.

              --mark