3 Replies Latest reply: Mar 29, 2012 12:11 AM by 927146 RSS

    Out of Memory Error when using BerkelyDB Java edition's Collections

    927146
      I have a problem similar to the one mentioned in the following discussion.

      5.0.34 Migration running out of memory

      The discussion ended without any solution.

      My problem:

      I need to create a collection having 10 million entries of the form 1,word1: 2,word2 .... etc. I defined a Key class, Data Class, SampleDatabase, SampleViews class and a program which inserts the data. I followed the tutorial in the following link http://docs.oracle.com/cd/E17277_02/html/collections/tutorial/BasicProgram.html. When i run the program, i get the error shown below. The problem is that i am running out of memory during the initial load phase itself. Any help regarding this issue would be greatly appreciated.

      java.lang.OutOfMemoryError: Java heap space
      at com.sleepycat.je.log.LogUtils.readBytesNoLength(LogUtils.java:338)
      at com.sleepycat.je.tree.LN.readFromLog(LN.java:773)
      at com.sleepycat.je.log.entry.LNLogEntry.readBaseLNEntry(LNLogEntry.java:196)
      at com.sleepycat.je.log.entry.LNLogEntry.readEntry(LNLogEntry.java:130)
      at com.sleepycat.je.log.LogManager.getLogEntryFromLogSource(LogManager.java:965)
      at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:807)
      at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:758)
      at com.sleepycat.je.log.LogManager.getLogEntryHandleFileNotFound(LogManager.java:819)
      at com.sleepycat.je.txn.UndoReader.<init>(UndoReader.java:41)
      at com.sleepycat.je.txn.Txn.undo(Txn.java:1156)
      at com.sleepycat.je.txn.Txn.abortInternal(Txn.java:1081)
      at com.sleepycat.je.txn.Txn.abort(Txn.java:1025)
      at com.sleepycat.je.txn.Txn.abort(Txn.java:998)
      at com.sleepycat.je.Transaction.abort(Transaction.java:123)
      at com.sleepycat.collections.CurrentTransaction.abortTransaction(CurrentTransaction.java:313)
      at com.sleepycat.collections.TransactionRunner.run(TransactionRunner.java:235)
      at berkelydbjava.Sample.run(Sample.java:97)
      at berkelydbjava.Sample.main(Sample.java:49)
      Exception during database close:
      com.sleepycat.je.EnvironmentFailureException: (JE 5.0.34) JAVA_ERROR: Java Error occurred, recovery may not be possible.
      at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1492)
      at com.sleepycat.je.Database.checkEnv(Database.java:1894)
      at com.sleepycat.je.Database.closeInternal(Database.java:411)
      at com.sleepycat.je.Database.close(Database.java:348)
      at berkelydbjava.SampleDatabase.close(SampleDatabase.java:89)
      at berkelydbjava.Sample.close(Sample.java:84)
      at berkelydbjava.Sample.main(Sample.java:59)
      Caused by: java.lang.OutOfMemoryError: Java heap space
      at com.sleepycat.je.log.LogUtils.readBytesNoLength(LogUtils.java:338)
      at com.sleepycat.je.tree.LN.readFromLog(LN.java:773)
      at com.sleepycat.je.log.entry.LNLogEntry.readBaseLNEntry(LNLogEntry.java:196)
      at com.sleepycat.je.log.entry.LNLogEntry.readEntry(LNLogEntry.java:130)
      at com.sleepycat.je.log.LogManager.getLogEntryFromLogSource(LogManager.java:965)
      at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:807)
      at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:758)
      at com.sleepycat.je.log.LogManager.getLogEntryHandleFileNotFound(LogManager.java:819)
      at com.sleepycat.je.txn.UndoReader.<init>(UndoReader.java:41)
      at com.sleepycat.je.txn.Txn.undo(Txn.java:1156)
      at com.sleepycat.je.txn.Txn.abortInternal(Txn.java:1081)
      at com.sleepycat.je.txn.Txn.abort(Txn.java:1025)
      at com.sleepycat.je.txn.Txn.abort(Txn.java:998)
      at com.sleepycat.je.Transaction.abort(Transaction.java:123)
      at com.sleepycat.collections.CurrentTransaction.abortTransaction(CurrentTransaction.java:313)
      at com.sleepycat.collections.TransactionRunner.run(TransactionRunner.java:235)
      at berkelydbjava.Sample.run(Sample.java:97)
      at berkelydbjava.Sample.main(Sample.java:49)
      Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
      at java.lang.Class.getDeclaredMethods0(Native Method)
      at java.lang.Class.privateGetDeclaredMethods(Class.java:2442)
      at java.lang.Class.getDeclaredMethod(Class.java:1952)
      at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1398)
      at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:69)
      at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:455)
      at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:430)
      at java.security.AccessController.doPrivileged(Native Method)
      at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:430)
      at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:327)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1130)
      at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1528)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1493)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
      at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:346)
      at com.sleepycat.bind.serial.SerialBinding.objectToEntry(SerialBinding.java:179)
      at com.sleepycat.collections.DataView.useKey(DataView.java:502)
      at com.sleepycat.collections.DataCursor.initForPut(DataCursor.java:805)
      at com.sleepycat.collections.DataCursor.put(DataCursor.java:751)
      at com.sleepycat.collections.StoredContainer.putKeyValue(StoredContainer.java:321)
      at com.sleepycat.collections.StoredMap.put(StoredMap.java:279)
      at berkelydbjava.Sample.addParts(Sample.java:140)
      at berkelydbjava.Sample.access$100(Sample.java:28)
      at berkelydbjava.Sample$PopulateDatabase.doWork(Sample.java:107)
      at com.sleepycat.collections.TransactionRunner.run(TransactionRunner.java:226)
      at berkelydbjava.Sample.run(Sample.java:97)
      at berkelydbjava.Sample.main(Sample.java:49)
      Java Result: 1

      Thanks.

      Edited by: user9108969 on Mar 28, 2012 6:42 PM
        • 1. Re: Out of Memory Error when using BerkelyDB Java edition's Collections
          Greybird-Oracle
          It looks like you are inserting all 10 million entries in a single transaction (one call to TransactionRunner.run). If so, that's not a good idea -- you'll use extremely large amounts of memory for the transaction locks -- and you'll need to use smaller transactions.

          Also, what is the heap size of your java process and your EnvironmentConfig properties?

          --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
          • 2. Re: Out of Memory Error when using BerkelyDB Java edition's Collections
            927146
            I am trying to increase the number of transactions. I will keep you posted if i get through this.

            Edited by: user9108969 on Mar 28, 2012 8:47 PM
            • 3. Re: Out of Memory Error when using BerkelyDB Java edition's Collections
              927146
              I increased the number of transactions and it worked. Thanks a lot.