6 Replies Latest reply: May 15, 2012 1:47 PM by Greybird-Oracle RSS

    5.0.34 Migration running out of memory

    921651
      Hello,

      We are trying to migrate a 4.0.92 database to 5.0.34 (with duplicates), but it keeps running out of memory in the migration process (when we first try to open it with the new version). We already gave it 60GB but it's not enough.
      The database is roughly 2TB big, 225M entries, ~4KB value size in average (I know it's too big, we are trying to make it smaller but first we need an efficient way to iterate over all the entries, currently it is veeeery slow, we hope DiskOrderedCursor would make it much faster).
      As far as I can tell, it is trying to preload all the internal nodes in memory, but our key size is 30 bytes long at most, so why is not 60GB enough?
      This is the relevant part of the stack trace:

      Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
           at java.util.HashMap.resize(HashMap.java:477)
           at java.util.HashMap.addEntry(HashMap.java:768)
           at java.util.HashMap.put(HashMap.java:402)
           at com.sleepycat.je.dbi.SortedLSNTreeWalker.addEntryToLsnMap(SortedLSNTreeWalker.java:677)
           at com.sleepycat.je.dbi.SortedLSNTreeWalker.addToLsnINMap(SortedLSNTreeWalker.java:662)
           at com.sleepycat.je.dbi.SortedLSNTreeWalker.accumulateLSNs(SortedLSNTreeWalker.java:442)
           at com.sleepycat.je.dbi.SortedLSNTreeWalker.fetchAndProcessLSN(SortedLSNTreeWalker.java:527)
           at com.sleepycat.je.dbi.SortedLSNTreeWalker.processAccumulatedLSNs(SortedLSNTreeWalker.java:353)
           at com.sleepycat.je.dbi.SortedLSNTreeWalker.walkInternal(SortedLSNTreeWalker.java:341)
           at com.sleepycat.je.dbi.EnvironmentImpl$PreloadLSNTreeWalker.walk(EnvironmentImpl.java:3131)
           at com.sleepycat.je.dbi.EnvironmentImpl.preload(EnvironmentImpl.java:2985)
           at com.sleepycat.je.tree.dupConvert.DupConvert.preloadAllDatabases(DupConvert.java:210)
           at com.sleepycat.je.tree.dupConvert.DupConvert.convertDatabases(DupConvert.java:151)
           at com.sleepycat.je.dbi.EnvironmentImpl.convertDupDatabases(EnvironmentImpl.java:1758)
           at com.sleepycat.je.dbi.EnvironmentImpl.finishInit(EnvironmentImpl.java:676)
           at com.sleepycat.je.dbi.DbEnvPool.getEnvironment(DbEnvPool.java:210)
           at com.sleepycat.je.Environment.makeEnvironmentImpl(Environment.java:246)
           at com.sleepycat.je.Environment.<init>(Environment.java:227)
           at com.sleepycat.je.Environment.<init>(Environment.java:170)

      Thanks,

      Diego
        • 1. Re: 5.0.34 Migration running out of memory
          Greybird-Oracle
          Hello Diego,

          Although it is only loading the internal nodes, in duplicates DBs the internal nodes contain both the key and the data. In any case, obviously you don't have enough memory to preload all duplicate DBs.

          This note from the change log has a suggestion that should help:
          >

          To make the conversion predictable during deployment, users should measure the conversion time on a non-production system before upgrading a deployed system. When duplicates are converted, the Btree internal nodes are preloaded into the JE cache. A new configuration option, EnvironmentConfig.ENV_DUP_CONVERT_PRELOAD_ALL, can be set to false to optimize this process if the cache is not large enough to hold the internal nodes for all databases. For more information, see the javadoc for this property. [#19165]
          >

          --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
          • 2. Re: 5.0.34 Migration running out of memory
            921651
            Thanks Mark for the answer, I forgot to mention that I already tried setting that flag to false with no luck.

            With the old log format (pre version 8, introduced in 5.0.34), I thought DIN and DBIN didn't have the entry values. This is taken from the DIN dupkey attribute javadoc: "Full key for this set of duplicates. For example, if the tree contains k1/d1, k1/d2, k1/d3, the dupKey = k1". And the same attribute is present in DBIN. Can someone clarify why the "internal" values would contain the entry values for databases with duplicates?

            Is there any other way of "migrating" the database information? Should I export/import? (as I mentioned before, export is really slow because it relies on key-oredered traversal)
            Thanks,

            Diego

            If it's of any help, this is a fragment of the heap histogram right before running out of memory:

            num #instances #bytes class name
            ----------------------------------------------
            1: 224110179 26270915504 [B
            2: 216191291 10377181968 java.util.HashMap$Entry
            3: 216072294 6914313408 com.sleepycat.je.dbi.SortedLSNTreeWalker$INEntry
            4: 216218458 5189242992 java.lang.Long
            5: 720141 4528335232 [J
            6: 29452 4301448560 [Ljava.util.HashMap$Entry;
            7: 660084 2705606392 [[B
            8: 665470 1868586832 [Lcom.sleepycat.je.tree.Node;
            9: 5864758 187672256 com.sleepycat.je.tree.LN
            10: 708745 136079040 com.sleepycat.je.tree.BIN
            11: 720802 57664160 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync
            12: 720801 46131264 com.sleepycat.je.latch.SharedLatch
            13: 728742 37603448 [C
            14: 723861 34745328 java.util.concurrent.ConcurrentHashMap$HashEntry
            15: 729152 29166080 java.lang.String
            16: 708760 22680320 com.sleepycat.je.utilint.TinyHashSet
            17: 720803 17299272 java.util.concurrent.locks.ReentrantReadWriteLock$Sync$ThreadLocalHoldCounter
            18: 720803 17299272 java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock
            19: 720803 17299272 java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock
            20: 660082 15841968 com.sleepycat.je.tree.INKeyRep$Default
            21: 454114 10898736 com.sleepycat.je.tree.INTargetRep$Default
            22: 720 8456320 [Ljava.util.concurrent.ConcurrentHashMap$HashEntry;
            23: 214922 7046848 [S
            24: 211356 6763392 com.sleepycat.je.tree.INTargetRep$Sparse
            25: 116971 5614608 com.sleepycat.je.cleaner.DbFileSummary
            26: 28385 4014880 <constMethodKlass>
            27: 28385 3414136 <methodKlass>
            28: 2465 2831544 <constantPoolKlass>
            29: 47715 2373352 <symbolKlass>
            30: 60707 1942624 com.sleepycat.je.tree.INKeyRep$MaxKeySize
            31: 11961 1913760 com.sleepycat.je.tree.IN
            32: 29406 1881984 java.util.HashMap
            33: 29393 1881152 java.util.TreeMap$Entry
            34: 2465 1862640 <instanceKlassKlass>
            35: 28849 1846336 com.sleepycat.je.cleaner.FileSummary
            36: 2080 1776192 <constantPoolCacheKlass>
            37: 28934 694416 java.util.HashSet
            38: 2774 510416 java.lang.Class
            39: 3894 359912 [[I
            40: 834 329584 <methodDataKlass>
            41: 280 163520 <objArrayKlassKlass>
            42: 2351 152472 [I
            43: 815 70800 [Ljava.lang.Object;
            44: 431 65512 java.lang.reflect.Method
            45: 3073 49168 java.lang.Object
            46: 908 43584 java.util.concurrent.locks.ReentrantLock$NonfairSync
            47: 720 34560 java.util.concurrent.ConcurrentHashMap$Segment
            48: 578 27744 java.util.Hashtable$Entry
            49: 800 25600 com.sleepycat.je.utilint.IntStat
            50: 412 23000 [Ljava.lang.String;
            51: 311 19904 java.util.LinkedHashMap$Entry
            52: 322 18032 java.lang.ref.SoftReference
            53: 52 17472 org.apache.xerces.impl.dv.xs.XSSimpleTypeDecl
            54: 140 16800 java.lang.reflect.Constructor
            55: 35 14016 [Lorg.apache.xerces.util.SymbolHash$Entry;
            56: 60 12768 [Ljava.util.Hashtable$Entry;
            57: 126 12096 java.lang.Package
            58: 68 11968 com.sleepycat.je.tree.dupConvert.DIN
            • 3. Re: 5.0.34 Migration running out of memory
              Greybird-Oracle
              >
              With the old log format (pre version 8, introduced in 5.0.34), I thought DIN and DBIN didn't have the entry values. This is taken from the DIN dupkey attribute javadoc: "Full key for this set of duplicates. For example, if the tree contains k1/d1, k1/d2, k1/d3, the dupKey = k1". And the same attribute is present in DBIN. Can someone clarify why the "internal" values would contain the entry values for databases with duplicates?
              >

              In the old log format, the key of each slot in a DBIN is the record data. In other words, the record data is in the tree. But this is an academic issue. It is not going to lead to a solution.

              >
              Is there any other way of "migrating" the database information? Should I export/import? (as I mentioned before, export is really slow because it relies on key-oredered traversal)
              >

              You are running out of memory in the preload phase and most of the memory is allocated by the preload process, not the Btree. I will think about possible solutions and get back to you later today.

              --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
              • 4. Re: 5.0.34 Migration running out of memory
                Greybird-Oracle
                I think the solution is for us to give you a way to specify the PreloadConfig that is used during the duplicate conversion process. Please send me email, mark.hayes at o.com (o == oracle), so I can work with you on this.

                Thanks,
                --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
                • 5. Re: 5.0.34 Migration running out of memory
                  liang_mic
                  I have meet the same problem.
                  After trying to call DiskOrderedCursorConfig.setInternalMemoryLimitVoid to use a fix memory(128M),I found it works well.
                  You can try it.

                  Edited by: 933643 on 2012-5-10 下午11:38
                  • 6. Re: 5.0.34 Migration running out of memory
                    Greybird-Oracle
                    If anyone is still running into this problem, please be sure to use the new 5.0.48 release and the JE 4.1.20 pre-upgrade utility. If you still have out-of-memory problems during the upgrade to JE 5, post here and I will suggest solutions.

                    Note that DiskOrderedCursorConfig.setInternalMemoryLimitVoid is a good solution if you're using a DiskOrderedCursor -- thanks for posting this -- but it doesn't apply to an out-of-memory error that occurs while you're upgrading to JE 5 (in the Environment constructor).

                    --mark

                    Edited by: greybird on May 15, 2012 11:47 AM