This discussion is archived
6 Replies Latest reply: May 15, 2012 11:47 AM by greybird RSS

5.0.34 Migration running out of memory

921651 Newbie
Currently Being Moderated
Hello,

We are trying to migrate a 4.0.92 database to 5.0.34 (with duplicates), but it keeps running out of memory in the migration process (when we first try to open it with the new version). We already gave it 60GB but it's not enough.
The database is roughly 2TB big, 225M entries, ~4KB value size in average (I know it's too big, we are trying to make it smaller but first we need an efficient way to iterate over all the entries, currently it is veeeery slow, we hope DiskOrderedCursor would make it much faster).
As far as I can tell, it is trying to preload all the internal nodes in memory, but our key size is 30 bytes long at most, so why is not 60GB enough?
This is the relevant part of the stack trace:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
     at java.util.HashMap.resize(HashMap.java:477)
     at java.util.HashMap.addEntry(HashMap.java:768)
     at java.util.HashMap.put(HashMap.java:402)
     at com.sleepycat.je.dbi.SortedLSNTreeWalker.addEntryToLsnMap(SortedLSNTreeWalker.java:677)
     at com.sleepycat.je.dbi.SortedLSNTreeWalker.addToLsnINMap(SortedLSNTreeWalker.java:662)
     at com.sleepycat.je.dbi.SortedLSNTreeWalker.accumulateLSNs(SortedLSNTreeWalker.java:442)
     at com.sleepycat.je.dbi.SortedLSNTreeWalker.fetchAndProcessLSN(SortedLSNTreeWalker.java:527)
     at com.sleepycat.je.dbi.SortedLSNTreeWalker.processAccumulatedLSNs(SortedLSNTreeWalker.java:353)
     at com.sleepycat.je.dbi.SortedLSNTreeWalker.walkInternal(SortedLSNTreeWalker.java:341)
     at com.sleepycat.je.dbi.EnvironmentImpl$PreloadLSNTreeWalker.walk(EnvironmentImpl.java:3131)
     at com.sleepycat.je.dbi.EnvironmentImpl.preload(EnvironmentImpl.java:2985)
     at com.sleepycat.je.tree.dupConvert.DupConvert.preloadAllDatabases(DupConvert.java:210)
     at com.sleepycat.je.tree.dupConvert.DupConvert.convertDatabases(DupConvert.java:151)
     at com.sleepycat.je.dbi.EnvironmentImpl.convertDupDatabases(EnvironmentImpl.java:1758)
     at com.sleepycat.je.dbi.EnvironmentImpl.finishInit(EnvironmentImpl.java:676)
     at com.sleepycat.je.dbi.DbEnvPool.getEnvironment(DbEnvPool.java:210)
     at com.sleepycat.je.Environment.makeEnvironmentImpl(Environment.java:246)
     at com.sleepycat.je.Environment.<init>(Environment.java:227)
     at com.sleepycat.je.Environment.<init>(Environment.java:170)

Thanks,

Diego
  • 1. Re: 5.0.34 Migration running out of memory
    greybird Expert
    Currently Being Moderated
    Hello Diego,

    Although it is only loading the internal nodes, in duplicates DBs the internal nodes contain both the key and the data. In any case, obviously you don't have enough memory to preload all duplicate DBs.

    This note from the change log has a suggestion that should help:
    >

    To make the conversion predictable during deployment, users should measure the conversion time on a non-production system before upgrading a deployed system. When duplicates are converted, the Btree internal nodes are preloaded into the JE cache. A new configuration option, EnvironmentConfig.ENV_DUP_CONVERT_PRELOAD_ALL, can be set to false to optimize this process if the cache is not large enough to hold the internal nodes for all databases. For more information, see the javadoc for this property. [#19165]
    >

    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
  • 2. Re: 5.0.34 Migration running out of memory
    921651 Newbie
    Currently Being Moderated
    Thanks Mark for the answer, I forgot to mention that I already tried setting that flag to false with no luck.

    With the old log format (pre version 8, introduced in 5.0.34), I thought DIN and DBIN didn't have the entry values. This is taken from the DIN dupkey attribute javadoc: "Full key for this set of duplicates. For example, if the tree contains k1/d1, k1/d2, k1/d3, the dupKey = k1". And the same attribute is present in DBIN. Can someone clarify why the "internal" values would contain the entry values for databases with duplicates?

    Is there any other way of "migrating" the database information? Should I export/import? (as I mentioned before, export is really slow because it relies on key-oredered traversal)
    Thanks,

    Diego

    If it's of any help, this is a fragment of the heap histogram right before running out of memory:

    num #instances #bytes class name
    ----------------------------------------------
    1: 224110179 26270915504 [B
    2: 216191291 10377181968 java.util.HashMap$Entry
    3: 216072294 6914313408 com.sleepycat.je.dbi.SortedLSNTreeWalker$INEntry
    4: 216218458 5189242992 java.lang.Long
    5: 720141 4528335232 [J
    6: 29452 4301448560 [Ljava.util.HashMap$Entry;
    7: 660084 2705606392 [[B
    8: 665470 1868586832 [Lcom.sleepycat.je.tree.Node;
    9: 5864758 187672256 com.sleepycat.je.tree.LN
    10: 708745 136079040 com.sleepycat.je.tree.BIN
    11: 720802 57664160 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync
    12: 720801 46131264 com.sleepycat.je.latch.SharedLatch
    13: 728742 37603448 [C
    14: 723861 34745328 java.util.concurrent.ConcurrentHashMap$HashEntry
    15: 729152 29166080 java.lang.String
    16: 708760 22680320 com.sleepycat.je.utilint.TinyHashSet
    17: 720803 17299272 java.util.concurrent.locks.ReentrantReadWriteLock$Sync$ThreadLocalHoldCounter
    18: 720803 17299272 java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock
    19: 720803 17299272 java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock
    20: 660082 15841968 com.sleepycat.je.tree.INKeyRep$Default
    21: 454114 10898736 com.sleepycat.je.tree.INTargetRep$Default
    22: 720 8456320 [Ljava.util.concurrent.ConcurrentHashMap$HashEntry;
    23: 214922 7046848 [S
    24: 211356 6763392 com.sleepycat.je.tree.INTargetRep$Sparse
    25: 116971 5614608 com.sleepycat.je.cleaner.DbFileSummary
    26: 28385 4014880 <constMethodKlass>
    27: 28385 3414136 <methodKlass>
    28: 2465 2831544 <constantPoolKlass>
    29: 47715 2373352 <symbolKlass>
    30: 60707 1942624 com.sleepycat.je.tree.INKeyRep$MaxKeySize
    31: 11961 1913760 com.sleepycat.je.tree.IN
    32: 29406 1881984 java.util.HashMap
    33: 29393 1881152 java.util.TreeMap$Entry
    34: 2465 1862640 <instanceKlassKlass>
    35: 28849 1846336 com.sleepycat.je.cleaner.FileSummary
    36: 2080 1776192 <constantPoolCacheKlass>
    37: 28934 694416 java.util.HashSet
    38: 2774 510416 java.lang.Class
    39: 3894 359912 [[I
    40: 834 329584 <methodDataKlass>
    41: 280 163520 <objArrayKlassKlass>
    42: 2351 152472 [I
    43: 815 70800 [Ljava.lang.Object;
    44: 431 65512 java.lang.reflect.Method
    45: 3073 49168 java.lang.Object
    46: 908 43584 java.util.concurrent.locks.ReentrantLock$NonfairSync
    47: 720 34560 java.util.concurrent.ConcurrentHashMap$Segment
    48: 578 27744 java.util.Hashtable$Entry
    49: 800 25600 com.sleepycat.je.utilint.IntStat
    50: 412 23000 [Ljava.lang.String;
    51: 311 19904 java.util.LinkedHashMap$Entry
    52: 322 18032 java.lang.ref.SoftReference
    53: 52 17472 org.apache.xerces.impl.dv.xs.XSSimpleTypeDecl
    54: 140 16800 java.lang.reflect.Constructor
    55: 35 14016 [Lorg.apache.xerces.util.SymbolHash$Entry;
    56: 60 12768 [Ljava.util.Hashtable$Entry;
    57: 126 12096 java.lang.Package
    58: 68 11968 com.sleepycat.je.tree.dupConvert.DIN
  • 3. Re: 5.0.34 Migration running out of memory
    greybird Expert
    Currently Being Moderated
    >
    With the old log format (pre version 8, introduced in 5.0.34), I thought DIN and DBIN didn't have the entry values. This is taken from the DIN dupkey attribute javadoc: "Full key for this set of duplicates. For example, if the tree contains k1/d1, k1/d2, k1/d3, the dupKey = k1". And the same attribute is present in DBIN. Can someone clarify why the "internal" values would contain the entry values for databases with duplicates?
    >

    In the old log format, the key of each slot in a DBIN is the record data. In other words, the record data is in the tree. But this is an academic issue. It is not going to lead to a solution.

    >
    Is there any other way of "migrating" the database information? Should I export/import? (as I mentioned before, export is really slow because it relies on key-oredered traversal)
    >

    You are running out of memory in the preload phase and most of the memory is allocated by the preload process, not the Btree. I will think about possible solutions and get back to you later today.

    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
  • 4. Re: 5.0.34 Migration running out of memory
    greybird Expert
    Currently Being Moderated
    I think the solution is for us to give you a way to specify the PreloadConfig that is used during the duplicate conversion process. Please send me email, mark.hayes at o.com (o == oracle), so I can work with you on this.

    Thanks,
    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
  • 5. Re: 5.0.34 Migration running out of memory
    936646 Newbie
    Currently Being Moderated
    I have meet the same problem.
    After trying to call DiskOrderedCursorConfig.setInternalMemoryLimitVoid to use a fix memory(128M),I found it works well.
    You can try it.

    Edited by: 933643 on 2012-5-10 下午11:38
  • 6. Re: 5.0.34 Migration running out of memory
    greybird Expert
    Currently Being Moderated
    If anyone is still running into this problem, please be sure to use the new 5.0.48 release and the JE 4.1.20 pre-upgrade utility. If you still have out-of-memory problems during the upgrade to JE 5, post here and I will suggest solutions.

    Note that DiskOrderedCursorConfig.setInternalMemoryLimitVoid is a good solution if you're using a DiskOrderedCursor -- thanks for posting this -- but it doesn't apply to an out-of-memory error that occurs while you're upgrading to JE 5 (in the Environment constructor).

    --mark

    Edited by: greybird on May 15, 2012 11:47 AM

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points