This discussion is archived
11 Replies Latest reply: Aug 6, 2009 6:16 AM by greybird RSS

DbDump DbVerify read-only cursor OutOfMemory EntityStore 3.1.0 3.3.82

715435 Newbie
Currently Being Moderated
I have a few EntityStores created using 3.1.0. Each has around 100,000 entity records in it. Records are variable length, between 12 bytes and 200KB. "du -sk" reports around 10--15GB. There is a primary key and a secondary key. I opened such a store in read-only, non-transactional mode, then opened a cursor through the primary index and scanned through end to end without any code in the for loop: "for (MyRecord mr : cursor) { ++numRecs; }". On an amd64 PC with 8GB RAM running jdk 1.6.0_07, this gives OutOfMemory error with any -Xmx up to 7GB. Giving more just delays the error. I fiddled with setCacheSize and setCachePercent to no avail. Assuming it was still a problem with my code, I then ran some of the administration scripts provided in the package, such as DbDump and DbVerify. I made sure each was given, at least 50 times the RAM needed to store the largest record. These also crash with OutOfMemory errors. For one store I might suspect disk corruption, but three separate stores are unlikely to all get corrupted. Also, random access via primary key works indefinitely; only the sequential scans bomb.

The expected behavior is that either DbVerify tells me that the database is corrupted, or DbDump dumps the whole database, no matter how large, given RAM to hold a few records at a time. I am very surprised to see this not reported on this forum, because I can create this situation easily by trying to dump a database that is much larger than RAM, which must overwhelmingly be the common usage of Berkeley DB. I am heavily invested in entity stores and want to avoid salvaging/recomputing the data and coding up a different object store if at all possible.

Here are some sample stack traces. I would be happy to provide more info as needed.

Verifying database persist#RecordStore#iitb.PackageName.Record
Checking tree for persist#RecordStore#iitb.PackageName.Record
Environment invalid because of previous exception: com.sleepycat.je.RunRecoveryException
at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:735)
at com.sleepycat.je.Environment.checkEnv(Environment.java:1172)
at com.sleepycat.je.Environment.close(Environment.java:231)
at com.sleepycat.je.util.DbVerify.closeEnv(DbVerify.java:175)
at com.sleepycat.je.util.DbVerify.verify(DbVerify.java:271)
at com.sleepycat.je.util.DbVerify.main(DbVerify.java:63)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
at com.sleepycat.je.log.FileSource.getBytes(FileSource.java:67)
at com.sleepycat.je.log.LogManager.getLogEntryFromLogSource(LogManager.java:660)
at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:605)
at com.sleepycat.je.log.LogManager.get(LogManager.java:731)
at com.sleepycat.je.tree.IN.fetchTarget(IN.java:958)
at com.sleepycat.je.dbi.CursorImpl.fetchCurrent(CursorImpl.java:2138)
at com.sleepycat.je.dbi.CursorImpl.getCurrentAlreadyLatched(CursorImpl.java:1254)
at com.sleepycat.je.dbi.CursorImpl.getNextWithKeyChangeStatus(CursorImpl.java:1400)
at com.sleepycat.je.dbi.CursorImpl.getNext(CursorImpl.java:1326)
at com.sleepycat.je.Cursor.retrieveNextAllowPhantoms(Cursor.java:1572)
at com.sleepycat.je.Cursor.retrieveNext(Cursor.java:1382)
at com.sleepycat.je.DbInternal.retrieveNext(DbInternal.java:82)
at com.sleepycat.je.dbi.DatabaseImpl.walkDatabaseTree(DatabaseImpl.java:738)
at com.sleepycat.je.dbi.DatabaseImpl.verify(DatabaseImpl.java:699)
at com.sleepycat.je.util.DbVerify.verifyOneDbImpl(DbVerify.java:304)
at com.sleepycat.je.util.DbVerify.verify(DbVerify.java:227)
... 1 more
Exit status = false

java.lang.OutOfMemoryError: Java heap space
at com.sleepycat.je.log.LogUtils.readByteArray(LogUtils.java:333)
at com.sleepycat.je.tree.LN.readFromLog(LN.java:665)
at com.sleepycat.je.log.entry.LNLogEntry.readEntry(LNLogEntry.java:126)
at com.sleepycat.je.log.LogManager.getLogEntryFromLogSource(LogManager.java:747)
at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:664)
at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1215)
at com.sleepycat.je.dbi.CursorImpl.fetchCurrent(CursorImpl.java:2320)
at com.sleepycat.je.dbi.CursorImpl.getCurrentAlreadyLatched(CursorImpl.java:1427)
at com.sleepycat.je.dbi.CursorImpl.getNextWithKeyChangeStatus(CursorImpl.java:1573)
at com.sleepycat.je.dbi.CursorImpl.getNext(CursorImpl.java:1499)
at com.sleepycat.je.dbi.DatabaseImpl.walkDatabaseTree(DatabaseImpl.java:1355)
at com.sleepycat.je.dbi.DatabaseImpl.verify(DatabaseImpl.java:1303)
at com.sleepycat.je.util.DbVerify.verifyOneDbImpl(DbVerify.java:371)
at com.sleepycat.je.util.DbVerify.verify(DbVerify.java:275)
at com.sleepycat.je.util.DbVerify.main(DbVerify.java:94)
Exit status = false

Edited by: user11758846 on Aug 3, 2009 8:24 PM
  • 1. Re: DbDump DbVerify read-only cursor OutOfMemory EntityStore 3.1.0 3.3.82
    greybird Expert
    Currently Being Moderated
    Hi,

    Please try your sequential scan with the latest 3.3.82 release. This will tell us whether the problem you're seeing is due to a bug fixed between 3.1 and present, and then we can try to narrow down what it was.

    Thanks,
    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
  • 2. Re: DbDump DbVerify read-only cursor OutOfMemory EntityStore 3.1.0 3.3.82
    greybird Expert
    Currently Being Moderated
    I see that you put 3.3.82 in your subject line but didn't say explicitly that you're using it. Were you using it to do the scan?

    Could you please also post your EnvironmentConfig setup and je.properties file (if any).

    I'm sure we'll be able to figure out what's going on with your app, but it may take a little back and forth.

    Thanks,
    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
  • 3. Re: DbDump DbVerify read-only cursor OutOfMemory EntityStore 3.1.0 3.3.82
    715435 Newbie
    Currently Being Moderated
    I greatly appreciate your help. Yes, I used both my application and provided scripts, for both revisions 3.1.0 and 3.3.82. I am not using any custom je.properties file. My EnvironmentConfig as printed using System.err.println(myStore.getEnvironemt().getConfig()); is

    allowCreate=false
    cacheSize=559271116
    txnNoSync=false
    txnWriteNoSync=false
    {je.env.isTransactional=false, je.env.isReadOnly=true}

    BTW, I had created the stores using 3.1.0, and when I tried to open using 3.3.82, the library complained that I had opened the stores readonly but it wanted to upgrade the log format. So I made a copy, opened for rw and immediately closed it, then used that with 3.3.82. The OutOfMemory problem persists, or I'd just migrate everything and move on.
  • 4. Re: DbDump DbVerify read-only cursor OutOfMemory EntityStore 3.1.0 3.3.82
    greybird Expert
    Currently Being Moderated
    Using JE 3.3.82 I have been able to reproduce the problem you're seeing. This a serious problem and I'll try to make a fix available ASAP. I'll post a status update later today.

    Thanks for reporting this, and I apologize for the bug.

    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
  • 5. Re: DbDump DbVerify read-only cursor OutOfMemory EntityStore 3.1.0 3.3.82
    greybird Expert
    Currently Being Moderated
    It turns out that this problem is related to opening the environment read-only. If you open the environment read-write (you may still open the store read-only), the problem does not occur.

    I still consider this serious, especially considering that the JE utilities open the environment read-only, and we'll treat this as high priority.

    In the mean time, are you able to open the environment read-write as a work around?

    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
  • 6. Re: DbDump DbVerify read-only cursor OutOfMemory EntityStore 3.1.0 3.3.82
    715435 Newbie
    Currently Being Moderated
    Opening only the environment read-write got me past the earlier crash point, but not the whole way through. Opening also the store read-write got me even farther. Now, with -Xmx400m, and giving JE 30% of that (120MB), I can scan a 11GB store end to end. So this is definitely progress! However, my largest record is around 200KB, so it is still unclear why 120MB is required. If I drop to -Xmx100m (30M to JE), I crash, but slightly differently compared to what was happening before (shown below). Summarizing, opening the Env/Store in rw mode is definitely making a difference, but this might not be the whole story. Thanks very much for chasing this down!

    Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
         at com.sleepycat.persist.impl.SimpleFormat$FDouble.newPrimitiveArray(SimpleFormat.java:621)
         at com.sleepycat.persist.impl.PrimitiveArrayFormat.newInstance(PrimitiveArrayFormat.java:79)
         at com.sleepycat.persist.impl.RecordInput.readObject(RecordInput.java:175)
         at com.sleepycat.persist.impl.ReflectionAccessor$ObjectAccess.read(ReflectionAccessor.java:376)
         at com.sleepycat.persist.impl.ReflectionAccessor.readNonKeyFields(ReflectionAccessor.java:275)
         at com.sleepycat.persist.impl.ComplexFormat.readObject(ComplexFormat.java:509)
         at com.sleepycat.persist.impl.PersistEntityBinding.readEntity(PersistEntityBinding.java:79)
         at com.sleepycat.persist.impl.PersistEntityBinding.entryToObject(PersistEntityBinding.java:49)
         at com.sleepycat.persist.EntityValueAdapter.entryToValue(EntityValueAdapter.java:56)
         at com.sleepycat.persist.BasicCursor.returnValue(BasicCursor.java:229)
         at com.sleepycat.persist.BasicCursor.next(BasicCursor.java:81)
         at com.sleepycat.persist.BasicIterator.hasNext(BasicIterator.java:50)
  • 7. Re: DbDump DbVerify read-only cursor OutOfMemory EntityStore 3.1.0 3.3.82
    greybird Expert
    Currently Being Moderated
    I'm glad you're seeing an improvement.

    When I run my test (which is an attempt to mimic the app and data set that you've described), I can read through the entire data set with a 16 MB heap. This is with JE 3.3.82.

    Are you testing with JE 3.1 by chance?

    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
  • 8. Re: DbDump DbVerify read-only cursor OutOfMemory EntityStore 3.1.0 3.3.82
    715435 Newbie
    Currently Being Moderated
    I am using 3.3.82 throughout now. By opening the env and store rw, I can at least make progress on my project---thanks! But I do need ~120MB for JE cache compared to an absolute maximum of a ~100KB per entity record, so the large amount of RAM needed by JE is still unexplained. Hopefully it's all going to be fixed in one fell swoop. I have more data being prepared that will fill ~50---100GB per entity store, so I might still run out of 8GB total RAM at that time.

    One more thing to note is that, prior to starting the cursor scan, my application does tens of thousands of arbitrary key probes in the primary index, many of which are not supposed to be found in the store (say, 50-50). Perhaps you can simulate this in your test code. These probes have no obvious locality and are (expected to be) slow. Meanwhile I can try to switch that off and see if my subsequent RAM requirement reduces.

    Based on GC profiling, it seems there is an intense bout of GC activity around 10--15 thousand keys into the cursor, then the VM gives up. This is on a store with around 50k keys with the record size range mentioned above.
  • 9. Re: DbDump DbVerify read-only cursor OutOfMemory EntityStore 3.1.0 3.3.82
    greybird Expert
    Currently Being Moderated
    Here's my (very simple) test. It's run twice, once passing 'write' and then 'read'.

    Could you please try a couple things:

    1. Try running this with a 16M heap and make sure it succeeds for you like it does for me, on 3.3.82 of course. If it fails, then there is something else different about our systems such as OS or JVM version that is having an impact.

    2. If it succeeds, then please try modifying it to make it more like your app and make it fail. If you're able to do that, it will give me something concrete to debug on my end.

    Thanks,
    --mark
    import java.io.File;
    import com.sleepycat.je.*;
    import com.sleepycat.persist.*;
    import com.sleepycat.persist.model.*;
    
    public class Test {
    
        @Entity
        static class Data {
    
            @PrimaryKey
            long priKey;
    
            @SecondaryKey(relate=Relationship.ONE_TO_ONE)
            long secKey;
    
            byte[] data;
    
            private Data() {}
    
            Data(long k) {
                priKey = k;
                secKey = k;
                data = new byte[100 * 1024];
            }
        }
    
        public static void main(String[] args) throws Exception {
    
            if (args.length != 1 ||
                !(args[0].equals("write") ||
                  args[0].equals("read"))) {
                throw new RuntimeException("One arg required, 'read' or 'write'");
            }
            boolean doRead = args[0].equals("read");
    
            EnvironmentConfig envConfig = new EnvironmentConfig();
            envConfig.setAllowCreate(true);
            //envConfig.setReadOnly(doRead);
            Environment env = new Environment(new File("./tmp"), envConfig);
    
            StoreConfig storeConfig = new StoreConfig();
            storeConfig.setAllowCreate(true);
            storeConfig.setReadOnly(doRead);
            EntityStore store = new EntityStore(env, "foo", storeConfig);
    
            PrimaryIndex<Long, Data> index =
                store.getPrimaryIndex(Long.class, Data.class);
    
            if (doRead) {
                EntityCursor<Data> cursor = index.entities();
                long i = 0;
                for (Data d : cursor) {
                    i += 1;
                    if (i % 1000 == 0) {
                        System.out.println("Read: " + i);
                        //System.out.println(env.getStats(null));
                    }
                }
                cursor.close();
            } else {
                for (long i = 1; i <= 100000; i += 1) {
                    index.put(new Data(i));
                    if (i % 1000 == 0) {
                        System.out.println("Write: " + i);
                    }
                }
            }
    
            store.close();
            env.close();
        }
    }
  • 10. Re: DbDump DbVerify read-only cursor OutOfMemory EntityStore 3.1.0 3.3.82
    715435 Newbie
    Currently Being Moderated
    First, on 3.3.82 on jdk 1.6.0_07, your code ran fine. Second, I have been trying to make it misbehave by making record sizes random (approximately truncated power law in my application) and actually reading the whole record, but have not been able to succeed yet. There are other differences that could be significant (many methods in my entity class, my entity class not being an inner static class but a package level class, etc.), so I will continue trying a few other tweaks.

    However, it is possible (?) that the problem arises only when scanning a store written using 3.1.0 (and migrated to be read by 3.3.82). Is it possible to dump my 3.1.0 stores to flat file and load it into 3.3.82? If you can send the few lines I need to change in DbDump of 3.1.0 that I need to change to create the env and store read-write, I can try that. I could scan the code and figure out, but your guiding me might be faster. Thanks.
  • 11. Re: DbDump DbVerify read-only cursor OutOfMemory EntityStore 3.1.0 3.3.82
    greybird Expert
    Currently Being Moderated
    However, it is possible (?) that the problem arises only when scanning a store written using 3.1.0 (and migrated to be read by 3.3.82). Is it possible to dump my 3.1.0 stores to flat file and load it into 3.3.82? If you can send the few lines I need to change in DbDump of 3.1.0 that I need to change to create the env and store read-write, I can try that. I could scan the code and figure out, but your guiding me might be faster. Thanks.
    Please send me email -- mark.hayes @ o.com -- and I can send you a quick patch.

    Another approach is to run my test in 'write' mode against the 3.1.0 jar, then in 'read' mode against the 3.3.82 jar. The test may have to be recompiled to run with 3.1.0.

    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points