Forum Stats

  • 3,769,960 Users
  • 2,253,036 Discussions
  • 7,875,253 Comments

Discussions

How to achieve for StoredSortedMap to persist data as soon as possible

3297832
3297832 Member Posts: 3
edited Aug 19, 2016 3:26AM in Berkeley DB Java Edition

I am using StoredSortedMap from Berkley Db Java edition. I want to achieve that the data I add to map are persisted to disk as soon as possible.

When I set database config as transactional, than I think every time I add something to the map its running in their own transaction and when put method is returned that the data were persisted on the disk, and in case of failure I will be able to read the data.

dbConfig.setTransactional(true);

Problem is that the performance is slow and is not enough for our workload.

The second possibility is to store more data in one transaction something like this:

dbConfig.setTransactional(true);
CurrentTransaction.getInstance(berkleyDbManager.dbEnv).beginTransaction(null);    for(long x = 0; x < 100; x++) {            map.put(x,  (byte[])contentToStore);    }  CurrentTransaction.getInstance(manager.dbEnv).commitTransaction();

And when the transaction is commited I think I can be sure that the data are persisted on disk and I am safe in case of failure.

I want to ask if with the following solution I can achieve the same result. I will set the DatabaseConfig as non transactional:

 dbConfig.setTransactional(true);

And I will create the another Thread that will regularly call  sync method on database environment.

berkleyDbManager.dbEnv.sync();

Can I be sure that after sync finished the everything that was put in the map is correctly persisted on disk? and in case  of crash/failure BerkleyDb will recover the data?

Answers

  • Greybird-Oracle
    Greybird-Oracle Member Posts: 2,690
    edited Aug 17, 2016 12:59PM

    Hi,

    You don't need to give up transactions to improve performance. See:

    Non-Durable Transactions

    The default durability, SYNC, is the most conservative since it protects against machine crashes; this performs slowly because the data must be fsync'ed all the way to disk. Most apps can use NO_SYNC or WRITE_NO_SYNC, to compromise durability, to a small degree, for performance.

    --mark

  • 3297832
    3297832 Member Posts: 3
    edited Aug 18, 2016 5:05AM

    Hello,

    The NO_SYNC or WRITE_NO_SYNC transaction I can use, but than I don't know when my data will be write on disk, so I can be sure that are persisted. This is something that I need to know.

    I am curios if the call dbEnv.sync() will also perform  data fsync to  disk, and If I can be sure that everything that was write to map before sync() is called is persisted on disk after sync() finished. I don't need transaction, I put data from one thread and read them from another thread like Queue.

  • Greybird-Oracle
    Greybird-Oracle Member Posts: 2,690
    edited Aug 18, 2016 4:35PM

    The doc I mentioned describes the guarantees made by WRITE_NO_SYNC, and these guarantees are normally sufficient for most apps. If you need SYNC durability after completing a sequence of 'put' operations that do *not* use SYNC durability (I think this is what you're asking), there are several ways to achieve it:

    1. Commit the last txn in the sequence with SYNC durability.

    2. After the sequence, commit a no-op txn with SYNC durability, for example, update the last record again.

    3. After the sequence, call Environment.flushLog(true).

    Environment.sync does a checkpoint, which is much more costly than you need to provide SYNC durability. Environment.flushLog should be used instead of Environment.sync for this purpose.

    --mark

  • 3297832
    3297832 Member Posts: 3
    edited Aug 19, 2016 3:26AM

    Hello,

    thank you flushLog, it is what I was looking for. I was looking at the method and it is not synchronized. I hope it is not  a problem If I have the separate thread that every 500MS would call the environmen.flushLog(), parallel with the thread that is calling put on StoredSortedMap.

    Thank you very much for your help.

This discussion has been closed.