This discussion is archived
1 Reply Latest reply: Nov 8, 2012 4:42 AM by Ashok_Ora RSS

Insert data to Berkeley DB very poor performance in Big Data

973153 Newbie
Currently Being Moderated
I have already held the 80G Berkeley DB file. I measure the average insert speed is 8ms for one record(32 byte key/100 byte value) without transaction.

Compare to insert to empty database with same interface, the average speed is 3~6 us.
  • 1. Re: Insert data to Berkeley DB very poor performance in Big Data
    Ashok_Ora Explorer
    Currently Being Moderated
    Hello,

    When the database is empty, the inserts are being written to Berkeley DB's buffer cache, so you're able to insert at memory speeds. Once the amount of data that you store exceeds the available memory on the machine, Berkeley DB will need to write data to disk, which will increase latency.

    How much memory are you using for Berkeley DB? How much data do you need to load into Berkeley DB? There are many options in Berkeley DB that can help. For example, Berkeley DB has a feature to write database pages to disk in the background (trickle). You can run Berkeley DB as a pure in-memory database.

    It would be great if you could post some additional information about your application and also the Berkeley DB configuration that you're using for your test.

    Thanks and warm regards.
    ashok

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points