This discussion is archived
6 Replies Latest reply: Jul 26, 2012 2:54 PM by greybird RSS

Optimizing huge bulk loads

vinothchandar Newbie
Currently Being Moderated
Hi,

As a part of removing sorted duplicates from Voldemort, I am trying to come up with a data conversion utility that simply scans an old BDB environment and writes that data into a new environment.
So, looking for confirmation about the fastest way to do these bulk writes, given that these puts don't have to be transactional (even though Voldemort does use transactions normally) and we can redo the whole run if ever it fails somehow.

I see huge improvements (5K per second to 40K per second) when I turn deferred writes on. Is this the way to go? Are there any other configs that could help?

Thanks
Vinoth
  • 1. Re: Optimizing huge bulk loads
    909806 Newbie
    Currently Being Moderated
    Did you time all the way through your conversion including checkpointing (via sync() or close())?

    Using deferred writes...defers writing, but it has to be done eventually, so if your benchmark is only looking at the speed during the first part of the data load, then it will be much much faster because it's not actually writing to the filesystem yet (since it's pushing this work off until as late as possible).

    From my reading of the Getting Started Guide about deferred writes, at http://docs.oracle.com/cd/E17277_02/html/GettingStartedGuide/databases.html#dwdatabase, this sounds like a good situation in which to use them, but I'm wondering if the speed difference will be as large as you are seeing so far.
  • 2. Re: Optimizing huge bulk loads
    greybird Expert
    Currently Being Moderated
    Yes, Jeffstyr is right, this is what deferred-write mode is intended for.

    If you can redo the entire load (delete the env directory and start over) if there is a failure, then disabling checkpoints during the load has a performance advantage. But then you must perform a checkpoint to flush it all to disk, and this needs to be included in your measurement as Jeffstyr said. Also, if you don't have enough JE cache to keep the entire thing in memory, eviction will occur during the load and you may be better off to do checkpoints every so often; this requires experimentation.

    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
  • 3. Re: Optimizing huge bulk loads
    vinothchandar Newbie
    Currently Being Moderated
    Hi Jeff,

    The numbers do not surprise me since I am doing this on SSD. I loaded an entire 178 million entry database. Btw, these inserts are strictly in Btree order i.e I am doing a scan on one database in key order and inserting to another. Guess this must be the best case for bulk load.

    Mark, sure will remember to do a sync() and properly shutdown the database. I cannot possibly hold everything in memory. But, checkpointing longer will definitely help out.

    Thanks
    Vinoth
  • 4. Re: Optimizing huge bulk loads
    greybird Expert
    Currently Being Moderated
    I'm not arguing with the results you get from using deferred-write. But for others benefit I need to point out a couple things to avoid misunderstandings.

    SSDs help with random IO, which means in general they help with reads and not so much with writes in JE, since JE uses append-only storage making writes mostly sequentially. Of course, if there is a mix of reads and writes, especially writes with SYNC durability, then the writes cause random IO as well and SSDs help even more.

    Deferred-write has the largest benefit when insertion is not sequential by key, but rather random, and when updates and/or deletions are done as well as insertions. With sequential inserts, deferred-write helps only because there is no initial, empty version of a BIN logged.

    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
  • 5. Re: Optimizing huge bulk loads
    vinothchandar Newbie
    Currently Being Moderated
    Of course, if there is a mix of reads and writes, especially writes with SYNC durability, then the writes cause random IO as well and SSDs help even more.
    That Random IO would be reads right.
  • 6. Re: Optimizing huge bulk loads
    greybird Expert
    Currently Being Moderated
    That Random IO would be reads right.
    I think you're asking whether all the random IO is reads. No, because when the disk head is moved for a random read IO, then it has to be moved back to the end of the log to write. Each of these is considered a random IO, and they show up in the JE stats that way.

    --mark                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points