This discussion is archived
10 Replies Latest reply: Jan 17, 2012 8:31 AM by Charles Lamb RSS

Oracle NoSQL Database YCSB Results

Charles Lamb Pro
Currently Being Moderated
I have posted some results of a YCSB benchmark that I ran on my blog. I thought it might be of general interest to this forum:

http://blogs.oracle.com/charlesLamb/entry/oracle_nosql_database_performance_tests

Charles Lamb
  • 1. Re: Oracle NoSQL Database YCSB Results
    user12003335 Newbie
    Currently Being Moderated
    Two questions:

    1) Can you share the consistency and durability settings used in the benchmark?
    2) The "getting started" guide mentions that large number of replication groups will improve performance (which makes sense, it will allow more concurrency). However, even your largest test has just 4 groups. Can you share the reason for this?

    -- Gwen
  • 2. Re: Oracle NoSQL Database YCSB Results
    Charles Lamb Pro
    Currently Being Moderated
    user12003335 wrote:
    Two questions:

    1) Can you share the consistency and durability settings used in the benchmark?
    2) The "getting started" guide mentions that large number of replication groups will improve performance (which makes sense, it will allow more concurrency). However, even your largest test has just 4 groups. Can you share the reason for this?

    -- Gwen
    Hi Gwen,

    Sure, no problem.

    For 1): The Durability was the NoSQL Database recommended (and default) value of no_sync, simple_majority, no_sync. The Consistency that we used for the 50/50 read/update test was Consistency.NONE.

    For 2): Cisco was able to let us use 12 C210 nodes and 12 C200 nodes. We wanted to make sure that the Rep Nodes were symmetric so we used the smaller nodes (C200s) for the client load generators and the more capable IO subsystems on the C210s for the Rep Nodes. To be honest, it's hard to find a lab that has lots of homogeneous boxes with capable IO subsystems. It's generally a mix of some of this and some of that. You might think that Amazon EC2 is the answer to our needs, but VM performance is not predictable -- we like to run on raw hardware. That said, we have run tests on 64 Rep Groups (on machines with similar CPUs, but much less-capable IO subsystems) and demonstrated NoSQL Database's scalability. The Cisco numbers that I published are the highest performance numbers we have seen for that number of nodes/rep groups.

    We are working with a couple of partners on additional large-scale configuration testing. If we get more results, I'll definitely post results on my blog.

    Charles Lamb
  • 3. Re: Oracle NoSQL Database YCSB Results
    896774 Newbie
    Currently Being Moderated
    Hi Charles,

    Thanks for those results. I would like to know the following:
    1) I assumed the number of client is 1 (not 3) for the first line the those results and 90 for the threads (not 30).
    2) What is your best benchmark today to show how Oracle NoSQL is scalable? Can you tell us the result of the best benchmark as you did for CISCO ?
    3) Can you explain what "95% Latency" and "99% Latency mean" ? The result are too different from avg latency.
    Thanks.
  • 4. Re: Oracle NoSQL Database YCSB Results
    Charles Lamb Pro
    Currently Being Moderated
    893771 wrote:
    1) I assumed the number of client is 1 (not 3) for the first line the those results and 90 for the threads (not 30).
    Several people have called that to my attention, but I believe the number is correct as it stands. The accompanying text implies that it should be 90, but the test was run with 30.
    2) What is your best benchmark today to show how Oracle NoSQL is scalable? Can you tell us the result of the best benchmark as you did for CISCO ?
    To be clear, the results were not done "for Cisco", they were done using Cisco's lab. Cisco was generous with their hardware, but as I mentioned earlier, it is difficult to put hundreds of homogeneous nodes together in one place at one time. A lab with this much equipment has many demands placed on it. Therefore, we were given access to 12 C210 and 12 C200 nodes for our tests. In the future we may receive access to larger configurations, but this access will have to be scheduled among many users. The bottom line is that the results I published are the largest tests we ran in the Cisco labs because that is the largest configuration we have been given access to (to date).

    As I mentioned in a previous post, we have run tests using 72 VMs (20 Rep Groups and 12 YCSB clients), and 192 nodes (64 Rep Groups). These tests validated that our architecture scales.

    Tests at this scale take a substantial amount of effort to provision, run, and (especially) analyze. We spent a great deal of time on the Cisco tests as well as the "72 VM" and "192 node" tests. We performed extensive analysis on the results before our access to those labs expired.
    3) Can you explain what "95% Latency" and "99% Latency mean" ? The result are too different from avg latency.
    95% of the results showed latency less than that number. Ditto 99%.

    Charles Lamb
  • 5. Re: Oracle NoSQL Database YCSB Results
    906230 Newbie
    Currently Being Moderated
    Hi Charles,
    The accompanying text implies that it should be 90, but the test was run with 30.
    What about the number of clients? If it is really 3. That mean for "50/50 Read/Update" you had only 10 threads on each client. And for "Insert Results", only 30 threads on each client. Can you confirm that?
    We have run tests using 192 nodes (64 Rep Groups)
    Are those results available some where to see how latency varies with the number of Rep Groups or the volume of data?

    Thanks.
  • 6. Re: Oracle NoSQL Database YCSB Results
    Charles Lamb Pro
    Currently Being Moderated
    user962305 wrote:
    Hi Charles,
    The accompanying text implies that it should be 90, but the test was run with 30.
    What about the number of clients? If it is really 3. That mean for "50/50 Read/Update" you had only 10 threads on each client. And for "Insert Results", only 30 threads on each client. Can you confirm that?
    Correct.

    >
    We have run tests using 192 nodes (64 Rep Groups)
    Are those results available some where to see how latency varies with the number of Rep Groups or the volume of data?
    No.
  • 7. Re: Oracle NoSQL Database YCSB Results
    906230 Newbie
    Currently Being Moderated
    Excuse-me Charles,

    I don't understand how Throughput and Latency are calculated from KVS size, Clients, Total Client Threads and Time.
    Thanks for your help.
  • 8. Re: Oracle NoSQL Database YCSB Results
    Charles Lamb Pro
    Currently Being Moderated
    user962305 wrote:
    Excuse-me Charles,

    I don't understand how Throughput and Latency are calculated from KVS size, Clients, Total Client Threads and Time.
    Calculated? I don't understand what you mean. Throughput and Latency are measured results of the benchmark.

    Charles Lamb
  • 9. Re: Oracle NoSQL Database YCSB Results
    906230 Newbie
    Currently Being Moderated
    If we have time. Why don't we have Throughput close to number of inserts / time ? and avg Latency close to time / number of inserts ?
  • 10. Re: Oracle NoSQL Database YCSB Results
    Charles Lamb Pro
    Currently Being Moderated
    user962305 wrote:
    If we have time. Why don't we have Throughput close to number of inserts / time ? and avg Latency close to time / number of inserts ?
    400m / 26,498 ~= 15k
    1.6b / 94,441 ~= 17k

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points