Overview
Yahoo Cloud Service Benchmark is a reasonably widely used benchmarking tool for testing key value stores for a significant number of key e.g 100 million, and a modest number of clients i.e. served from one machine.
In this article I look at how a test of 100 million * 1 KB key/values performed using Chronicle Map on a single machinewith 128 GB memory, dual Intel E5-2650 v2 @ 2.60GHz, and six Samsung 840 EVO SSDs.
The 1 KB value consists of ten fields of 100 byte Strings. For a more optimal solution, primitive numbers would be a better choice. While the SSDs helped, the peak transfer rate was 700 MB/s which could be supported by two SATA SSD drives.
These benchmarks were performed using the latest version at the time of the report, Chronicle Map 2.0.5a-SNAPSHOT.
Micro-second world
Something which confounds me when reading benchmarks about key-value stores is that they start with the premise that performance is really important. IMHO, about 90% of the time, performance is not the most important feature, provided you have sufficient performance.
These benchmark reports then continue to report times in milli-seconds, not micro-seconds and throughputs in thetens of thousands instead of the hundreds of thousands or millions. If performance really was that important, they would have built their products around performance, instead of the useful features they do support, like multi-key transactionality, quorum updates and other features Chronicle Map doesn’t support, for performance reasons.
So how would a key-store built for performance look with YCSB?
Throughput measures
The “50/50” tests 50% random reads and 50% random writes, the “95/5” tests 95% reads to 5% writes. It is expected that writes will be more expensive, and a higher percentage of reads results in higher throughputs.
| Threads | 50/50 read/update | 95/5 read/update |
|---|---|---|
| 1 | 122 K/s | 245 K/s |
| 2 | 235 K/s | 414 K/s |
| 4 | 339 K/s | 750 K/s |
| 8 | 646 K/s | 1.295 M/s |
| 15 | 819 K/s | 1.452 M/s |
| 30 | 900 K/s | 1.641 M/s |
Latencies
The following latencies are in micro-seconds, not milli-seconds.
| Threads: 8 | 50/50 read | 95/5 read | 50/50 update | 95/5 update |
|---|---|---|---|---|
| average | 5 µs | 3.9 µs | 15.9 µs | 11.3 µs |
| 95th | 12 µs | 8 µs | 31 µs | 19 µs |
| 99th | 19 µs | 14 µs | 42 µs | 27 µs |
| worst | 67 ms | 70 ms | 67 ms | 70 ms |
Note: the benchmark is not designed to be GC free and creates some garbage. This is not particularly high and the benchmark itself uses only about 1/4 of CPU according to flight simulator, however it does impact the worst latencies.
Conclusion
Make sure the key-value store has the features you need, but if performance is critical, look for a solution designed for performance as this can be 100x faster than full featured products.
Other high performance examples
Room for improvement
Footnote
| Reference: | Chronicle Map and Yahoo Cloud Service Benchmark from our JCG partner Peter Lawrey at the Vanilla Java blog. |
