A couple of weeks ago I asked the question “Why non-blocking?”. And I didn’t reach a definitive answer, although it seemed that writing non-blocking code is not the better option – it’s not supposed to be faster or have higher throughput, even though conventional wisdom says it should.
So, leaving behind the theoretical questions, I decided to do a benchmark. The code is quite simple – it reads a 46kb file into memory and then writes it to the response. That’s the simplest scenario that’s still close the the regular usecase of a web application – reading stuff from the database, performing some logic on it, and then writing a view to the client (it’s disk I/O vs network I/O in case the database is on another server, but let’s disregard that for now)
There are 5 distinct scenarios: Servlet using BIO connector, Servlet using NIO connector, Node.js, Node.js using sync file reading and Spray (a scala non-blocking web framework). Gatling was used to perform the tests, and was run on a t2.small AWS instance; the application code was run on a separate m3.large instance.
The code used in the benchmark as well as the full results are available on GitHub. (Note: please let me know if you spot something really wrong with the benchmark that skews the results)
What do the results tell us? That it doesn’t matter whether it’s blocking or non-blocking. Differences in response time and requests/sec (as well as the other factors) are negligible.
Spray appears to be slightly better when the load is not so high, whereas BIO happens to have more errors on a really high load (but being fastest at the same time), Node.js is surprisingly fast for a javascript runtime (kudos to Google for V8).
The differences in the different runs are way more likely to be due to the host VM current CPU and disk utilization or the network latency, rather than the programming model or the framework used.
After reaching this conclusion, the fact that spray is seemingly faster bugged me (especially given that I executed the spray tests half an hour after the rest), so I wanted to rerun the tests this morning. And my assumption about the role of infrastructure factors could not have been proven more right. I ran the 60 thousand requests test and the mean time was 3 seconds (for both spray and servlet), with a couple of hundred failures and only 650 requests/sec. This aligned with my observation that AWS works a lot faster when I start and delete cloud formation stacks early in the morning (GMT+2, when Europe is still sleeping and the US is already in bed).
The benchmark is still valid, as I executed it within 1 hour on a Sunday afternoon. But the whole experiment convinced me even more of what I concluded in my previous post – that non-blocking doesn’t have visible benefits and one should not force himself to use the possibly unfriendly callback programming model for the sake of imaginary performance gains. Niche cases aside, for the general scenario you should pick the framework, language and programming model that people in the team are most comfortable with.
| Reference: | A Non-Blocking Benchmark from our JCG partner Bozhidar Bozhanov at the Bozho’s tech blog blog. |
