When you add 2 second delay every 10 requests you make the comparison totally meaningless. You are mesuring delays, not the code.
Reading the file in every loop mesures reading from disk, not actual program performance.
Also, ab is not a good tool for measuring. Usually you are measuring the performace of ab, not the system that can be 20 times faster than what ab can measure. Use github.com/wg/wrk instead.
When I remove the delay then go program crashes when testing with wrk: Error reading:EOF.
It's remarkable too that none of the other commenters noted this. As the adage goes: just because you read it on the internet doesn't make it true! Good catch.
I think it's a lot better! I suspect that some of the differences now are due to technical trivia of superficially irrelevant details you happened to choose when implementing these programs, but that's the real-world for you. The current data highlights much more clearly just how many req/sec any of this options can handle - because I think that's the real takeaway here; the web-frameworks themselves are unlikely to be a significant bottleneck in any real-world usecase; and if the language and/or framework matters for heavier, real workloads - well, that's the kind of thing you can't microbenchmark well; you need a real use case.
What the current data also highlights more clearly is just how finicky perf at this level is; e.g. the way the program using the go http stack apparently is much more efficient than the program you labelled TCP; or how wrk and drill results are quite different. And that's important to understand; microbenchmarks are notoriously flaky and sensitive to all kinds of details you don't actually care about. Taking a microbenchmark to mean that task X takes time Y is usually the wrong way to think about it - it takes time Y only in the hyper-specific circumstances of the test - but generalizing a specific result is quite error-prone.
I did add a disclaimer that this is a simple concurrency benchmark. I don't agree that its meaningless as I'm comparing exact same impl across languages to see if the language/platform makes any difference, sleep was added to introduce a concurrency bottleneck. This is not a HTTP performance comparison, its a concurrency comparison and for that I think AB is as good as any other tool. I'll try wrk and post the results.
System sleep is not exactly 2 seconds, it can vary wildly.
Basically you are comparing nonsense. This is like comparing people who jump at top of 2000m mountain and and you tell that someone jumped 2000.3 meters and someone else jumped 2000.26 meters and you tell that "results are almost same". And acctually in every jump the mountain height varies a lot.
If you want ot test concurrency then take the web code away. Give every program same amount of loops to run. And let the tests run 5 seconds or preferably more, anything less is not statistically valid.
All you demonstrated is that thread sleep works in all languages, and that overhead is significantly smaller than 200ms. You did not really benchmark the various language / server combos.
If it makes you all happy I'll update the post with numbers from the code without sleep. Oh and there are cases where a request takes more than 2 seconds, I have performance tuned many such systems when I was working for enterprise companies. Also do suggest better approach to simulate a thread blocking request
And I reiterate again if it wasn't clear from the title, intro or all the previous posts. I was trying to benchmark performance of concurrency and not web server performance. They are related but not the same
I was about to comment on the topic, thank you for pointing this out and not blindly trusting the internet.
The massive hint is the extremely similar performance between all languages. I'm sure the intent of this article was to help the community, but I hope the author will understand their mistake and update the results accordingly.
When you add 2 second delay every 10 requests you make the comparison totally meaningless. You are mesuring delays, not the code.
Reading the file in every loop mesures reading from disk, not actual program performance.
Also, ab is not a good tool for measuring. Usually you are measuring the performace of ab, not the system that can be 20 times faster than what ab can measure. Use github.com/wg/wrk instead.
When I remove the delay then go program crashes when testing with wrk: Error reading:EOF.
It's remarkable too that none of the other commenters noted this. As the adage goes: just because you read it on the internet doesn't make it true! Good catch.
I have updated the benchmarks with more data. WDYT now?
I think it's a lot better! I suspect that some of the differences now are due to technical trivia of superficially irrelevant details you happened to choose when implementing these programs, but that's the real-world for you. The current data highlights much more clearly just how many req/sec any of this options can handle - because I think that's the real takeaway here; the web-frameworks themselves are unlikely to be a significant bottleneck in any real-world usecase; and if the language and/or framework matters for heavier, real workloads - well, that's the kind of thing you can't microbenchmark well; you need a real use case.
What the current data also highlights more clearly is just how finicky perf at this level is; e.g. the way the program using the go http stack apparently is much more efficient than the program you labelled TCP; or how wrk and drill results are quite different. And that's important to understand; microbenchmarks are notoriously flaky and sensitive to all kinds of details you don't actually care about. Taking a microbenchmark to mean that task X takes time Y is usually the wrong way to think about it - it takes time Y only in the hyper-specific circumstances of the test - but generalizing a specific result is quite error-prone.
I think this is an excellent takeaway
I did add a disclaimer that this is a simple concurrency benchmark. I don't agree that its meaningless as I'm comparing exact same impl across languages to see if the language/platform makes any difference, sleep was added to introduce a concurrency bottleneck. This is not a HTTP performance comparison, its a concurrency comparison and for that I think AB is as good as any other tool. I'll try wrk and post the results.
System sleep is not exactly 2 seconds, it can vary wildly.
Basically you are comparing nonsense. This is like comparing people who jump at top of 2000m mountain and and you tell that someone jumped 2000.3 meters and someone else jumped 2000.26 meters and you tell that "results are almost same". And acctually in every jump the mountain height varies a lot.
If you want ot test concurrency then take the web code away. Give every program same amount of loops to run. And let the tests run 5 seconds or preferably more, anything less is not statistically valid.
Instead of sleeping 2 sec or even 200 ms, you should sleep 3-8 ms to simulate access to fast SQL server. Then you would have meaningful request rate.
I have updated the benchmarks with more data. WDYT now?
All you demonstrated is that thread sleep works in all languages, and that overhead is significantly smaller than 200ms. You did not really benchmark the various language / server combos.
If it makes you all happy I'll update the post with numbers from the code without sleep. Oh and there are cases where a request takes more than 2 seconds, I have performance tuned many such systems when I was working for enterprise companies. Also do suggest better approach to simulate a thread blocking request
And I reiterate again if it wasn't clear from the title, intro or all the previous posts. I was trying to benchmark performance of concurrency and not web server performance. They are related but not the same
I have updated the benchmarks with more data. WDYT now?
Good. Could you update the code in repo too - they contain sleep().
Its a different branch now (nosleep)
I was about to comment on the topic, thank you for pointing this out and not blindly trusting the internet.
The massive hint is the extremely similar performance between all languages. I'm sure the intent of this article was to help the community, but I hope the author will understand their mistake and update the results accordingly.
I have updated the benchmarks with more data. WDYT now?