Following on my previous Simple web server implemented in Perl and Go
blog, which has been pointed out as unfair comparison. This time, thanks to Axel Wagner, who replaced the net/http.Server layer with direct translation of Perl code, the code is now reading and writing directly to a socket, just as what the Perl code is doing.
And here are the test results from my machine again (which I published here):
100 concurrency requests (the number of multiple requests to perform at a time):
500 concurrency requests:
Once again,
- This is the implementation in Perl, with Cache-Control Header added, to be fair
- And this is the implementation in Go
- As discussed previously, statistic summaries of Apache Bench are a bit confusing so I'm leaving them out this time. Also concluded was that the measured & reported response time are truly trustworthy.
From the charts, we can see that the response time for Go 500 concurrency requests is almost identical to previous result, while the 100 concurrency is actually worth than before.
In summary, I'm still seeing Perl performs much better than Go, with this direct translation. Why? Could starting the Go server as myself instead of root be the culprit, or something else?
UPDATE:
What I was expecting to see? Quoting from a comment from previous blog:
... That perl still ends up with significantly higher average response times is… honestly, a confirmation that it's not a very fast language.
BTW, I also used a different tool to benchmark (github.com/rakyll/hey). AIUI, it uses multiple cores more efficiently. With that tool, the perl implementation gets absolutely demolished, with an average response time ~40x higher than the Go implementation on my machine.
Having ab failed for me to produce the anticipated result, I installed the suggested hey, and test results indeed show that Go is better than Perl.
Further UPDATE:
I tested with hey's default settings.
For Perl:
hey -disable-keepalive http://192.168.0.10/
For Go:
hey -disable-keepalive http://192.168.0.11:8888/
(Note that both 192.168.0.10 and 192.168.0.11 are my local machine, as I have configured two IP address for it.)
And the conclusion that "the test results indeed show that Go is better than Perl" was draw upon the Response time histogram. However, when I thought more into it, it doesn't make sense to me any more.
The hey's default settings are:
Usage: hey [options...] <url>
Options:
-n Number of requests to run. Default is 200.
-c Number of requests to run concurrently. Total number of requests cannot
be smaller than the concurrency level. Default is 50.
As we can see in the previous result, the Perl server survives the 500 concurrent requests gracefully, with 95% percentile reading of only 7ms. However hey is telling a different story that, with only 200 concurrent requests, the Perl server's response time is mostly 0.105 seconds. However,
That is 15 times than ab's reading.
Hey's inability of getting accurate reading at ms level makes me doubt it's own accuracy, and I have to suspect that hey's 15-time response time reading is due to its own slack.
Moreover, if you take a closer look at hey's Latency distribution readings, we can see that Perl at 75% is still 0.0181secs, while Go, at 75% is already 0.0222secs. Let alone at 10%, Perl is 0.0045secs while Go is 0.0150secs (>3 times more). For this we can see that hey's reading is contradicting with themselves and Perl clearly performs much better than Go from hey's Latency distribution readings.
Furthermore, that 13 long 1.043sec response time from the Perl server is only observed in hey, all the other performance testing software, including the not-documented-here httperf, none of them have exhibited such behavior. So it is fair to conclude that the long 1.043sec response time is caused by hey itself, not by the Perl server.
So, up to now, for all above three reasons, I have to conclude that hey is not suitable for this test due to its own limitations, thus cannot stacks up against the most popular Apache Bench.
Now, it is getting more and more interesting -- who and what to trust. Yeah, that deserves another blog, which I will turn into commercial web performance testing software, which I happen to have access to. It's not as simple as one command, but needs lots of steps just to set it up. But, I'll do it, to see how it looks like from the commercial tool (by Microsoft) that has been used and depended on by millions of companies.
Stay tuned...
Top comments (1)
Really interesting results. I'm not fluent with either Go or Perl, but with Go being lower level code it's really surprising to me that it's not showing as well as I expect. It makes me think that either the Perl or Go implementation ends up doing something fundamentally different, but I don't know enough about either to warrant a good guess.