A few weeks ago I published a number for Auvis, the proxy I've been building. 6,541 requests per second, measured with hey,full workload running. I was proud of it. I wrote a whole post about the fix that got us there.
Then I read a post by Sergey Yakushevich, Making Go 104x faster than Ruby because I configured things incorrectly. Benchmarked Go against Ruby, gotten a wild 104x number, and instead of publishing it, spent the next several days finding six separate mistakes in his own test setup. The real number, once everything was fixed, was 1.6x. Not 104x.
One of his six mistakes stuck with me specifically. He'd been running his load-testing tool on the same machine as the server it was testing, and the tool itself turned out to have a ceiling, a maximum speed it could generate requests at, regardless of how fast the server actually was. If your server's number gets uncomfortably close to that ceiling, you're not measuring your server anymore. You're measuring your tool.
I went to check if I'd made the same mistake.
I hadn't checked before, so I ran a baseline, pointed hey at a server that does nothing, just replies instantly with no real work. On my own machine, that ceiling came out around 92,000 requests per second. Then I compared it against Auvis doing real work.
81% of the ceiling.
Not the clean number I'd assumed. hey and Auvis were fighting over the same CPU cores the entire time, and my published 6,541 number, while not fabricated, wasn't a clean measurement of Auvis on its own. It was Auvis measured while sharing hardware with the thing measuring it.
I tried isolating cores with taskset, pinning hey to four cores and Auvis to four separate cores on an 8-vCPU box, thinking physical separation on the same machine would fix it. It didn't. Both processes still shared the same kernel network stack, and hey's loopback connection handling capped out around 100,000 requests per second no matter how the cores were split. Single-machine benchmarking, it turns out, has a hard ceiling that core pinning alone can't get around.
The actual fix was two separate machines. I spun up a second EC2 instance in the same AWS VPC, one running Auvis with full policy enforcement, the other running nothing but hey, talking to Auvis over a real private network hop instead of loopback. Complete physical CPU isolation.
Real number: 19,682 requests per second. Auvis's actual processing overhead, on top of real network latency, works out to +0.70 milliseconds per request.
That's roughly three times higher than the number I'd originally published.
If you're publishing throughput numbers for anything, and your load generator lives on the same machine as your server, go check your own tool's ceiling first. It's a five-minute test, and it might change your headline number more than you'd expect.
Top comments (0)