Fixing my server's HTTP/1.1 keep-alive violation introduced a much worse bug.
In my previous post, I wrote about how Ariadne, my custom C++ HTTP framework, was claiming to speak HTTP/1.1 but acting like HTTP/1.0 by immediately closing sockets after serving a single request.
To fix the keep-alive protocol violation, I restructured the server so that each worker thread owned a connection until the client explicitly requested Connection: close or the socket timed out.
Adding a CPU-Bound Endpoint
With persistent connections working, I added a CPU-intensive endpoint (GET /api/compute recursively calculating fib(30)) to the benchmark workload to test how the server handled mixed I/O and CPU-bound traffic.
The results were dramatically worse than expected. Throughput collapsed to roughly 40 requests per second, and k6 reported widespread timeouts, even for the lightweight routes like / and /api/user.
The recursive computation wasn't the bottleneck. The root cause was how the thread pool was initialized.
Before: One Thread, One Request
Before the keep-alive fix, a worker thread processed exactly one request and immediately closed the socket, returning to the thread pool:
ThreadPool pool(std::thread::hardware_concurrency());
while (serverRunning) {
int clientFd = accept(serverFd, nullptr, nullptr);
if (clientFd == -1) {
if (errno == EINTR) {
continue;
}
perror("accept");
break;
}
pool.enqueue([this, clientFd] {
Request req = parseRequest(clientFd);
Response res(clientFd);
executeMiddlewares(0, req, res);
close(clientFd);
});
}
After: One Thread, One Connection
After implementing proper HTTP/1.1 persistent connections, each worker thread instead entered a continuous loop pinned to an entire TCP connection:
pool.enqueue([this, clientFd] {
while (true) {
try {
auto req = parseRequest(clientFd);
if (!req)
break;
Response res(clientFd);
bool keepAlive = true;
auto it = req->headers.find("Connection");
if (it != req->headers.end() && it->second == "close") {
keepAlive = false;
}
if (keepAlive)
res.set("Connection", "keep-alive");
else
res.set("Connection", "close");
executeMiddlewares(0, *req, res);
if (!keepAlive)
break;
} catch (const std::exception &e) {
Response res(clientFd);
res.status(400).send("Bad Request");
break;
}
}
close(clientFd);
});
The Root Cause
The issue was std::thread::hardware_concurrency(). For the 4-core benchmark container, this resulted in a fixed pool of 4 worker threads.
Once each worker owned an entire persistent connection, the pool size became the hard limit on simultaneous active client connections. The server could only process 4 connections at a time, regardless of how lightweight the requests were.
During the benchmark, k6 generated up to 200 concurrent virtual users (VUs). Four client connections occupied the worker threads, and the remaining 196 waited in the accept queue. No additional requests could begin processing until an existing connection closed.
The expensive fib(30) endpoint merely exposed this architectural limitation by keeping each connection busy for longer. The thread pool had been sized for CPU parallelism rather than expected connection concurrency.
The fix was straightforward:
ThreadPool pool(250);
At first, allocating 250 threads seemed expensive. At 8MB per thread stack, that implies 2GB of RAM. However, Linux reserves Virtual Memory for thread stacks, and Docker's memory limits only restrict Resident Set Size (Physical RAM). A thread only uses physical RAM for the stack space it actually touches. During the benchmark, total RAM usage by the container never crossed 15MB.
With the thread pool correctly sized to accommodate the 200 concurrent VUs, performance recovered completely. Throughput hit 756 req/s, and the p95 latency during the ramping load dropped to 3.77ms.
Where This Still Falls Short
A thread-per-connection model like this works, but it doesn't scale indefinitely. Every idle connection still holds a thread hostage even when it's doing nothing but waiting on the network. Get to a few thousand concurrent connections and you're paying real scheduling and context-switch overhead just to keep sockets open, long before any of them have actual work to do.
The real fix for that isn't a bigger thread pool, it's not blocking a thread per connection at all. An event-driven model using epoll lets a single thread monitor thousands of file descriptors and only wake up when a socket actually has data ready, instead of one thread sitting parked on each connection. That's the architecture Go's netpoller and Node's event loop are both built on, in different ways. I'll be implementing epoll-based I/O in Ariadne next, and writing that up when it's done.
I'll be back with more next week. Till then, stay consistent!
Top comments (2)
epoll is great idea but I reckon you'll find it difficult to get a cross platform equivalent working at scale. Windows wants you to go the IOCompletionPort route while Linux etal will be happy with epoll. However Linux will be even faster with an IOURing implementation.
The issue I faced was how to integrate an IOCompletionPort solution on Windows and an IOURing solution on Linux so they have a common interface and work the same. It's a long answer involving CoRoutines and a background IO Service. I'll write about it sometime. It ate most of my January and needs to pay me back :-)
Haha! I get you. Looking forward to reading that post.