I've been building Ariadne, an Express-inspired HTTP framework written from scratch in modern C++ using raw POSIX sockets. Everything was going smoothly, routing, middleware, and request parsing all worked beautifully.
Then I load tested it with k6. That's when the weirdness started.
As soon as traffic ramped up, my terminal flooded with errors saying:
"unsolicited response received on idle HTTP channel"
Connection resets and aborted requests filled my terminal up. My first instinct was that request parsing was failing under load — dropping bytes, maybe confusing response boundaries somewhere. I pored over my recv loops and buffer management. Nothing looked wrong. The server wasn't crashing. It was successfully handling some requests and failing spectacularly on others, only under high concurrency.
Turns out, the bug had nothing to do with how I was parsing data, and everything to do with how I was managing the TCP connection lifecycle.
The Reveal
My server was lying to its clients. It claimed to speak HTTP/1.1, but it behaved like HTTP/1.0.
Under HTTP/1.1, connections are persistent by default. Unless either side explicitly sends a Connection: close header, both client and server assume the TCP connection stays open and gets reused for future requests.
Here's what my worker thread looked like at the time:
pool.enqueue([this, clientFd] {
// 1. Read and parse the request
auto req = parseRequest(clientFd);
if (req) {
Response res(clientFd);
// 2. Route the request and send the response
executeMiddlewares(0, *req, res);
}
// 3. Immediately close the socket!
close(clientFd);
});
See it? Accept a connection, parse one request, send one response, and close the socket.
Since I never sent Connection: close, k6 correctly assumed — because I'd told it I was speaking HTTP/1.1 — that the connection was still alive, and returned it to its pool. Milliseconds later, k6 grabbed that "idle" socket to send another request. On my end, that socket was already dead.
Hence the error: k6 was getting EOFs and resets on what it believed was a healthy, idle channel.
The Fix
The fix required a shift in how worker threads thought about their job. A worker thread couldn't own a request anymore — it needed to own a connection.
That meant wrapping request parsing and response generation in a loop, keeping the socket alive until the client explicitly asked to close it or an error occurred:
pool.enqueue([this, clientFd] {
while (true) {
try {
// Iteratively parse requests from the same connection
auto req = parseRequest(clientFd);
if (!req) break;
Response res(clientFd);
bool keepAlive = true;
// Check if the client wants to close the connection
auto it = req->headers.find("Connection");
if (it != req->headers.end() && it->second == "close") {
keepAlive = false;
}
// Set the appropriate response header
if (keepAlive)
res.set("Connection", "keep-alive");
else
res.set("Connection", "close");
executeMiddlewares(0, *req, res);
// Break the loop if we're done with this connection
if (!keepAlive) break;
} catch (const std::exception &e) {
Response res(clientFd);
res.status(400).send("Bad Request");
break;
}
}
// Finally close the socket when the loop ends
close(clientFd);
});
This later on introduced a new bug that I also found out under load tests. That is what the next post will be about.
The Takeaway
Once this landed, the protocol violations disappeared completely and as a side effect, my benchmarks against Node and Go finally became fair comparisons, since all three were now speaking the same protocol correctly.
It's easy to take keep-alive for granted when you're working inside a high-level framework. Write the socket code yourself, though, and you realize HTTP/1.1 is really a continuous conversation over one wire. You can't hang up after your first sentence and expect the other side to still be listening when you start talking again.
Have you ever shipped something, load tested it, and discovered you'd been quietly violating a protocol the whole time? Let me know in the comments. I'd love to hear you guys out.
Feel free to visit my portfolio to checkout my other projects! I would love some feedback on the portfolio itself as well.
I'll be back with more next week, till then stay consistent!
Top comments (0)