WebSocket architecture looks deceptively simple until the moment it doesn't. You spin up a few servers, open persistent connections, ship some messages. Everything holds together fine at a few hundred concurrent clients. Then you hit a real stress test, not a load test you designed in a controlled environment, but the kind that happens in production at 2am when a carrier outage drops ten thousand connections simultaneously and they all try to reconnect within the same thirty-second window.
This is the thundering herd problem, and it punishes stateful systems in ways that stateless REST APIs simply never have to worry about.
Why Stateful Connections Break Horizontal Scaling Assumptions
Most engineers who have grown up in Kubernetes-native environments carry a mental model where scaling is straightforward: traffic spikes, you add pods, a load balancer distributes the work, everyone goes home happy. That model works beautifully for stateless request-response cycles.
WebSockets are not stateless. A connection carries context. The server holding that connection may have buffered messages in flight, session state, or protocol negotiation history that lives nowhere else. When you scale horizontally and a client reconnects, it may land on a completely different node with no memory of the prior session. At best, you serve a degraded experience. At worst, you corrupt state.
EV charging networks make this failure mode unusually concrete. A charging station running OCPP over WebSockets is not just streaming telemetry. It is executing a stateful protocol where the server tracks transaction IDs, authorization states, and meter values tied to a specific session. When a network carrier blip drops three thousand stations and they all hammer your servers at reconnection time, you are not dealing with a load problem. You are dealing with a stateful session reconstruction problem under hostile concurrency conditions.
The Reconnection Backoff Problem Is an Architectural Decision, Not a Client Bug
The instinctive response to thundering herd scenarios is to tell clients to implement exponential backoff with jitter. That advice is correct but incomplete. Client-side backoff only works if you can actually enforce it, and in many embedded or third-party device ecosystems, you have limited control over client behavior. A firmware version deployed across thousands of charging stations may have a fixed two-second retry interval baked in, and you are not pushing an OTA update during an incident.
Robust infrastructure needs server-side defenses that do not assume cooperative clients. This means:
# Simplified example of server-side connection rate limiting per client group
from collections import defaultdict
import time
class ConnectionRateLimiter:
def __init__(self, max_connections_per_window, window_seconds):
self.max = max_connections_per_window
self.window = window_seconds
self.buckets = defaultdict(list)
def allow(self, client_group_id):
now = time.monotonic()
timestamps = self.buckets[client_group_id]
# Evict timestamps outside the window
self.buckets[client_group_id] = [t for t in timestamps if now - t < self.window]
if len(self.buckets[client_group_id]) >= self.max:
return False
self.buckets[client_group_id].append(now)
return True
This kind of gate at the connection layer buys your backend time to reconstruct state without collapsing under the reconnection surge. It is not a complete solution, but it is the difference between a degraded recovery and a full outage cascade.
Vertical Scaling Is Not the Enemy Here
There is a reflexive instinct in modern infrastructure culture to treat vertical scaling as a legacy crutch. That instinct is wrong at the WebSocket layer. A single beefy node can hold vastly more open connections than a fleet of small pods can coordinate across a message bus. At Netflix scale, the Pushy notification system went through exactly this evolution: what started as simple best-effort delivery had to be rebuilt as a load-aware proxy with explicit session affinity, because connection counts at millions of concurrent devices expose failure modes that simply do not appear at thousands.
The mature answer is both. You vertically scale your WebSocket gateway nodes to maximize per-node connection capacity, and you horizontally scale that gateway tier behind a connection-aware load balancer that understands session affinity. Then you route downstream processing to stateless workers that can scale independently. The WebSocket tier and the application logic tier have different scaling profiles and should be treated as separate concerns in your architecture.
Treating this as a binary choice is how teams end up either with a single overloaded server or a horizontally scaled cluster that drops state on every reconnection.
What Actually Matters in Production
The systems that hold up under real-world pressure share a few characteristics that are not obvious from tutorials or toy examples.
They treat the reconnection surge as a first-class failure mode, not an edge case. They implement connection draining during deploys so that rolling restarts do not manufacture their own thundering herds. They separate the concern of connection management from the concern of message processing. And they build observability into the connection lifecycle itself: tracking not just message throughput but connection churn rate, reconnection latency distribution, and per-session state reconstruction time.
The hardest part of WebSocket infrastructure at scale is not opening connections. It is knowing what to do when thousands of them close at the same time and every client wants back in immediately. Designing for that scenario from the start is the difference between infrastructure that survives incidents and infrastructure that becomes one.
Top comments (0)