Avoid WebSocket Reconnect Storms in Multi-Region Systems

Summary

A single metric—"settle time"—is often the difference between a graceful region evacuation and a destructive reconnection storm. This article describes a concise, operational playbook you can apply today to avoid thousands of clients reconnecting at once. The primary goal: evacuate a region while keeping new connection arrival rates within receiver headroom.

The problem in one line

When a region is drained (planned maintenance, failover, or an outage), all sockets in that region can disconnect nearly simultaneously. If clients reconnect with deterministic delays, you get a synchronized spike that overwhelms TLS handshakes, auth, and accept queues. This is a WebSocket reconnection storm. The solution is a tactical playbook—drain, jitter, pinning/DNS, sizing/admission, and telemetry—that avoids a full re-architecture.

The playbook (drain, jitter, pinning, sizing, telemetry)

Drain

• Signal intent. When taking a gateway or region out of rotation, mark active connections as "draining" and send a drain code (e.g., 1001 / custom code + reason). Do this with a short server-managed TTL.
• For stateful clients, let them migrate gracefully. For stateless or headless clients, return a RECONNECT hint that includes a server-suggested delay window. This lets the backend control arrival rate instead of clients arbitrarily retrying.

Why this matters: a graceful drain turns a simultaneous disconnect into staggered reconnections under server guidance.

Jitter (full jitter is non-negotiable)

Use full jitter for client reconnection: delay = random(0, min(cap, base * 2^n)). This decorrelates clients and flattens arrival peaks. Important implementational detail: only reset the attempt counter on onopen (successful connection), not on onclose.

Client-side full-jitter example (JavaScript):

// full jitter
const BASE = 500 // ms
const CAP = 30000 // ms
let attempts = 0

function nextDelay(attempts) {
  const ceiling = Math.min(CAP, BASE * 2 ** attempts)
  return Math.floor(Math.random() * ceiling)
}

function onopen() {
  attempts = 0 // reset only on success
}

function onclose() {
  const delay = nextDelay(attempts)
  attempts++
  setTimeout(connect, delay)
}

Notes: full jitter minimizes peak concurrency. Equal jitter is an alternative if you require a minimum wait, but full jitter is the default for reconnection storm mitigation.

Pinning & DNS

• Pin a small fraction of critical clients to a standby region (short-lived sticky routing) so those flows can be preserved during an evacuation.
• Use short DNS TTLs and perform step-wise cutovers: first update DNS, then return RECONNECT windows rather than hard disconnects. Staging DNS over a few TTL steps reduces new arrivals into the draining region while letting existing sockets exit cleanly.

Example: change DNS TTL to 30–60s well before planned evacuations, then apply step changes during the actual cutover.

Sizing & Admission

Do the headroom math and expose it: available_capacity = max_conn - safety_margin. Use this to set caps and admission thresholds.

• If you expect N evacuated clients and you use a jitter window W seconds, your receiving region needs headroom for approximately N / W new connects/sec on top of steady-state traffic.
• For stateful or IoT-heavy fleets, put an admission gate (Redis or a small rate-limiter) in front of the gateway. That gate can reject or queue excess connect attempts during recovery windows (e.g., limit to 10k/s).

Pseudocode for a Redis admission gate (conceptual):

# simplistic rate limiter per-second
key = f"connects:{region}:{second}" 
if redis.incr(key) > allowed_per_second:
    return 429  # or RECONNECT window hint
else:
    accept_connection()

This preserves the receiving region from being swamped and gives operators control during a cutover.

Telemetry (the secret sauce)

Define settle time: the interval from the first forced reconnect to when the new connection rate returns to steady-state. Measure and alert on it.

• If settle time is higher than expected, throttle earlier or widen your jitter window.
• Instrument by region: new_connections/sec, connection_accept_failures, TLS handshake CPU, and headroom exposure. Plot settle time in runbooks and use it as the success criterion for drills.

Concrete example (evac drill)

During a us-east evacuation drill we predicted a surge that would exceed our gateway headroom. The steps we took:

1. Sent drain codes and a server RECONNECT window of 30–90s.
2. Pushed a DNS TTL change staged over 90s to stop new clients from landing in the draining region.
3. Enabled Redis admission control limiting new connects to 10k/s.
4. Ensured clients used full-jitter windows so arrivals flattened.

Result: peak connections stayed under capacity and settle time dropped from 7 minutes to ~45 seconds.

This is the practical payoff: small, predictable changes that keep the system within operational bounds.

Rules checklist you can copy

• Server:

  • Send RECONNECT hints + suggested window.
  • Enforce caps and expose headroom metrics.
  • Hot-reload gateway config rather than restarting gateways when possible.

• Client:

  • Full jitter: random(0, min(cap, base*2^n)).
  • Reset attempts only on successful onopen.
  • Respect server-suggested windows and drain codes.

• Infra:

  • Stage DNS cutovers; use short TTLs for failover windows.
  • Add Redis (or similar) admission control for stateful systems.
  • Size receiving regions by expected peak new-connections/sec (not just total sockets).

Operational advice and drills

• Rehearse evacuations during low-traffic windows and measure settle time. If settle time > 60s, either widen jitter windows or add headroom.
• Alert on settle time and connection-rate anomalies. Use an SLO for maximum acceptable settle time during planned and unplanned evacuations.
• Prefer controlled drains and RECONNECT hints over hard kills. Give the fleet a server-directed window instead of leaving timing to client defaults.

Conclusion

A WebSocket reconnection storm is avoidable with a short, operational playbook. You don't need to re-architect your system—apply drain signaling, full jitter on clients, staged DNS, admission gates, and measure settle time. Discipline, observability, and rehearsed drills turn an outage risk into a predictable operational task.

If you want a copyable checklist or a runnable drain script for your gateway, tell me your stack (NGINX/ALB/Envoy, Node/Go/Kotlin backend) and I’ll sketch a tailored example.