Why Your WebSocket Users Go "Ghost Online" — A Real Production War Story

First, a true story from our production environment.

Users reported that messages wouldn't send — the spinner just kept spinning. But the UI said "Connected", with a nice green dot. The server logs were even weirder: those users' connections were still alive on the server. The server thought they were online, and was faithfully pushing messages into dead sockets.

Restarting the app fixed it instantly. A few hours later, it came back.

Zombie connections — also known as half-open connections. Below is everything we learned, plus the four-part reliability kit we ended up shipping. Don't ask what version one looked like. Version one was the absence of everything below.

Why TCP Lies to You

WebSocket sits on top of TCP, and TCP has a property many people only learn about in production:

If the peer drops off the network abruptly — cable unplugged, laptop asleep, Wi-Fi to 4G — your socket fires no events at all. No close, no error, nothing.

In TCP's world, you only learn a connection is dead when you actively send data and the send fails. If neither side speaks, a "connection" can stay half-dead for hours: the routers in the middle dropped the session long ago, while both endpoints still believe the other is there.

So here's the iron rule: never rely on onclose to detect a dead connection. It will not come.

Part 1: Bidirectional Heartbeats

The fix is unglamorous: make both sides talk periodically, and declare death on silence.

// Client: ping every 30s; if no pong within 15s, kill the socket and reconnect
const HEARTBEAT_INTERVAL = 30_000;
const PONG_TIMEOUT = 15_000;

let pongTimer: number | undefined;

function startHeartbeat(ws: WebSocket) {
  setInterval(() => {
    if (ws.readyState !== WebSocket.OPEN) return;
    ws.send(JSON.stringify({ type: 'ping' }));

    clearTimeout(pongTimer);
    pongTimer = window.setTimeout(() => {
      ws.close();
      reconnect();
    }, PONG_TIMEOUT);
  }, HEARTBEAT_INTERVAL);
}

// Server: reply to pings, track last-seen time
ws.on('message', (raw) => {
  const msg = JSON.parse(raw.toString());
  if (msg.type === 'ping') {
    ws.send(JSON.stringify({ type: 'pong' }));
    markAlive(ws);
  }
});

// Reaper: clean up connections silent for 90+ seconds
setInterval(() => {
  for (const ws of clients) {
    if (Date.now() - lastActive(ws) > 90_000) {
      ws.terminate();
    }
  }
}, 30_000);

Two traps here, both of which we stepped on ourselves.

First: heartbeats must be bidirectional. Our first version only had client-side pings, so the server kept pushing into dead sockets — the client zombie problem was solved, the server-side zombie queue was not. The same logic applies in reverse.

Second: use terminate(), not close(), for server-side reaping. close() runs a graceful handshake and waits for the peer to respond. The peer is already dead. Stop waiting. terminate() destroys the socket immediately.

Part 2: Exponential Backoff Reconnect

Once you detect death, you reconnect. The laziest implementation is setTimeout(reconnect, 1000), which buys you two problems: when the server restarts, every client slams it at the same second (the reconnect storm), and during a long outage clients keep hammering retries every second for nothing.

The standard fix is exponential backoff with jitter:

let attempt = 0;

function reconnect() {
  const base = Math.min(1000 * 2 ** attempt, 30_000); // 1s → 2s → 4s … capped at 30s
  const delay = base * (0.7 + Math.random() * 0.6);   // ±30% jitter
  attempt += 1;
  setTimeout(connect, delay);
}

function connect() {
  const ws = new WebSocket(url);
  ws.onopen = () => {
    attempt = 0;
    resubscribe(); // don't skip this — next section
  };
}

The jitter line is what everyone skips. Don't. It exists precisely to prevent "all clients reconnecting in perfect unison", and it costs one line.

Part 3: State Recovery After Reconnect

The most commonly missed piece: a successful reconnect does not mean everything is fine.

While you were offline, other people sent messages, task states changed, and your subscriptions evaporated. Do nothing after reconnecting, and your users see a mysterious "time hole" in the conversation.

Our approach: reconcile after every reconnect.

ws.onopen = () => {
  attempt = 0;
  // Send the last message id we have; server replies with the delta.
  // If the gap is too large, fall back to a full refresh.
  ws.send(JSON.stringify({
    type: 'sync',
    lastMsgId: store.lastMsgId,
  }));
};

Delta catch-up with a full-refresh fallback — I haven't found a simpler scheme that actually holds up.

Part 4: Acks and Stale-Event Guards

Two smaller things, worth their own section.

Send acks. "The client sent it" and "the server received it" are different facts. For important collaborative operations, have the server acknowledge with a message id; only then does the client mark it as delivered, and unacked messages go into a retry queue. That gray-to-blue checkmark in your UI is local state — don't let it lie to you.

Stale events. Here's a nasty bug hiding in reconnect logic: after an old socket dies, it can still fire onmessage one last time — leftovers from the kernel buffer. Handle it normally, and a message from "the previous world" pollutes your current state. This one took us a full night to track down. The fix is a generation counter, three lines:

let generation = 0;

function connect() {
  const myGen = ++generation;
  const ws = new WebSocket(url);
  ws.onmessage = (e) => {
    if (myGen !== generation) return; // event from a stale socket — drop it
    handleMessage(e);
  };
}

This kit holds up well at a few thousand concurrent connections. Past tens of thousands, the heartbeat scan needs its own optimization pass — but that's a story for another post.

The Checklist

Audit your project against this:

• [ ] Client pings and waits for pong; on timeout it declares death itself, never waiting for onclose
• [ ] Server tracks last-active time and reaps zombies with terminate()
• [ ] Reconnects use exponential backoff + jitter + a cap
• [ ] After reconnecting: delta sync, or a full reconciliation fallback
• [ ] Important messages have acks; unacked messages go to a retry queue
• [ ] Events from stale sockets are dropped via a generation guard

Next post: reliable message delivery — how a server guarantees a message is delivered exactly once, and how out-of-order "paranormal events" actually happen. Follow me if that sounds like your kind of problem.