WebSocket Engineering — Part 2

WebSocket connection :- Heartbeats, Zombie Connections, and the Connections That Never Said Goodbye

In Part 1, I reached one uncomfortable conclusion:

WebSocket disconnecting is normal.

Reliable recovery is the feature.

But that created another question.

How do I even know that a WebSocket connection has died?

At first this sounded obvious.

Surely if the client disappears, the server gets a disconnect event.

Surely TCP knows.

Surely WebSocket knows.

Then I learned about a type of connection failure that does not announce itself.

No close frame.

No goodbye.

No clean error.

The other side simply disappears.

And the connection becomes a ghost.

---

The connection that never said goodbye

Imagine a user is connected over Wi-Fi.

Browser ───────────── Server
        WebSocket

Everything works.

Then the user walks into a lift.

Wi-Fi disappears instantly.

What I initially imagined:

Wi-Fi dies
   ↓
WebSocket close event
   ↓
Server cleans connection

But that is not always what happens.

Sometimes there is no clean TCP close handshake.

Packets simply stop arriving.

Browser
   X
   X
   X
Server

The client is gone.

But the server may still have:

socket
userId
subscriptions
rooms
presence
memory
file descriptor

The server thinks:

client connected ✅

Reality:

client disappeared ❌

That is a zombie connection.

And suddenly I understood why simply listening for:

ws.on("close", ...)

is not enough.

Sometimes nobody tells you that the connection is dead.

You have to discover it yourself.

---

Heartbeats are not just “keep the socket alive”

Before going deeper, I used to think heartbeat meant:

Send something every few seconds so the WebSocket does not disconnect.

That is only part of the story.

A heartbeat can answer two different questions:

1. Is the other side still alive?

2. Can I stop an intermediate proxy
   from considering this connection idle?

Those are related.

But they are not exactly the same problem.

And then I discovered something else:

There isn't just one heartbeat mechanism.

There are at least three layers involved.

Application
    ↓
WebSocket protocol
    ↓
TCP

Each layer has its own idea of keep-alive.

---

Three layers of keep-alive

The easiest mental model for me became:

┌─────────────────────────────┐
│ Application heartbeat       │
│ {"type":"ping"}             │
│ {"type":"pong"}             │
├─────────────────────────────┤
│ WebSocket ping/pong frames  │
│ protocol control frames     │
├─────────────────────────────┤
│ TCP keepalive               │
│ operating system probes     │
└─────────────────────────────┘

At first they all sound like:

ping something and check whether it responds.

But the layer matters.

A lot.

---

Layer 1 — WebSocket protocol ping/pong

The WebSocket protocol itself defines special control frames:

PING
PONG

The server can send:

Server ───── PING ─────→ Client

And the client responds:

Server ←──── PONG ────── Client

If the pong comes back:

connection path appears healthy ✅

If nothing comes back within the expected time:

connection may be dead ❌

This is efficient because it is built directly into the WebSocket protocol.

It is not one of our normal application messages.

This:

{
  "type": "price",
  "symbol": "BTC",
  "price": 117000
}

is application data.

A protocol PING is something different.

It belongs to the WebSocket protocol itself.

---

Then the browser surprised me

On a Node.js WebSocket server, libraries can expose protocol-level ping/pong.

For example, using ws:

ws.ping();

ws.on("pong", () => {
  ws.isAlive = true;
});

Simple.

So naturally I thought the browser could do something similar:

const ws = new WebSocket(url);

ws.ping();

Nope.

Browser JavaScript does not expose protocol ping/pong like that.

The browser handles those control frames internally.

So the flow can be:

Node server
    │
    │ protocol PING
    ▼
Browser
    │
    │ automatic protocol PONG
    ▼
Node server

But my React or browser JavaScript does not get to see that conversation.

That created a strange gap.

The browser might be responding to WebSocket pings perfectly.

But my application code still cannot ask:

"Did my server answer my heartbeat?"

"How long did it take?"

"Should I show Connection Lost?"

"Should I reconnect?"

Now we need another layer.

---

Layer 2 — Application-level heartbeat

If browser JavaScript cannot send a WebSocket protocol ping, we can create our own message.

For example:

{
  "type": "ping"
}

The server recognizes it and responds:

{
  "type": "pong"
}

Now the conversation happens inside our application protocol:

Browser JS
    │
    │ {"type":"ping"}
    ▼
Server application
    │
    │ {"type":"pong"}
    ▼
Browser JS

This is completely visible to our code.

Now the browser can say:

pong received ✅
server reachable

or:

pong missing ❌
connection looks unhealthy

And then trigger:

show disconnected state
        ↓
close stale socket
        ↓
start reconnection

This is slightly more expensive than protocol-level ping/pong because we are sending a regular WebSocket application message.

But in most applications, the extra few bytes are not the interesting part.

The interesting part is that now our JavaScript has visibility.

---

Protocol heartbeat vs application heartbeat

This distinction finally became clear to me:

Protocol-level ping/pong
=
WebSocket layer checking connectivity

while:

Application heartbeat
=
my application checking connectivity
and making decisions from it

The messages may look conceptually similar.

The ownership is different.

And that difference matters especially in browsers.

---

Layer 3 — TCP keepalive

Then there is another heartbeat-like mechanism below WebSocket entirely.

TCP itself can use keepalive probes.

The operating system can periodically check whether a TCP peer is still reachable.

At first I thought:

Perfect. Then why are we building all this heartbeat logic ourselves?

Because the default timing can be completely wrong for WebSocket applications.

A typical Linux TCP keepalive configuration may wait a very long time before probing an idle connection.

Potentially hours.

For a user-facing real-time application, that is useless.

If a user disappears:

10:00 client disappears

10:01 server still thinks socket exists

10:05 still exists

10:30 still exists

...

I do not want to wait for some long operating-system-level timeout before cleaning up the user.

And there is another reason.

There may be infrastructure sitting between my browser and server.

---

The connection can die even when both sides are healthy

Imagine:

Browser
   │
   ▼
Cloudflare
   │
   ▼
Load Balancer
   │
   ▼
Nginx
   │
   ▼
Node.js

The browser is alive.

Node is alive.

The network is fine.

But suppose no messages flow for a while.

One intermediate proxy may have an idle timeout.

It may say:

"No traffic for too long.

Close this connection."

So this:

Browser ───────── Server

is actually more like:

Browser
   │
Proxy A
   │
Load Balancer
   │
Proxy B
   │
Server

And every component may have its own timeout.

That changed another mental model for me.

I used to think the connection belonged to:

client ↔ server

But operationally, the path belongs to every network component in between too.

---

Heartbeats can keep proxies happy

Suppose a proxy kills idle WebSockets after 60 seconds.

And my app has no natural traffic for two minutes.

Without heartbeat:

0 sec
│
│ nothing
│
│ nothing
│
60 sec
│
└── proxy closes connection

But if I send a small heartbeat before that:

0 sec
│
45 sec → heartbeat
│
90 sec → heartbeat
│
135 sec → heartbeat

the connection never remains idle long enough for the proxy to kill it.

So heartbeat is doing another job:

heartbeat
   ↓
traffic exists
   ↓
proxy sees connection active
   ↓
connection stays open

This is different from:

heartbeat
   ↓
no response
   ↓
peer is probably dead

Same mechanism.

Different purpose.

---

Why not heartbeat exactly at the timeout?

Suppose my shortest proxy timeout is 60 seconds.

My first instinct might be:

heartbeat every 60 seconds

But that is cutting it too close.

Imagine:

heartbeat scheduled: 60.0 sec
proxy timeout:        60.0 sec

Now add:

event-loop delay
network jitter
CPU load
packet scheduling
temporary congestion

The heartbeat may arrive at:

60.2 sec

Too late.

The proxy already closed the connection.

So one useful rule is to leave margin.

For example:

proxy timeout = 60 sec

heartbeat around 45 sec

Not because 45 is magical.

Because we do not want our correctness to depend on two timers racing each other.

That is a pattern I keep seeing in distributed systems:

Don't design exactly at the boundary.

Leave margin for reality.

---

Server heartbeat or client heartbeat?

Then another question appeared.

Who should send the heartbeat?

Server → Client?

or:

Client → Server?

The answer depends on what we are trying to detect.

---

Why server-initiated heartbeat makes sense

Imagine a server holds 20,000 WebSocket connections.

For every client it may be keeping:

socket
subscriptions
session state
presence
rooms
memory
file descriptor

If a client disappears silently, the server is the side leaking resources.

So the server has a strong reason to ask:

"Are you still there?"

Pattern:

Server → PING
Client → PONG

If no pong comes back:

timeout
   ↓
terminate socket
   ↓
remove subscriptions
   ↓
cleanup presence
   ↓
free resources

Server-side heartbeat is almost like garbage collection for connections.

It helps the server discover ghosts.

---

The isAlive trick

A very common Node.js ws pattern looks like this:

const interval = setInterval(() => {
  wss.clients.forEach((ws) => {
    if (!ws.isAlive) {
      return ws.terminate();
    }

    ws.isAlive = false;
    ws.ping();
  });
}, 30000);

wss.on("connection", (ws) => {
  ws.isAlive = true;

  ws.on("pong", () => {
    ws.isAlive = true;
  });
});

At first this line confused me:

ws.isAlive = false;

Why would we mark a connected socket as dead before sending the ping?

Then I realized:

false does not really mean:

"This client is dead."

It means:

"I am waiting for this client
to prove that it is alive."

The flow is:

isAlive = true
      ↓
heartbeat cycle begins
      ↓
set isAlive = false
      ↓
send PING
      ↓
receive PONG
      ↓
isAlive = true

Now suppose the pong never arrives.

State remains:

isAlive = false

Next heartbeat cycle:

still false?
     ↓
yes
     ↓
terminate()

That small pattern is actually very elegant.

---

Why terminate()?

There is also a subtle difference between:

ws.close();

and:

ws.terminate();

close() is polite.

It tries to perform a proper WebSocket close handshake.

Conceptually:

Server → CLOSE
Client → CLOSE ACK

But if heartbeat already told me:

"This peer probably disappeared."

waiting politely for that peer makes little sense.

So:

ws.terminate();

means:

Stop waiting.

Kill the connection.

That makes sense for heartbeat failures.

---

But the client has its own problem

Server heartbeat answers:

"Is my client alive?"

The browser has the opposite question:

"Is my server alive?"

Suppose the backend disappears.

The browser might need to:

show "Connection Lost"
stop allowing certain actions
start reconnection
restore state later

This is one reason application-level client heartbeats can be useful.

The browser sends:

{"type":"ping"}

Server replies:

{"type":"pong"}

If no pong appears:

client detects unhealthy server
        ↓
close old socket
        ↓
start reconnection

So I now think about the responsibilities like this:

Server heartbeat
=
"Are you still using the resources
I am holding for you?"

Client heartbeat
=
"Is my backend still reachable?"

They are related but not identical responsibilities.

---

Then mobile makes everything more complicated

Suppose I write:

setInterval(() => {
  sendHeartbeat();
}, 30000);

Looks reliable.

Until the phone goes into the background.

Mobile operating systems care about battery.

Background apps may have:

timers throttled
network suspended
WebSocket disconnected
execution paused

So what I think is happening:

30 sec → heartbeat
60 sec → heartbeat
90 sec → heartbeat

may not happen at all.

The app can disappear into the background for ten minutes.

Meanwhile:

proxy closes connection
network switches
server restarts
socket dies

Then the user opens the application again.

The UI may still have stale assumptions from before suspension.

That means waiting for the next scheduled heartbeat is not ideal.

Instead, when the app becomes active:

foreground
   ↓
verify connection immediately

For a browser:

document.addEventListener("visibilitychange", () => {
  if (document.visibilityState === "visible") {
    checkConnection();
  }
});

The important idea is not this exact API.

It is:

application wakes up
      ↓
don't trust old connection state
      ↓
verify reality again

That is another pattern that extends beyond WebSockets.

After suspension, cached assumptions may no longer match the world.

---

And now the pieces start connecting

This is where I started enjoying the topic.

Heartbeat is not isolated.

Suppose the client detects a missed heartbeat.

Then:

heartbeat timeout
       ↓
connection considered dead
       ↓
close stale socket
       ↓
reconnect

But from Part 1, reconnecting immediately can create a thundering herd.

So:

missed heartbeat
       ↓
disconnect
       ↓
exponential backoff
       ↓
jitter
       ↓
reconnect

Then reconnecting is not enough because state may have diverged.

So:

reconnect
   ↓
restore session
   ↓
restore subscriptions
   ↓
find last sequence
   ↓
replay missed messages

Now one heartbeat failure can activate an entire recovery protocol.

That made me realize that production reliability is rarely one mechanism.

It is usually a chain.

---

My current heartbeat mental model

I no longer think:

heartbeat
=
send ping every 30 seconds

I think:

                HEARTBEAT
                    │
        ┌───────────┴───────────┐
        │                       │
        ▼                       ▼
 detect dead peer        prevent idle timeout
        │                       │
        ▼                       ▼
clean resources        keep network path active
        │
        ▼
trigger recovery

And below that, there are multiple layers:

Application heartbeat
        ↓
visible to our business/application code

WebSocket ping/pong
        ↓
efficient protocol-level health check

TCP keepalive
        ↓
lower-level operating-system mechanism

They overlap.

They are not interchangeable.

---

The bigger lesson

I came into this topic asking:

How do I keep a WebSocket connection alive?

Now I think that question is slightly wrong.

A better question is:

How do I know whether my connection is healthy, and what should my system do when it is not?

Because “alive forever” is unrealistic.

What matters is detection and recovery.

healthy
   ↓
unhealthy
   ↓
detect quickly
   ↓
clean up
   ↓
reconnect safely
   ↓
recover state

That is much closer to how a production system has to think.

---

Next: Reconnection Without Attacking Your Own Server

Heartbeats solve one problem:

We discovered the connection is dead.

Great.

Now imagine 50,000 clients discover that at the same time.

And all of them execute:

connect();

immediately.

We just turned failure detection into a denial-of-service attack against our own recovering backend.

That leads to the next question:

How should thousands of disconnected clients
reconnect without creating another outage?

In Part 3, I want to go deep into:

exponential backoff
jitter
retry ceilings
thundering herd
server recovery

Because retries look harmless when there is one client.

At scale, retry behavior becomes part of the architecture.