My WebSocket was open the whole time. It just stopped sending anything.

CARL Bouvet — Mon, 31 Aug 2026 22:56:43 +0000

I found it in the data, not in the logs. A gap of about three hours in one market's trade history, on a service that had not restarted, had not thrown, and had not logged a single error.

The socket was OPEN the whole time. The TCP connection was still established. No close event, no error event. Every reconnection strategy I had written — and I had written one, everybody has — was waiting for one of those two events, and neither was ever coming. The service believed it was ingesting a live feed. It was ingesting nothing.

I doubt I am the only one running into this on a hand-rolled WebSocket setup.

I keep about a dozen exchange WebSocket connections open continuously for a market-data service I maintain. This failure has cost me more than every disconnect combined, for one reason: a disconnect is loud. This one is silent, and you find it hours later, in the data, as a hole.

The part that makes it expensive

Every alarm I had was watching the wrong thing.

Process alive? Yes. Socket open? Yes. Memory, CPU, restarts? Normal. Health endpoint? Returning 200, because the health endpoint checked that the process could answer HTTP, which it could. The service was, by every signal I had built, in perfect health. It simply was not doing its job.

It took me a while to accept what that meant: the connection is not broken, it is abandoned. The transport has no opinion about whether the peer is still sending you anything. TCP will happily hold an idle connection open for as long as both sides keep the tuple alive. If you want an opinion about delivery, you have to form it yourself.

My first instinct was wrong

My first hypothesis was that my hosting provider was throttling long-lived connections. It is a comfortable hypothesis: it explains everything and requires no work from me.

Then I went looking for something to back it up and found nothing. Every serious description of VPS steal time describes involuntary oversubscription — your CPU cycles go to a noisy neighbour — and steal time does not sever a TCP connection. I could not find a single fair-use clause or acceptable-use policy restricting long-lived WebSocket connections on a standard VPS plan. As far as I can tell, "my host is throttling my sockets" is a forum story that survives because nobody repeating it has measured anything.

My second wrong instinct was to open tcpdump straight away. I spent an evening in a capture and learned nothing I could not have learned in five minutes with two shell commands, because half the plausible causes are inside your own process and the cheap checks separate them immediately.

Here is the order I would use now, cheapest first:

Cause	Mechanism	Cheap check
Middlebox idle timeout (proxy, LB, NAT)	each hop runs its own timer and drops the entry with no FIN or RST — `readyState` stays `OPEN`	read the timeout config. An AWS load balancer defaults to 60 s, Cloudflare ~100 s, consumer routers 60–300 s. Disconnects at suspiciously regular intervals are this.
NAT `conntrack` eviction	the connection entry (the 5-tuple) disappears; server→client packets are dropped silently	leave the connection idle past the shortest timeout in the chain
Blocked event loop	the kernel receives and ACKs, but no `'message'` is ever emitted — externally indistinguishable from a dead socket	`perf_hooks.monitorEventLoopDelay({ resolution: 10 })`, alert above p99 50 ms
Backpressure / zero window	your receive buffer fills, the kernel advertises Window=0, the sender stops	`ss -tipm`, watch `rcv_space` collapse; Wireshark shows `TCP ZeroWindow`
TCP keepalive miscalibrated	Linux defaults `tcp_keepalive_time` to 7200 seconds — two hours before the kernel even asks	`sysctl net.ipv4.tcp_keepalive_time`
VPS steal time	slows your loop, which feeds back into the blocked-event-loop case; it does not cut TCP	`vmstat 1 30`, column `st`; above 5 % the host is oversubscribed

Three of those six are local to your own process. That is the part I got wrong for weeks.

"Just use ping/pong" — no

The standard answer is a heartbeat: send a ping, expect a pong, kill the connection if the pong does not arrive. The ws heartbeat pattern is the canonical implementation and it is genuinely good. socket.io does the equivalent internally.

But it answers a narrower question than the one you have. Ping/pong proves the peer's protocol layer is responsive. It does not prove the peer is still delivering what you subscribed to. A server can answer pings perfectly while the process behind the feed has stalled, and your connection will look healthy for hours.

I did not expect to find this in CCXT, which is the reference library for crypto exchange connectivity and a lot more battle-tested than anything I have written. Its WebSocket client tracks exactly one liveness timestamp, lastPong, and tears the connection down when now - lastPong > keepAlive × maxPingPongMisses — defaults 30 000 ms and 2.0. Inbound application messages never update it. So a feed whose ping/pong loop stays healthy while its data stops will not trip that watchdog at all. In browser builds, where real ping frames cannot be sent, the client refreshes lastPong on its own interval tick, and the zombie check has nothing left to detect.

You can see it land in their issue tracker, reported repeatedly and still open:

#22662 — watch_order_book hangs indefinitely: socket open, pings and pongs succeeding, no data, no exception raised.
#12918 — fails silently; onOpen, onPing and onPong all fire, nothing is delivered, and verbose: true still reports the socket as alive.
#20667 — a random subset of symbols stops delivering while the connection stays up.

The maintainers' answer is consistently to wrap the call in try/catch and re-enter the loop from user code. For a library that spans a hundred exchanges, that is a defensible place to draw the line. It does mean that if you use it, this part is yours to build.

What I ended up writing down

Once I accepted that the transport was never going to tell me, what was left was small, and more discipline than cleverness:

Proof of life is any accepted inbound message, not only a pong. Liveness must not depend on the peer choosing to answer pings specifically.
One clock per connection — a single lastSeenAt timestamp, set to now whenever rule 1 fires. No per-message-type counters.
One staleness watchdog per connection. If now - lastSeenAt > staleAfterMs, presume the peer dead: stop the watchdog's own timer first, then force the socket closed. It does nothing else. staleAfterMs must be strictly greater than your ping interval, so one slow round-trip cannot cause a false kill.
Reconnect is owned by the close handler, unconditionally — whether the close came from the peer, from a network error, or from your own watchdog. The handler never asks why it fired. That is what keeps stale-kill and network-drop on one code path instead of two.
Every close cancels every connection-scoped timer, idempotently. The watchdog and the close handler will both try; cancelling twice must be safe.
Exponential backoff, capped, N attempts, then a longer cooldown and a counter reset. Never stop retrying forever, never retry in a tight loop.

Rule 5 is the one that actually bit me, and it is the most boring one, which is probably why nobody writes about it.

My reconnect loop leaked one interval per reconnection. In testing that is invisible — you reconnect two or three times and nothing happens. After a week of uptime the service was simply busy. Not leaking memory in any way I could point at, not erroring, just doing a little more work every day than the day before. I went through the ingestion code, then the database, then the parser. The bug was that my watchdog and my close handler each assumed the other had cleared the timer, so on the paths where both ran, one timer survived. It took me much longer to find than the silent socket itself, and the fix was one line and an idempotent guard.

And the exception, because there is always one

On my heaviest feed, rule 1 is wrong.

Binance sends a server ping roughly every 20 seconds whether or not the market is moving. That makes the ping a stronger signal than the data: a quiet market legitimately produces no trades, but a healthy connection always produces pings. So that connector deliberately does the opposite — it keeps two timestamps, watches the ping clock to decide whether to kill the connection, and keeps the data clock for diagnostics. Killing on data silence there would mean reconnecting every time order flow goes quiet.

So the rule is not "data is liveness". It is: measure liveness on whatever the peer has promised to send unconditionally. For most feeds that is the data, because their pings are reactive — they only answer yours. For a feed that pings on its own schedule, the ping is the better clock, and data silence becomes something you alert on rather than something you reconnect on.

What happens after the reconnect

A reconnect, on its own, loses everything that happened during the gap. Detecting the hole is not the same as filling it.

What I run around it, in tiers:

Short gaps, at reconnect. On re-open, refetch exactly [last stored timestamp + 1, now] from the REST endpoint, bounded by a threshold — mine is five minutes — past which the reconnect path gives up and defers. A timestamp boundary is simpler than a sequence-number one and, for a trade feed, sufficient. One known limit I have not fixed: a single unpaginated call caps how many events a gap may contain.
Longer gaps, asynchronously. A separate detector scans stored history for holes over a rolling window and refills them, then recomputes whatever was derived from the missing rows. Writes are upserts keyed on {market, timestamp}, so the same gap can be filled twice without consequence. The layers overlap on purpose; idempotency is what makes that safe.
Historical loads. An offline importer for backfills measured in months, entirely outside the live path.

The handoff between these layers is explicit in the code rather than implied — the reconnect path logs that it is deferring, and to whom. That was the fix for a real class of bug: two layers each assuming the other had covered the gap.

What I did not build

Worth stating, because a post that only lists what you did is marketing. None of this exists in my stack: redundant streams with failover (Coinbase publishes two endpoints precisely for this), sequence-number gap detection — a timestamp boundary will never tell you that you lost message N+1 — proactive reconnection before a forced disconnect, jitter on the backoff, snapshot-driven state reset.

If you are maintaining an order book you need all of it. Detection is the sensor that would trigger the failover; the system should survive the sensor being wrong, in both directions.

The code

I extracted the invariant from my own connectors — 13 of them across seven services, which had drifted into three different answers to the same question — and published it: github.com/CARL-Bouvet/resilient-ws. One file, 366 lines, no dependencies, MIT, four tests on injected fake timers. It runs anywhere the standard WebSocket global exists.

Those near-copies were never really separate implementations. They were one design decision that nobody had written down, drifting apart quietly for a year. So I wrote it down.

English is not my native language — I write it a good deal better than I speak it. Some sentences may read oddly.

DEV Community: CARL Bouvet