DEV Community

Cover image for Autopsy #10: 10,000 Users, 80,000 WebSockets, and a Synchronized Stampede
frontend.rocks
frontend.rocks

Posted on Edited on Originally published at frontend.rocks

Autopsy #10: 10,000 Users, 80,000 WebSockets, and a Synchronized Stampede

Eight tabs per user at nine in the morning

Thomas asked for live ticker widgets on the trading dashboard so portfolio managers could watch price ticks without hitting refresh. We shipped useRealtimeFeed(), which opened a WebSocket inside each widget tree. In local testing with one tab open and a mock feed running at 5 messages per second, CPU sat at 2% and our integration suite passed in four seconds. Nobody tested what happens when someone opens eight browser tabs before coffee.

At 09:00 on market open, 10,000 power users logged in with an average of 8 tabs open across watchlists, order books, and charts, and upstream traffic climbed to 400 messages per second. Marc watched 80,000 concurrent WebSockets hit the realtime gateways, pushing TLS handshake CPU to 100%. On the client, each user's 8 tabs were running 3,200 JSON.parse() calls per second on their main threads, pushing p95 INP to 1,840 ms. Clicking a tab or typing a ticker symbol took nearly two seconds to respond.

Then a single gateway node restarted.

One thousand milliseconds of precision

Karim had written a neat reconnect helper when he built the hook:

ws.onclose = () => {
  setTimeout(() => connect(), 1000);
};
Enter fullscreen mode Exit fullscreen mode

When the alerts fired in #frontend, we also added a 100 ms React setState throttle to calm down rendering.

When the gateway node dropped, 80,000 sockets waited 1,000 milliseconds in lockstep and slammed the TLS terminator in the exact same millisecond, knocking it over again. They waited another 1,000 milliseconds and repeated the charge for fourteen minutes straight, turning a 5-second node recycle into a 14-minute 503 outage. Meanwhile, the 100 ms React throttle only delayed setState; it did nothing to stop 8 tabs from opening 8 sockets or running 3,200 JSON.parse() calls per second before the throttle even ran. Every tab was still paying the full deserialization cost for data that was already obsolete two frames later.

Sharing one socket across eight tabs

First, we needed cross-tab leader election via Web Locks and BroadcastChannel (navigator.locks.request('realtime-feed', ...)). Only one tab per browser origin holds the upstream WebSocket at a time, cutting Marc's gateway load from 80,000 sockets down to 10,000. The remaining 7 tabs subscribe passively to BroadcastChannel frames without opening a single TCP connection, and if the leader tab closes, the browser automatically grants the lock to the next waiting tab.

Second, we moved JSON.parse() and frame normalization into a Dedicated Web Worker so 400 msg/s bursts never touched the UI thread's input responsiveness.

Backpressure and jitter

Interactive Failure Simulator — Tap to step through the live model

The background tab timer trap

First try after deploying leader election: when a user switched away from the leader tab and left it in the background while trading in tab two, all 7 visible tabs stopped updating after a minute.

Chrome throttles setTimeout and requestAnimationFrame in background tabs to 1 Hz (and even lower after five minutes) to save battery. Because our coalescing flush loop was running inside the leader tab's window context, putting that tab in the background starved every foreground tab listening on BroadcastChannel. Portfolio managers staring at tab two thought the market feed had frozen while tab one sat quietly throttled in the background.

Moving the WebSocket and its coalescing loop into the Worker, and releasing the Web Lock on visibilitychange whenever a visible tab could take over, kept frames flowing at full speed regardless of which tab had focus. We also verified in Playwright across two simultaneous pages that closing the leader tab transfers the Web Lock and resumes BroadcastChannel delivery in under 50 milliseconds.

What stayed in the codebase

A reconnect delay without randomness is just a scheduled denial-of-service attack.

Marc framed the graph of the 80,000 synchronized reconnects and hung it above his desk, right next to the gateway CPU chart that now sits quietly at 18% even when every trader in London opens ten tabs before the opening bell rings on Monday morning. Nobody has shipped a raw setTimeout reconnect since.

Top comments (0)