A single-process WebSocket server is easy. You keep a Map<roomId, Set<WebSocket>>, you loop over the set, you send. It works beautifully right up until the moment you run a second instance, at which point half your users stop receiving half your messages and the bug looks like a networking problem rather than an architectural one.
pub-trivia.app pushes questions, answers and leaderboard updates to every phone in a venue in real time. Here is the fan-out layer, which is small enough to read in full, and the five decisions inside it that are not obvious.
The shape
Each instance tracks only the sockets it owns:
export const sessionStore = new Map<string, Set<WebSocket>>()
export function addClient(sessionId: string, ws: WebSocket): void {
let clients = sessionStore.get(sessionId)
if (!clients) { clients = new Set(); sessionStore.set(sessionId, clients) }
clients.add(ws)
}
export function getClients(sessionId: string): Set<WebSocket> {
return sessionStore.get(sessionId) ?? new Set<WebSocket>()
}
Broadcasting does not touch that map directly. It publishes to Redis, and every instance, including the one that published, delivers to its own local sockets when the message comes back round:
function channelFor(sessionId: string): string {
return `ws:session:${sessionId}`
}
export async function publishEvent(sessionId: string, event: WsEvent): Promise<void> {
const payload = JSON.stringify(event)
try {
await getPublisher().publish(channelFor(sessionId), payload)
} catch (err) {
console.warn('[redis] publish failed, falling back to local broadcast:', err)
deliverToLocalClients(sessionId, payload)
}
}
That is the entire horizontal scaling story. No sticky sessions, no shared socket registry, no consistent hashing of rooms onto instances.
Decision 1: one channel per room, not one channel for everything
ws:session:<id>, subscribed to only while an instance has at least one client in that session.
The lazy alternative is a single global channel with the room id in the payload, so every instance receives every message and filters. That works at ten rooms and quietly stops working at a thousand: each instance is now deserialising every message in the system to discard most of them, and your CPU scales with total traffic rather than with your share of it.
Per-room channels mean an instance receives exactly the messages it has someone to deliver to.
Decision 2: two Redis connections, because ioredis makes you
// One client for general commands (publish, get, set), and one dedicated
// subscriber, because a subscribed connection can only run sub/unsub commands.
This is a Redis protocol property, not a library quirk: once a connection enters subscriber mode it will not answer anything else. If you have ever seen a mysterious Connection in subscriber mode, only (P)SUBSCRIBE / (P)UNSUBSCRIBE and quit are allowed, this is it.
The publisher is created lazily on first publish, and the subscriber on first subscription. On a single-instance deploy with no traffic, neither connection ever opens.
Decision 3: Redis being down must not be an outage
const client = new Redis(REDIS_URL, { lazyConnect: true, enableReadyCheck: true, maxRetriesPerRequest: 3 })
client.on('error', (err) => {
// Log but never crash. The WS server must survive Redis being temporarily
// unavailable. In-process broadcasts still work for single-instance deploys.
console.warn('[redis] Connection error (non-fatal):', err.message)
})
Three things are doing work here. lazyConnect means a missing Redis does not crash the process at startup. The error handler means it does not crash later either, and in Node an unhandled error event on an EventEmitter is fatal, so this is not optional. And the publish fallback delivers to local clients, so with one instance the app is fully functional with Redis entirely absent.
Degrade along the axis you can afford. Here, losing Redis costs you cross-instance delivery, not delivery.
Decision 4: a send failure is one client's problem
export function sendToClient(ws: WebSocket, event: WsEvent): void {
try {
ws.send(JSON.stringify(event))
} catch (err) {
console.error('[broadcaster] Failed to send to client:', err)
}
}
Loops over a set of sockets are exactly where a single dead connection takes down a broadcast. One phone that walked out of WiFi range mid-question throws on send, the exception escapes the loop, and everyone after it in iteration order gets nothing. They will assume the app froze.
Catch at the individual send. It is the smallest unit where the failure is meaningful, and it is the only one where "skip and continue" is the right answer.
Decision 5: authenticate with a deadline
An open WebSocket that has not told you who it is is an open resource with no owner. Ours has five seconds:
const AUTH_TIMEOUT_MS = 5_000
const timeout = setTimeout(() => {
sendAndClose(ws, 'auth_timeout', 'No auth message received within 5 seconds')
resolve(null)
}, AUTH_TIMEOUT_MS)
The client's first message must be an auth frame naming a session and a role. We then validate that the session exists and is not already completed, before the socket is put in any map. Without the deadline, a script can open sockets and simply never speak, and every one of them costs you a file descriptor and a slot in your connection limit.
The handshake also does something worth copying: on success, before any live event arrives, the server sends a state snapshot. A player who joins mid-question gets the current question, the remaining time and the leaderboard immediately, rather than sitting on a blank screen until the next event happens to fire. Real-time systems need a "here is where we are" as well as a "here is what changed", and the cheapest place to send it is the moment after auth.
And a fallback that is allowed to be dumb
Phones on pub WiFi lose WebSockets. When the socket cannot be established, the client polls a snapshot endpoint every ten seconds instead. The quiz continues, a little coarser.
That fallback has its own consequence, which caught us out once: a hundred phones polling from behind one venue NAT is a lot of requests from one IP, and our global rate limiter was sized for the happy path where nobody polls. The fallback triggered a 429 for the whole room, which is the exact opposite of what a fallback is for. Size your limits for your degraded path, not your healthy one.
Watch it run
pub-trivia.app/features/live-leaderboard is the product-facing description. The interesting way to try it is with two devices: the free tier needs no card, so start a session on a laptop, join from a phone with the QR code, and launch a question. The phone's screen changes before you have finished lifting your finger off the host's button, and everything above is what makes that true whichever instance each of you happened to land on.
Top comments (0)