Deploying in the middle of a question: graceful shutdown for a WebSocket game server

Stateless HTTP services made everyone lazy about shutdown. A request is in flight for eighty milliseconds, the platform drains the load balancer, you are done. Nobody notices a deploy.

A WebSocket server for a live quiz is different. At any moment there are ninety open connections, several of them counting down a fifteen second answer window, and the process holds setTimeout handles that are going to auto close those questions. If you SIGTERM that carelessly, ninety phones freeze on a question that will never resolve and their owners look at the quizmaster.

Here is the whole shutdown path, and the order of it is the interesting part.

Arm the escape hatch first

const FORCE_EXIT_TIMEOUT_MS = 10_000

const shutdown = (signal: string): void => {
  console.log(`[shutdown] Received ${signal}. Initiating graceful shutdown.`)

  const forceExitTimer = setTimeout(() => {
    console.error('[shutdown] Force-exiting after timeout.')
    process.exit(1)
  }, FORCE_EXIT_TIMEOUT_MS)

  if (forceExitTimer.unref) forceExitTimer.unref()

  // ...everything else
}

The force exit is the first statement, before any of the graceful work. This is not an ordering aesthetic, it is the only ordering that works. If you arm it after the cleanup, then any cleanup step that hangs means you never arm it at all, and your orchestrator waits out its own grace period and SIGKILLs you anyway, except now it took thirty seconds instead of ten and you got no log line explaining why.

The unref() is the pair to that. An armed ten second timer would keep the event loop alive for ten seconds all by itself, which would turn a shutdown that should complete in eighty milliseconds into one that always takes the full ten. unref() says: this timer may fire if the process is still alive, but it does not by itself count as a reason to stay alive.

Arm the deadline, then do not let the deadline be the thing that causes the delay.

Cancel the timers you own before you cancel the connections

clearAllTimers()

The server holds one setTimeout per open question, armed for reveal + timeLimit + grace, which fires and broadcasts question_closed. Those have to be cleared before the sockets go, not after. Clearing them after means a timer can fire during shutdown and try to broadcast into a session whose clients are already terminated, which is a stack of handled but pointless errors in the log of every single deploy, exactly the kind of noise that trains you to ignore shutdown logs.

Tell the clients, then hang up

const shutdownEvent = buildServerShutdown()
const serialised = JSON.stringify(shutdownEvent)

for (const sessionId of getAllSessionIds()) {
  for (const ws of getClients(sessionId)) {
    try {
      ws.send(serialised)
    } catch (err) {
      console.error('[shutdown] Error sending server_shutdown to client:', err)
    }
  }
}

for (const sessionId of getAllSessionIds()) {
  for (const ws of getClients(sessionId)) {
    try {
      ws.terminate()
    } catch (err) {
      console.error('[shutdown] Error terminating WebSocket:', err)
    }
  }
}
sessionStore.clear()

Two separate passes, not one. If you send and terminate in the same loop, the first client's terminate can start tearing down shared machinery while the ninetieth client's send is still queued. Send to everybody, then hang up on everybody.

Every send and every terminate is individually wrapped, because the one thing you can be certain of during shutdown is that some of those sockets are already half dead. An exception on client 12 must not prevent clients 13 through 90 from being told anything.

And then:

wss.close((err) => {
  if (err) console.error('[shutdown] Error closing WebSocket server:', err)
  clearTimeout(forceExitTimer)
  console.log('[shutdown] All connections closed. Exiting.')
  process.exit(0)
})

wss.close() stops accepting new connections and calls back when the existing ones are done. That is where the force exit timer gets cleared and where we exit 0.

process.once('SIGTERM', () => shutdown('SIGTERM'))
process.once('SIGINT', () => shutdown('SIGINT'))

once, not on. An impatient operator pressing Ctrl+C twice should not run the whole sequence twice concurrently.

The part that makes it survivable

All of the above is good manners. None of it is what actually saves the quiz. What saves the quiz is that the process holds no state worth preserving.

The client hook treats a closed socket as a normal event. It reconnects with exponential backoff, 1s, 2s, 4s, capped at 30s. On reconnect it sends its auth message again, and the last thing the server does after auth_ok is push a complete state snapshot:

addClient(sessionId, ws)
subscribeSession(sessionId)
incrementPresence(sessionId)
send(ws, { type: 'auth_ok' })

try {
  const snapshot = await buildSessionStateSnapshot(sessionId)
  sendToClient(ws, buildStateSnapshot(snapshot))
} catch (err) {
  console.error('[auth] Failed to build/send state_snapshot after auth_ok:', err)
  // deliberately not fatal: auth succeeded, the client can still receive events
}

That snapshot is built from Postgres. Which question is current, when it launched, whether it has closed, the leaderboard. It is not read from the memory of the process that just died, and it does not require the old instance to hand anything to the new one. There is no session affinity to preserve and no sticky routing to get right.

So a deploy mid question looks like this from a player's phone: a brief "Reconnecting", then the same question with the countdown in the right place, because the deadline was a timestamp in the database all along and the client computes the remaining time against a synced server clock rather than counting down locally.

The in memory Map of session id to socket set is a routing table, not a source of truth. Reconstructing it costs one database read per reconnecting client.

That is the design decision the shutdown code depends on, and the one worth stealing. The shutdown handler is thirty lines and it can afford to be blunt because nothing it discards was irreplaceable. If your graceful shutdown needs to save something before it exits, the real problem is upstream.

Watch it happen

The recovery path is visible without any special tooling. Read how the live leaderboard updates for what is supposed to arrive and when, then start a free session at pub-trivia.app, no card required, join it on a phone, and toggle airplane mode for a couple of seconds in the middle of a question.

The countdown you come back to is the room's countdown, not one that restarted for you. If it were restarting, the timer would be living in the client, and a deploy would be a much worse day.