bounded queues and back‑pressure for high‑throughput websockets in next.js

when a realtime dashboard needs to push thousands of events per second to dozens of browsers, a naive ws.send() loop can saturate the V8 event loop, drop frames, and eventually crash the server. i ran into this while extending a Next.js API route to broadcast sensor data from a Rust edge worker. the problem isn’t just bandwidth – it’s how quickly the Node process can drain its internal socket buffers.

why back‑pressure matters

Node’s WebSocket implementation respects the underlying TCP back‑pressure: socket.bufferedAmount grows when the OS can’t keep up. if you keep queuing messages without checking that metric, the process memory spikes and GC pauses become noticeable. the usual pattern of for (msg of msgs) ws.send(msg) ignores this signal, leading to “write after end” errors when the client disconnects mid‑burst.

a bounded queue solution

instead of hammering the socket, we can introduce a fixed‑size async queue. producers push payloads into the queue; a consumer task drains it at a rate the socket can sustain. if the queue is full, producers back‑off or drop low‑priority events. this gives a predictable memory ceiling and lets us surface back‑pressure upstream.

// simple bounded async queue (capacity = 5000)
class BoundedQueue<T> {
  private buffer: T[] = []
  private resolvePush!: () => void
  private resolvePop!: (value: T) => void
  constructor(public readonly capacity: number) {}

  async push(item: T) {
    while (this.buffer.length >= this.capacity) {
      await new Promise(r => (this.resolvePush = r)) // wait for space
    }
    this.buffer.push(item)
    this.resolvePop?.()
  }

  async pop(): Promise<T> {
    while (this.buffer.length === 0) {
      await new Promise(r => (this.resolvePop = r)) // wait for item
    }
    const val = this.buffer.shift()!
    this.resolvePush?.()
    return val
  }
}

// usage inside a Next.js API route
export default async function handler(req, res) {
  const ws = new WebSocket(req)
  const queue = new BoundedQueue<string>(5000)

  // producer: pipe Rust edge worker events into queue
  rustWorker.on('data', (msg) => queue.push(JSON.stringify(msg)))

  // consumer: respect socket back‑pressure
  while (ws.readyState === WebSocket.OPEN) {
    const payload = await queue.pop()
    if (ws.bufferedAmount > 1_000_000) {
      // pause draining until OS catches up
      await new Promise(r => setTimeout(r, 10))
    }
    ws.send(payload)
  }
}

tradeoffs to watch

• latency vs drop rate – a small capacity reduces memory but may discard events during spikes. decide whether stale data is acceptable or if you need a replay buffer.
• complexity – adding an async queue introduces another failure surface (e.g., dead‑locks if push/pop promises never resolve). unit‑test the queue under saturation.
• horizontal scaling – if you scale the API route across instances, each will have its own queue. coordinating a global rate limit may require a shared broker like Redis Streams.
• CPU overhead – the while loops with setTimeout are cheap, but in a hot path they can add ~0.2 ms per iteration. profile with node --prof if you see CPU spikes.

production lesson

the biggest surprise was how quickly ws.bufferedAmount grew under burst traffic from the Rust edge worker. once the queue capped at 5 k messages, memory usage stayed flat and the server recovered gracefully after the spike. without that guard, the pod would OOM in minutes.

anyone built a similar back‑pressure layer for websockets in a serverless or edge environment? what queue size or drop‑policy worked best for your latency‑sensitive use case?

Comments

[G]

push keeps one resolvePush. A second producer waiting on a full buffer replaces it, so the first wait never resumes when a slot frees. pop does this with one slot too, and a second consumer on an empty queue stays pending.

[Abhinav Sharma]

Checking bufferedAmount before queueing is the line between a bounded system and a slow leak: without it the event loop keeps accepting work the OS cannot drain, and memory grows until GC pauses become the outage. A fixed-capacity queue turns that into an explicit decision - back off or drop low-priority events - and moves the pressure signal upstream where it can actually be handled.