Streaming LLM Responses in Node Without Breaking Backpressure

#node #typescript #streams #ai

Book: AI That Answers
The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

The streaming handler everyone writes first looks like this, and it
works on your laptop every single time.

app.post("/chat", async (req, res) => {
  res.setHeader("Content-Type", "text/event-stream");
  const stream = client.messages.stream({ /* ... */ });

  for await (const event of stream) {
    if (event.type === "content_block_delta") {
      res.write(`data: ${JSON.stringify(event.delta)}\n\n`);
    }
  }
  res.end();
});

It works on your laptop because your laptop is talking to itself over
loopback and res.write always returns true. Put it behind a
mobile connection on a train and the same code grows a memory leak,
keeps paying for tokens nobody will read, and holds a socket open
after the user has closed the tab.

Three separate problems, and they share a root: res.write has a
return value and the loop ignores it.

What the return value means

res.write() returns a boolean. true means the data went into the
kernel socket buffer and you may keep writing. false means the
internal buffer has passed highWaterMark and you should stop until
the drain event fires.

Ignoring it does not throw. Node keeps accepting writes and queues
them in memory. So when the model produces tokens faster than the
client can accept them — which on a slow connection is always — the
difference accumulates in your process heap.

One request doing this is nothing. A few hundred concurrent requests
each buffering a long response is an out-of-memory kill, and the
stack trace points at whatever allocated last rather than at the
handler responsible.

// the bug, stated plainly
const ok = res.write(chunk);  // ok === false, ignored
// loop continues immediately, buffer grows

Let the pipeline handle it

You can honour drain manually, and it is instructive to see once:

async function write(res: ServerResponse, chunk: string) {
  if (!res.write(chunk)) {
    await once(res, "drain");
  }
}

But do not build a whole handler this way. Node has a primitive that
gets the details right — pipeline propagates backpressure,
destroys the source on downstream failure, and cleans up on abort.

import { pipeline } from "node:stream/promises";
import { Readable } from "node:stream";

app.post("/chat", async (req, res) => {
  res.setHeader("Content-Type", "text/event-stream");
  res.setHeader("Cache-Control", "no-cache, no-transform");
  res.setHeader("Connection", "keep-alive");

  const controller = new AbortController();
  const stream = client.messages.stream(
    { model: "claude-opus-5", max_tokens: 4096, messages },
    { signal: controller.signal },
  );

  const sse = Readable.from(toSse(stream));

  try {
    await pipeline(sse, res, { signal: controller.signal });
  } catch (err) {
    if (!isAbort(err)) logger.error("stream failed", { err });
  }
});

Readable.from accepts an async iterable, and an async generator is
the natural way to write the SSE framing:

async function* toSse(stream: MessageStream) {
  for await (const event of stream) {
    if (event.type === "content_block_delta") {
      yield `data: ${JSON.stringify(event.delta)}\n\n`;
    }
  }
  yield "data: [DONE]\n\n";
}

The generator only advances when the consumer pulls. That is the
whole fix — pipeline pulls at the rate res drains, the generator
suspends at yield, and the upstream SDK stream stops being consumed
while the client catches up. No manual drain handling, no unbounded
buffer.

Stop paying when the client leaves

The second problem is money. A user closes the tab; your handler
keeps consuming the model stream to completion and you are billed for
every token after they left.

Wire the socket close to the abort controller:

res.on("close", () => {
  if (!res.writableEnded) controller.abort();
});

close fires when the underlying connection goes away, whether or
not the response completed. Guarding on writableEnded distinguishes
a client disconnect from your own normal end().

Passing the same signal to both the SDK call and pipeline means one
abort tears down both sides. Without the SDK signal, aborting the
pipeline stops your writes but leaves the HTTP request to the
provider running.

Whether you are billed for a cancelled request depends on the
provider and on how far generation got, so treat this as reducing
waste rather than guaranteeing zero. The point is that continuing to
read a stream nobody is listening to has no upside.

The proxy will cut you off

The third problem is not in your code. Reverse proxies and load
balancers apply idle timeouts, and a model that thinks for forty
seconds before its first token looks exactly like an idle connection.

Two defences. Flush headers immediately so the proxy sees a response
has begun:

res.writeHead(200, {
  "Content-Type": "text/event-stream",
  "Cache-Control": "no-cache, no-transform",
  "X-Accel-Buffering": "no",
});
res.flushHeaders();

And send a comment line as a heartbeat while waiting:

const beat = setInterval(() => {
  if (!res.writableEnded) res.write(": ping\n\n");
}, 15_000);
res.on("close", () => clearInterval(beat));

A line starting with : is an SSE comment. Clients ignore it;
proxies count it as activity. no-transform and X-Accel-Buffering
matter because a proxy that buffers your response defeats streaming
entirely — the user waits for the whole answer, then gets it at once,
and the bug report says "streaming does not work" with no error
anywhere.

Check your own platform's idle timeout rather than assuming one. They
vary widely and they change.

If you are on Web Streams

Newer runtimes and frameworks hand you a ReadableStream instead.
The shape differs; the principle does not.

export async function POST(req: Request) {
  const stream = client.messages.stream({ /* ... */ });
  const encoder = new TextEncoder();

  const body = new ReadableStream({
    async pull(controller) {
      const { value, done } = await iter.next();
      if (done) return controller.close();
      controller.enqueue(encoder.encode(frame(value)));
    },
    cancel() { stream.abort(); },
  });

  return new Response(body, { headers: sseHeaders });
}

pull is called when the consumer wants more, which is backpressure
by construction. cancel fires on client disconnect, which is where
the abort belongs. If you use start with a while loop and enqueue
everything as fast as it arrives, you have rebuilt the original bug
with different syntax.

The check

Throttle a client to a slow connection, stream a long response, and
watch process.memoryUsage().heapUsed. Flat means the pipeline is
pulling. A climb that tracks output length means something in the
chain is buffering, and it is worth finding before a few hundred
users find it for you.

If this was useful

AI That Answers covers
streaming as a first-class concern rather than an afterthought —
framing, cancellation, partial-response handling, and what to do when
a stream ends mid-structure.