A route handler that calls a model is fifteen lines. A route handler that still streams once it is behind a CDN, on a serverless runtime, with users who close tabs, needs four more things — and each of them fails silently rather than loudly.
The handler
In the App Router a route handler is a file named route.ts exporting a function per HTTP method. It receives a standard Request and returns a standard Response, which is the part that matters here: Response accepts a ReadableStream as its body, so streaming needs no framework feature at all.
// app/api/chat/route.ts
export const runtime = "nodejs";
export const dynamic = "force-dynamic";
export async function POST(request: Request) {
const { prompt } = await request.json();
if (typeof prompt !== "string" || prompt.length > 4000) {
return Response.json({ error: "bad prompt" }, { status: 400 });
}
const upstream = await fetch("https://api.multigrid.ai/v1/chat/completions", {
method: "POST",
headers: {
"Content-Type": "application/json",
Authorization: "Bearer " + process.env.LLM_API_KEY,
},
body: JSON.stringify({
model: "openai/gpt-4o-mini",
messages: [{ role: "user", content: prompt }],
max_tokens: 800,
stream: true,
}),
signal: request.signal,
});
if (!upstream.ok || !upstream.body) {
const detail = await upstream.text().catch(() => "");
return Response.json(
{ error: "upstream failed", status: upstream.status, detail: detail.slice(0, 500) },
{ status: 502 },
);
}
return new Response(upstream.body, {
headers: {
"Content-Type": "text/event-stream; charset=utf-8",
"Cache-Control": "no-cache, no-transform",
"Connection": "keep-alive",
"X-Accel-Buffering": "no",
},
});
}
dynamic = "force-dynamic" is there because a route handler that Next.js believes to be static gets evaluated at build time and served from a file. A POST handler is dynamic by default, so this is belt-and-braces — but on a GET streaming route it is the difference between a live stream and a cached snapshot of the first response anybody ever got.
Pass the stream through, do not rebuild it
Note what the handler does not do: it does not read the upstream body, parse it, and write a new stream. upstream.body is already a ReadableStream and handing it straight to new Response() keeps the whole thing zero-copy and preserves backpressure end to end — if the browser stops reading, the pause propagates back to the upstream socket by itself.
When you do need to touch the bytes — to count tokens, to inject your own event, to strip a field — use a TransformStream rather than a read-and-rewrite loop. It keeps the same property.
// Count characters as they pass, without buffering the answer.
let seen = 0;
const meter = new TransformStream<Uint8Array, Uint8Array>({
transform(chunk, controller) {
seen += chunk.byteLength;
controller.enqueue(chunk);
},
flush() {
console.log("bytes streamed", seen); // runs when upstream closes
},
});
return new Response(upstream.body.pipeThrough(meter), { headers });
The temptation to await upstream.text() “just to log it” is how streaming routes become non-streaming routes. That one line buffers the entire answer before a single byte reaches the browser, and it turns a 400ms time-to-first-token into an eight-second wait while the endpoint still reports itself as streaming.
Edge or Node
Next.js lets a route handler declare export const runtime. The two values are "nodejs" (the default) and "edge". The choice is not about speed in the way it is usually described.
| Concern | Description |
|---|---|
| Cold start | Edge starts a V8 isolate, which is fast and cheap. Node starts a container. For a route that is called rarely and must feel instant, edge wins on the first request. |
| Available APIs | Edge is web-standard only: fetch, ReadableStream, crypto.subtle. No fs, no net, no native modules. Most Postgres drivers, most Node crypto and most SDKs that touch the filesystem simply do not load. |
| Streaming | Both stream. Edge is built around it; on Node the runtime handles a ReadableStream response body natively. Neither is a reason to choose. |
| Duration limits | Different ceilings on different platforms and plans, and they move — see deploying without timing out. Do not memorise a number; check the one your project is actually configured with. |
The practical rule: if the handler only calls fetch, edge is a reasonable default. The moment it touches a database over TCP, reads a file, or uses a library that has not been audited for edge compatibility, use Node and stop fighting it. Discovering this at deploy time rather than in development is common, because next dev is more forgiving about which globals exist than the edge runtime is.
Route segment options like runtime, dynamic and maxDuration are Next.js conventions and their accepted values have changed across major versions — edge in particular has been renamed and re-scoped more than once. Check the route segment config documentation for the exact version in your package.json rather than copying a value from a blog post.
When streaming silently stops streaming
The most confusing failure in this whole area: everything works locally, and in production the answer arrives all at once at the end. Nothing errored. The cause is almost always something between your function and the browser that buffers.
- A reverse proxy. nginx buffers proxied responses by default. The
X-Accel-Buffering: noheader in the handler above is the documented opt-out and nginx honours it; other proxies need their own configuration. - Compression. A gzip layer that buffers to get a better ratio destroys streaming.
Cache-Control: no-transformasks intermediaries not to recompress, which is why it is in the header block rather than justno-cache. - A CDN. Some edge caches will not stream a response they consider cacheable. A
Content-Typeoftext/event-streamplusno-cacheis the signal that usually prevents it. - Your own logging. Any
await res.text()orres.clone().json()on the path buffers everything. Clone plus read is the sneaky one, because it looks side-effect-free.
Diagnose it with curl rather than the browser, which has its own buffering heuristics for small responses:
curl -N -s -X POST https://your-app.example/api/chat \
-H 'Content-Type: application/json' \
-d '{"prompt":"count slowly to twenty"}' \
| while IFS= read -r line; do printf '%s %s\n' "$(date +%T.%3N)" "$line"; done
Timestamps spread across seconds mean it streams. Timestamps all identical mean something buffered, and now you know it is not your React code.
When the client goes away
request.signal aborts when the client disconnects, and passing it to the upstream fetch is what makes a closed tab actually stop the generation instead of leaving your function billing tokens into a void for another thirty seconds. It is one property and it is left out of nearly every example.
Two caveats. Serverless platforms differ in how reliably a client disconnect propagates to the function, so treat this as an optimisation rather than a guarantee: pair it with a sane max_tokens so the worst case is bounded regardless. And if you need to run work after responding — writing the finished answer to a database, say — that work must not depend on the request signal, or it will be cancelled along with the stream.
The next two problems from here are the duration ceiling, in deploying an AI app without timing out, and whether any of this can be a Server Action instead, in Server Actions and AI.
Top comments (0)