DEV Community

Youngjun Yoon
Youngjun Yoon

Posted on Edited on

Streaming an AI agent's tool calls into a Next.js UI: the FastAPI pattern I keep reusing

Most AI app prototypes I get handed are stuck on the same screen: the chat box shows a spinner, then a fully formed paragraph appears. Under the hood the agent may have called four tools, waited thirty seconds and retried twice, but the person using it sees nothing until the end. On a slow model that reads as a broken app, and people close the tab before the answer lands.

Streaming the agent's steps, not just its tokens, fixes most of that. Here is the shape I use for FastAPI plus Next.js builds, with the parts that actually broke in production.

The event contract comes first

Before any code, decide what the client is allowed to see. I keep four event types and nothing else:

  • token: a piece of assistant text
  • tool_start: {"name": "...", "args": {...}}
  • tool_end: {"name": "...", "ok": true, "ms": 812, "summary": "..."}
  • done: {"finish_reason": "stop"}

Everything the UI renders comes from one of these. That constraint is what stops the front end from turning into a pile of special cases, and it makes replaying a run trivial later: you store the events, not the rendered HTML.

FastAPI side: an async generator behind StreamingResponse

The endpoint does not call the model directly; it yields events from a queue that the agent runner writes to. That separation matters because tool calls happen inside the agent loop, often in a thread, and you do not want the HTTP handler to know about them.

import asyncio, json
from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()

async def run_agent(prompt: str, q: asyncio.Queue):
    try:
        async for ev in agent_loop(prompt):          # yields dicts
            await q.put(ev)
    except Exception as e:
        await q.put({"type": "error", "message": str(e)[:200]})
    finally:
        await q.put(None)

@app.post("/api/agent")
async def agent(payload: dict):
    q: asyncio.Queue = asyncio.Queue()

    async def gen():
        asyncio.create_task(run_agent(payload["prompt"], q))
        while True:
            ev = await q.get()
            if ev is None:
                break
            yield f"data: {json.dumps(ev, ensure_ascii=False)}\n\n"

    return StreamingResponse(
        gen(),
        media_type="text/event-stream",
        headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
    )
Enter fullscreen mode Exit fullscreen mode

Two headers there are not optional. Without X-Accel-Buffering: no, a proxy may buffer the whole stream and the user gets the spinner experience anyway. Without Cache-Control: no-cache, some CDNs will happily cache a partial response.

Do not put blocking calls inside an async generator. If your model SDK is synchronous, wrap the call with asyncio.to_thread and push into the queue from there; a blocking loop inside the generator stalls the whole worker for every other request it is serving.

Next.js side: one hook, no library

EventSource only does GET, so for a POST body I use fetch with a reader:

export async function streamAgent(prompt, onEvent) {
  const res = await fetch("/api/agent", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ prompt }),
  });
  if (!res.body) throw new Error("no stream");

  const reader = res.body.getReader();
  const dec = new TextDecoder();
  let buf = "";

  while (true) {
    const { value, done } = await reader.read();
    if (done) break;
    buf += dec.decode(value, { stream: true });
    const parts = buf.split("\n\n");
    buf = parts.pop() ?? "";
    for (const p of parts) {
      const line = p.split("\n").find((l) => l.startsWith("data: "));
      if (line) onEvent(JSON.parse(line.slice(6)));
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

The buffering detail (buf, split("\n\n"), keep the remainder) is where these break. A chunk boundary can land in the middle of a JSON object, and JSON.parse on a partial object throws once, then the handler is gone and the stream silently stops. Keep the leftover in the buffer and parse only complete frames.

Three failure modes worth designing for on day one

Idempotency. A user double-clicking send, or a client reconnecting, can start the same run twice. Send a client-generated run_id and have the server treat a repeat within a short window as a lookup rather than a new run. Cheap to add, expensive to debug later.

Cancellation. When the tab closes, the generator is cancelled, but the agent task keeps burning tokens unless you cancel it. Hold the task reference and cancel it in a finally block, or run the agent under a timeout shorter than the user's patience.

Persistence of events, not text. Store the event list per run so a refresh can rebuild the transcript, including which tools ran and how long they took. Users read "searched pricing, 1.2s" as evidence the app is working, and it costs you one table.

A note on the database layer

If this is a Supabase-backed app, the quickest way to make the prototype "work" is to turn off row level security or grant table access to the anon role. Do not do that. An anon grant makes every row readable by anyone who opens the network tab, including other users' conversations. Keep RLS enabled on every table that holds user data, write policies that bind rows to auth.uid(), and let the API use the service role key on the server only, never in the browser bundle. If a policy is missing you get empty results instead of private data, which is the failure you want.

What this buys you

With the event contract in place the UI work becomes small: a step list that appends on tool_start and closes on tool_end, text that appends on token. Users can see the agent working, cancel it, and come back to it after a refresh. In a five-day build, that is usually the difference between a demo that gets shared and one that gets closed.

I build these agent backends in Python and TypeScript from Seoul, usually finishing a half-built Lovable, Bolt or Cursor project in three to five days. If you have one that works locally and dies in production, the studio page is at cpcai-studio.netlify.app/en and email works too: yyhak12345@gmail.com

Top comments (0)