DEV Community

Royal Simpson Pinto
Royal Simpson Pinto

Posted on

Building a streaming investigation assistant that tool-calls an MCP server and cites case IDs

#cloudflare #ai #typescript #webdev

I keep a public registry of documented AI-agent failures called AgentPostmortem: real incidents where an agent got prompt-injected, deleted a database, or spun into a runaway loop, each written up as a case file with an ID like APM-0048. The registry is useful, but browsing it is a chore. If I want to know why refund agents get prompt-injected, I have to guess search terms, open cases, and read them one by one.

So I built casebook-chat: a chat interface that does the searching for me. You describe an incident or ask about a failure mode, and the assistant searches the live registry, pulls the relevant case files, and answers in plain language while citing the real case IDs it used. The whole thing runs as a single Cloudflare Worker.

The core idea

The registry already speaks MCP (Model Context Protocol). It exposes three tools over JSON-RPC at mcp.agentpostmortem.com/mcp. Rather than reimplement search or copy the data into a new database, I wanted the chat model to call those tools directly and ground its answers in whatever they returned.

The three tools map cleanly onto how you actually investigate an incident:

  • search_cases(query) runs a full-text search over the failure case files and returns ranked summaries with case IDs.
  • get_case(id) fetches one case in full: outcome, verified facts, unknowns, and lessons.
  • similar_failures(description) takes a plain-language description of an incident and matches it against known failures.

The model decides which to call and when. Ask "why do refund agents get prompt-injected?" and it searches. Say "my agent deleted a database" and it reaches for similar_failures. Once it has a case ID, it can pull the full detail with get_case.

How it works

The stack is deliberately small. The browser runs a Vite and React app using @ai-sdk/react's useChat, which POSTs the message history to /api/chat. That endpoint is a Hono route inside a Cloudflare Worker, and the same Worker also serves the static UI through Workers assets. One deploy, one origin, no separate backend.

Inside the route, I call the Vercel AI SDK's streamText against Groq's llama-3.3-70b-versatile, passing the three tools defined with the AI SDK tool() helper and zod input schemas. Each tool's execute is just a fetch to the MCP server wrapping the arguments in a JSON-RPC tools/call:

search_cases: tool({
  description: "Full-text search over documented AI-agent failure cases...",
  inputSchema: z.object({ query: z.string() }),
  execute: async ({ query }) => callMcpTool("search_cases", { query }),
}),
Enter fullscreen mode Exit fullscreen mode

Because these are genuine failure investigations, the model often needs more than one hop: search, find a promising case, then fetch its full detail before answering. I allow multi-step tool chains but cap the loop with stopWhen: stepCountIs(5), so a model that keeps calling tools cannot run away and burn the whole budget.

Streaming with tool calls rendered inline

The nice part is the streaming. The result comes back through toUIMessageStreamResponse(), which turns the run into a UI message stream where tool inputs and outputs arrive as typed message parts, not just text. On the client, useChat exposes each assistant message as an ordered list of parts, and I render them in order: text parts become markdown, and anything whose type starts with tool- becomes an inline activity chip.

Each chip is a collapsible component that reads the part's state field to show where the call is. While the model is still forming or running the call (input-streaming, input-available), it shows a spinner and the word "running". When the output lands (output-available) it flips to a checkmark and a one-line summary; on failure (output-error) it shows a cross and the error text. Expand a chip and you see the exact arguments the model sent and the raw result it got back, pretty-printed. So the tool calls appear woven into the answer as it streams, in the order they actually happened, and you can audit every step.

Citations that mean something

Grounding is enforced through the system prompt, not wishful thinking. The investigator is told to answer only from what the tools returned, to cite the real case IDs it used, and never to invent case IDs or facts. If the registry returns nothing relevant, it is instructed to say so plainly and label any general-knowledge answer as such. It also calls tools silently instead of narrating "let me search the registry", so the visible answer stays clean while the chips carry the process.

The MCP client itself is built to never throw. It retries once on a network error or a 5xx, times out each attempt at around eight seconds, and on persistent failure returns a structured "the registry is temporarily unavailable" string that the model is told to relay to the user rather than crash on. It also handles the fact that the MCP server can answer either as JSON or as a text/event-stream, parsing the last JSON-RPC frame out of the SSE body when needed.

One honest limitation

The grounding is only as strong as the prompt. There is no post-hoc verification step that checks the case IDs in the final answer against the IDs the tools actually returned. The system prompt tells the model not to invent IDs, and in practice llama-3.3-70b on Groq follows that well, but "the model was instructed not to hallucinate" is a softer guarantee than "the app refuses to emit an ID the tools did not surface". If I wanted a hard guarantee, I would parse the assistant's cited IDs and cross-check them against the tool outputs before rendering. That is the honest gap between "cites case IDs" and "provably cannot fabricate a case ID".

Two smaller notes: inference runs on the Groq free tier, so rate limits and the occasional slowdown apply, and 429s are caught and surfaced as a friendly "the model is busy" message rather than an error. And the registry data is community-documented, so the answers are only as complete as the casebook behind them.

Closing

What I like about this build is how little glue it needed. An MCP server that already exposes the right tools, an AI SDK that streams typed tool parts, and a single Worker to host both halves. The result is a chat that does not just talk about agent failures but shows its work: every claim traceable to a case file you can open yourself.

Code is here: https://github.com/AgentPostmortem/casebook-chat

Top comments (0)