Making a knowledge base queryable by any agent over MCP

Every team debugging an AI-agent incident is quietly rediscovering a failure mode that someone else already documented. A support bot approves hundreds of fake refunds because a ticket told it to. A retry loop runs away and spams customers. A stale cache turns into a confident hallucination. These are not novel bugs. They are recurring shapes, and the knowledge about how they played out and what fixed them is usually locked in a blog post or a postmortem doc nobody thinks to search at 2am.

I built AgentPostmortem as a public registry of these documented failures. But a registry a human has to remember to visit is a registry that goes unused during the exact moment it matters: while an agent is actively investigating. So I built Casebook MCP: a remote MCP server that turns the registry into tools any agent can call. Now Claude Code, Cursor, or an agent built on the Claude Agent SDK can ask, mid-investigation, "has anything like this happened before?"

The core idea

Casebook exposes four tools over the Model Context Protocol:

• search_cases(query, tag?) for ranked full-text search over the case files, with an optional tag filter.
• get_case(id) for the full detail of one case: outcome narrative, verified facts, unknowns, and lessons.
• similar_failures(description) which takes a free-text incident description and returns the closest documented failures by keyword overlap.
• list_tags() for every failure-mode tag with its description.

The tool that earns its keep is similar_failures. You paste in what is actually happening ("our support bot was tricked by text in a ticket into approving refunds") and get back real precedents, ranked, with the shared keywords, the outcome, and the lessons attached. That is the difference between a lookup table and something an agent can reason against.

How it works

The whole thing runs as a single Cloudflare Worker. I chose to implement the MCP transport directly against the 2025-03-26 streamable HTTP spec in stateless mode rather than pull in a framework. There is one POST /mcp endpoint that handles initialize, tools/list, and tools/call. No sessions, no Durable Objects, no auth, because the data is public and read-only. That decision keeps the server a plain request/response function, which is exactly what a Worker is good at.

The dispatch layer is just a switch over JSON-RPC methods. Here is the shape of it:

async function dispatch(req: JsonRpcRequest): Promise<unknown | null> {
  switch (req.method) {
    case "initialize":
      return {
        protocolVersion: "2025-03-26",
        capabilities: { tools: {} },
        serverInfo: { name: "casebook-mcp", version: "0.1.0" },
        instructions:
          "Use similar_failures when investigating an incident, " +
          "search_cases for topical research, get_case for full detail.",
      };
    case "tools/list":
      return { tools: TOOLS };
    case "tools/call": {
      const name = String(req.params?.name ?? "");
      const args = (req.params?.arguments ?? {}) as Record<string, unknown>;
      return callTool(name, args);
    }
    default:
      if (req.method.startsWith("notifications/")) return null;
      throw { code: -32601, message: `Method not found: ${req.method}` };
  }
}

Notifications (JSON-RPC requests with no id) get swallowed and produce no response body, which is what the spec wants. The request handler also accepts batched arrays, so a client can pipeline initialize and tools/list in one POST.

The ranking logic lives in its own pure module with no I/O, which makes it trivial to unit test. Scoring is deliberately simple and legible: a query token hitting a case title weighs 3, a tag hit weighs 2, and a body hit weighs 1, with ties broken by case number for stable ordering.

export function scoreCase(c: CaseFile, tokens: string[]): number {
  if (tokens.length === 0) return 0;
  const title = new Set(tokenize(c.title));
  const tags = new Set(c.tags.flatMap((t) => tokenize(t)));
  const body = new Set(tokenize(caseText(c)));
  let score = 0;
  for (const tok of tokens) {
    if (title.has(tok)) score += 3;
    else if (tags.has(tok)) score += 2;
    else if (body.has(tok)) score += 1;
  }
  return score;
}

similar_failures uses a related but even simpler measure: tokenize the description, tokenize each case, and rank by the size of the keyword intersection. No embeddings, no vector store.

Data comes from the live public endpoints on agentpostmortem.com (/api/export for the corpus, /api/search for rich detail, /api/tags) behind a 5 minute in-memory cache. When the network is unavailable, it falls back to a bundled dataset of representative case files that ships in the repo and doubles as the deterministic fixture for tests. So the server keeps answering even offline, just from a smaller snapshot.

There is also a companion investigator agent (agent/investigate.ts) built on the Claude Agent SDK query() API. Given an incident description it connects to the MCP server, finds similar failures, pulls the top cases with get_case, and writes a postmortem-draft.md grounded in the documented lessons. It runs on your local Claude Code subscription auth, so no API key appears in the code, and it has a --dry-run mode that stubs the model but still exercises the MCP server end to end, which is what CI uses.

An honest limitation

The similarity matching is pure keyword overlap, not semantic search. If your incident description and a documented case describe the same failure in different vocabulary, for example one says "loop" and the other says "recursion", the overlap can miss it. That is a conscious tradeoff: keyword ranking is transparent, dependency-free, and fully unit-testable, and it keeps the Worker cold-start cheap with no vector database to stand up. For a corpus of curated, tagged case files it works well in practice, but I would not claim it generalizes to fuzzy paraphrase the way an embedding index would. Adding an optional embedding pass is the obvious next step if recall becomes the bottleneck.

The other honest note: the rate limit is a simple in-memory per-IP counter (60 requests per minute) scoped to a single Worker isolate. It resets when the isolate recycles. That is intentionally lightweight for a public read-only endpoint, not a hardened quota system.

Try it

Add it to Claude Code with a single command once it is running:

claude mcp add --transport http casebook http://localhost:8787/mcp

Then your agent can consult a growing body of real agent failures instead of rediscovering each one from scratch. Code, the four tools, the investigator agent, and the bundled dataset are all here: github.com/AgentPostmortem/casebook-mcp.