Your MCP Server Returns the Source. Does the Answer Still Cite It?

A benchmark for what happens to citations after a retrieval server hands its results to an AI
agent it doesn't control.

Part 1 of 6 in the series Citations at the MCP Boundary. This is where the series starts. The parts after this one go deeper into each finding; the full list is at the end of this post.

Here's a situation that's becoming normal very quickly.

A company turns its documentation into a search tool. It exposes that tool over MCP (the Model Context Protocol), so any agent can use it: Claude, ChatGPT, a coding assistant in someone's IDE, an agent a customer built with LangGraph or the OpenAI Agents SDK. The tool does its job well. For every query it returns the most relevant chunks of documentation, and each chunk comes with its source URL:

{
  "source_url": "https://fastapi.tiangolo.com/tutorial/body-updates/",
  "content": "# Body - Updates ..."
}

Then the agent takes over. It reads the chunks, decides what matters and writes the final answer.

The retrieval server never sees that answer. So the provider can be sure the right source went in, but has no idea whether a citation comes out.

For a documentation assistant that's a real problem. An answer about your API that's correct but uncited can't be checked, and the user can't click through to the page that explains the edge cases. An answer that cites the wrong page is worse: it looks verified when it isn't.

I couldn't find anyone measuring this, so I built a benchmark for it.

The question

When a RAG knowledge base is consumed by an AI agent over MCP, instead of a single retrieve-then-answer call, do the citations survive?

The obvious hypothesis was that agents lose citations. An agent loops: it searches, reads, maybe searches again, and then writes. The sources from the first call end up far back in the context by the time the answer is written. It seemed natural that they'd get lost along the way.

That hypothesis turned out to be mostly wrong, and the way it was wrong is more useful than the hypothesis itself.

The setup

I built a complete, reproducible copy of this boundary:

• The corpus: the FastAPI documentation, 153 pages split into 2,085 heading-aware chunks. FastAPI has clean Markdown, stable public URLs and is widely used, so anyone can check the work.
• The MCP server: one tool, search_fastapi_knowledge_sources, returning exactly {source_url, content} over streamable HTTP. It mirrors the contract of a real hosted MCP retrieval product, down to the naming pattern.
• The questions: 250 drafted, 127 kept after I checked every one by hand against the actual docs page: 82 answerable from one page, 29 that need two or more pages, and 16 the docs don't cover at all (to catch made-up answers).
• Three pipelines, held as equal as possible: the same model (gpt-4o-mini), the same docs, the same system prompt and the same required citation format ([url] right after each claim):
  1. Single-shot RAG. Search once, then answer. This is the control.
  2. A LangGraph agent. A ReAct agent that decides when and how often to call the MCP tool.
  3. An OpenAI Agents SDK agent. The same idea, built with OpenAI's agent framework and its built-in MCP support.

The metric: Citation Survival Rate

Measuring "does it cite" turned out to need more care than I expected. An answer can be full of URLs and still be badly attributed. It can cite a real page that doesn't support the sentence, or cite a plausible FastAPI URL the agent never actually retrieved (models know what FastAPI URLs look like).

So CSR (Citation Survival Rate) requires every citation to pass two checks:

1. The support check. Does the cited chunk actually support the claim? Each answer is split into small atomic claims (the FActScore approach), and an entailment model checks each claim against the chunk it cites (the ALCE approach).
2. The trace check. Was that URL really in what the agent retrieved? Every tool call is logged, so a citation to a page the agent never saw is caught as fabricated.

CSR = of the correct sources the agent actually retrieved, the fraction that end up in the final answer, cited on a claim the source genuinely supports.

Notice the condition: the sources the agent actually retrieved. If the agent never found the right page, that's a retrieval failure, and CSR leaves it out. CSR only measures what happens after the right source has crossed the boundary.

The results, and the mistake I almost published

Here's what the metric said on the first pass:

Pipeline (gpt-4o-mini)	CSR	95% CI
Single-shot RAG	0.618	[0.528, 0.708]
LangGraph agent	0.614	[0.524, 0.705]
OpenAI Agents SDK agent	0.323	[0.237, 0.414]

The LangGraph agent matched single-shot RAG (a difference of +0.000 question by question), so an agent loop by itself didn't cost citations. The OpenAI Agents SDK agent seemed to lose about half: in 66 of its 127 answers there was no citation in the [url] format the prompt asked for.

I took that at face value for longer than I should have. When I finally read those 66 answers one by one, most of them did cite their sources:

• 51 cited the URL right after the claim, but as a Markdown link: [source](https://...)
• 7 used another form, like [source: https://...]
• only 8 had no URL at all (single-shot RAG has 19 of those)

My scorer only recognised the requested [url] format, so it counted every one of those Markdown links as a missing citation. I fixed the definition (written down before re-scoring: [url], Markdown links and the other bracket format all count) and re-scored everything:

Pipeline (gpt-4o-mini)	Only the requested format counted	Any citation format counted
Single-shot RAG	0.618	0.618
LangGraph agent	0.614	0.643
OpenAI Agents SDK agent	0.323	0.470

Counted in any format, the SDK's remaining gap to single-shot on that run was −0.16 under the original scorer (significant) but −0.10 under a gpt-4o judge (not significant). One run couldn't settle it, so I ran every setup three times. A smaller loss held in every run and under both
scorers: −0.12 (original scorer) and −0.09 (gpt-4o judge), pooled, against about −0.30 counting only the requested format. So the drop is about 60% format and 40% a real, smaller loss of citations. At temperature 0 no loss is detectable.

The bigger and more stable effect is simpler:

The OpenAI Agents SDK, on its default API, ignored the requested citation format in 51 of 127
answers (40%), and 53 and 46 in the two repeat runs. Single-shot RAG did that 0 times; the
LangGraph agent 8 times.

That matters more than it sounds. Anything downstream that reads citations in a fixed format (a docs widget, a "view sources" panel, a guardrail, an eval like mine) sees nothing at all. The citation is gone as far as the system is concerned, even though a human reading the text could find it.

Why? Not the reason I expected

Part 2 has the full investigation. The short version: I ran five controlled experiments on the SDK agent, changing one thing at a time, with the plan written down before running anything:

• Not the agent loop. LangGraph loops too and kept the format. The agents also barely looped: about 1.2–1.3 searches per answer.
• Not temperature, at least not alone. The SDK pipeline had never set a temperature (LangGraph used 0). Matching it barely changed the format drift (60 → 54 answers without [url]).
• Not the transport. I dumped exactly what the model receives through the SDK: every chunk arrives with its source_url intact, and 98 of the 118 Markdown links point to pages the agent retrieved. The sources make it across the boundary.
• The API. The OpenAI Agents SDK calls OpenAI's Responses API by default; LangGraph's OpenAI integration uses Chat Completions. Switching the same SDK agent to Chat Completions cut the wrong-format answers from 41 to 9. A published system prompt for grounded answers cut them to 20. Both held across three runs. Neither made a significant difference to whether sources were cited at all (measured on one run): they fix the format.
• A fix on the server side. Both of those live in the agent, which a retrieval provider doesn't control. So I tested one the server can make: tag every chunk with a short source ID ([S3fa21c]), ask for [ID] citations, and map IDs back to URLs after the answer is written. On my self-hosted mirror, wrong-format answers fell from about 50 to about 5 per run, the SDK agent's CSR rose by 0.10–0.13, and its gap to single-shot closed. (Part 6 has the numbers and the caveats, including that no human has audited those citations yet.)

Can you trust these numbers?

A benchmark is only as good as its scorer, so I spent as long checking the measurement as running the experiment:

• Two kinds of scorer. An entailment model and an LLM judge (gpt-4o), chosen by benchmarking checkers against 115 of my own blind judgements; the judge agreed with me best (κ 0.86). Where they disagree, I say so.
• Checked against a human. On 30 blind cases, CSR's verdicts agreed with mine on 25 to 27, depending on the scorer. (Two of those cases had the same Markdown-link blind spot, which is part of how I found it.)
• Repeated. Every gpt-4o-mini setup ran three times, with a robustness rule written down first: same sign in every run, and a pooled interval that excludes zero.
• Audited before publishing. Going through the pipeline as a hostile reviewer would, I found five bugs, four of which would otherwise have shipped. (That story is Part 4.) The format issue above is the sixth, and the one I'm least proud of: I'd already sent the overstated number to people before I caught it, and had to correct it.

And the limits, stated plainly: one corpus, two models, three runs per gpt-4o-mini setup (one for the Claude runs), small human checks, and a self-hosted mirror of an MCP server rather than anyone's production system.

What I'd take away from this

1. If you serve RAG through MCP, your accuracy guarantee stops at the boundary. You control the retrieval. You don't control the agent that writes the answer. If citations matter to your product, measure what happens on the other side.
2. Framework defaults are part of your system. Two agents with the same model and the same prompt followed the citation format very differently because of one default setting a developer might never look at.
3. Measure format and presence separately. "No citation in the format I parse" and "no citation" are different failures with different fixes. Count both.
4. Read the outputs, not just the metric. A count of "uncited answers" hid the fact that most of them did contain the source. Reading 66 answers changed the conclusion, and should have happened before I told anyone the number.
5. Run it more than once. One run of the remaining gap could land anywhere from −0.05 to −0.16. Three runs showed it was real, and smaller than it first looked.
6. The retrieval side can fix the format. Give the agent a short source ID to copy instead of a long URL to reformat, and map it back yourself.

---

The series: Citations at the MCP Boundary

1. Your MCP Server Returns the Source. Does the Answer Still Cite It?
2. One Agent Framework Seemed to Lose Half Its Citations. Most Were Just Reformatted.
3. When Your Eval Grades Its Own Homework
4. I Audited My Own Benchmark Like a Hostile Reviewer. It Had Five Bugs.
5. Before You Measure an Agent, Fix Your Corpus: 20% of My Chunks Had No Code
6. Serving RAG Over MCP? A Practical Playbook for Keeping Your Citations

Next: Part 2: One Agent Framework Seemed to Lose Half Its Citations. Most Were Just Reformatted.

A new part is published every week.

Repo, traces and every number: csr-mcp-benchmark

I'm an AI engineer who builds production RAG and agent systems and the evals that prove they work. I'm looking for a remote AI engineering role at a US or EU startup. If your team is building with RAG, agents or MCP, I'd love to talk.