An MCP server that has to prove itself: citations, abstention and entitlements as CI gates

The demo, in one screen

Here's the eval scorecard for a small MCP server on a green build:

retrieval   (staff, n=22):  hit@1 1.00   recall@5 1.00   MRR 1.00
abstention  (n=10):         rate  1.00   (target 1.00)
leakage     : 0 (must be 0)
redaction   : 0 (must be 0)
EVAL GATE: PASS

Now I weaken one thing. Two environment variables drop the server's relevance bar to
zero, the setting where it returns its best guess for every question instead of
admitting defeat:

abstention  (n=10):         rate  0.40   (target 1.00)
  FALSE-ANSWER  'customer refund policy' -> internal/hr/expenses-policy.md (score 4.29)
  FALSE-ANSWER  'office dog policy'      -> internal/hr/expenses-policy.md (score 4.29)
EVAL GATE: FAIL

Exit code 1. The build is red. Nobody had to notice the server had started making
things up, because the pipeline noticed first.

Why bother?

The Model Context Protocol has made it absurdly easy to hand an agent your knowledge
base. There are hundreds of servers that expose notes, wikis and docs as searchable
tools, and most of them work fine as plumbing. But before you let an agent answer
questions out of your company handbook, there are three questions the plumbing never
answers.

Can you trust what comes back? When the server returns a snippet, can the agent cite
where it came from, precisely enough that a human can check?

Does it know when it doesn't know? Ask a vault about a policy that isn't in it and
BM25 will cheerfully return the closest-sounding note anyway. The agent, primed to
trust its tools, turns that into a confident answer about a document that does not
exist.

And who is allowed to see what? If your vault has a restricted/ folder, "the search
tool filters those results out" is not an answer. It's the start of an incident report.

So I built grounded-mcp: a small Python
MCP server over a folder of markdown (Obsidian works as-is) with three guardrails.
Every content-bearing response carries a stable citation id, path plus heading plus
line span. Search abstains, explicitly and structurally, when nothing clears the
relevance bar. And entitlements are enforced at index level: content a profile can't
see is never indexed for it in the first place.

None of which would be worth writing about, except for one decision: the repo ships
with its own eval suite, and the suite is the CI gate. That decision paid for itself
on the first run.

The first honest run found a real bug

The eval suite runs four families against a committed demo vault (a fictional company
handbook with public, internal and restricted zones): a golden retrieval set, a set of
questions whose answers are deliberately not in the vault, a leakage set answerable
only from restricted content, and a redaction check against a seeded fake credential.

First run: retrieval perfect, leakage zero, redaction zero. Abstention: 0.30.
Seven of ten unanswerable questions came back with confident wrong answers.

The interesting part is why. I had assumed a BM25 score threshold would separate real
answers from plausible noise. The data said no:

true answers:    scores 2.44 to 14.68
false answers:   scores 2.12 to 4.29

The ranges overlap. "How long is probation" scores 2.44 against the note that
genuinely answers it, while "customer refund policy" scores 4.29 against an expenses
policy that says nothing about refunds. One common word landing in a heavily weighted
title field looks exactly like relevance. No threshold exists that keeps the first and
rejects the second.

What does separate them is coverage: what fraction of the query's content words the
note actually contains. The false answers ride on one matching word ("policy") and
their coverage collapses once you stop counting scaffolding like "what is the". So
abstention became a double gate: score above threshold AND coverage above half. That
took the rate from 0.30 to 1.00 with zero retrieval loss.

Then the gate caught two more bugs, in my fix. Naive substring matching decided
"code" doesn't appear in a note that only says "coding", and a hyphenated
"AI-generated" hid both of its parts from a query about AI. Light suffix-stripping and
splitting compounds fixed both, and I know the fixes didn't break anything else
because 24 golden queries and the leakage set re-ran on every attempt. The eval suite
was the code review.

Leakage is a ranking problem, not a filter problem

The entitlements design deserves one more paragraph, because the obvious
implementation is quietly wrong. Most access control on search works by filtering
results: retrieve everything, then drop what the caller shouldn't see. That leaks. A
filtered result still influenced rankings. Its terms still matched. In some systems
the result count still moves, and now your search tool is an existence oracle for the
documents you hid.

grounded-mcp builds a separate index per entitlements profile, containing only the
notes that profile may see. Denied content is never scored, so there is nothing to
filter and nothing to leak. The eval suite's leakage family checks this at the
harshest setting, thresholds off entirely, and any hit from a denied path fails the
build. Same principle for direct reads: a denied note and a nonexistent note return
byte-identical responses.

What it doesn't do

Honesty section. Over stdio, client and server run as the same user, so profiles
demonstrate the deployment pattern rather than defend against a hostile peer; real
per-client enforcement arrives with the HTTP transport in v0.2. Retrieval is plain
BM25, deliberately: it's deterministic, dependency-free and measurable, and when
hybrid semantic retrieval lands it has to publish its eval delta against this
baseline or it doesn't land. The redaction patterns catch key-shaped strings, not
every secret. And there are no write tools. A server that can quote your vault but
never rewrite it is a trust feature, not a gap in the roadmap.

Try it

The repo is github.com/rich-atkins/grounded-mcp,
MIT licensed. pip install -e ., point GROUNDED_VAULT at your notes, wire it into
Claude Code or Claude Desktop with a five-line config. Run the evals, then run the
sabotage demo and watch the gate go red.

Better still: write ten questions your vault can't answer and add them to the
abstention set. If your knowledge server passes that test today, you're doing better
than mine was on Friday morning.