DEV Community

Artem KK
Artem KK

Posted on

Building an MCP Server That Verifies Its Sources: Inside footnote-mcp

footnote-mcp is a Python MCP server installable via pip, Docker, or pipx. No API keys required — it falls back to scraped Bing + DuckDuckGo search and automatic headless Chromium for JavaScript-heavy pages.

The Verification Pipeline

The core tool is evidence_entailment. It takes a claim and a source text, and returns whether the claim is supported, unsupported, or contradicted. The heuristic backend extracts numeric and named-entity tokens from both the claim and source, then checks for exact matches and contradictions. On its design domain — numeric and factual data claims — it achieves 100% accuracy on a labeled benchmark set.

For semantic cases (negation, paraphrase), the ollama backend uses a local LLM as a judge.

Three tools build on this: corroborate_claim triangulates a claim across multiple sources, locate_claim_span finds the exact supporting sentence with character offsets, and build_research_debug_report produces a compact report of queries, URLs, and verification gaps.

The Fetch Ladder

web_read fetches pages through a 5-tier escalation ladder: HTTP (curl_cffi) to rotating proxy to headless Chromium to Chromium through proxy to hosted scrape API (Firecrawl/ScrapingBee). A block/quality detector decides when to escalate, and per-domain rate limiting, circuit breakers, and negative cache keep it polite.

Search Backends

web_search supports Tavily, Brave, Google, or scraped Bing + DuckDuckGo as fallback. Pass semantic: true to reorder results by meaning using local Ollama embeddings.

Structured Data and Browser Tools

Beyond text, the server handles tables, CSV/XLSX/PDF/JSON, date validation, unit resolution, and time series reconciliation. For JavaScript-heavy pages, 10 browser tools let you drive a headless Chromium session. When generic parsers fail, the server can synthesize sandboxed extraction code through a controlled recipe system.

Benchmark Results

The heuristic backend achieves 100% accuracy on numeric and factual data claims (n=15). Overall accuracy including semantic cases is 83% (n=18). The blind spot is purely semantic negation and paraphrase — the ollama backend closes that gap.

Try It

pip install footnote-mcp
python -m playwright install chromium
footnote-mcp
Enter fullscreen mode Exit fullscreen mode

Then add to your MCP client config and start researching. Full source at github.com/KazKozDev/footnote-mcp.

Feedback welcome — especially on the verification approach. The benchmark is designed to be extended, so if you have a claim/source pair that should be caught, open an issue or PR.

Top comments (2)

Collapse
 
mads_hansen_27b33ebfee4c9 profile image
Mads Hansen

Promising separation of retrieval and entailment. I would make every verdict carry an evidence receipt: exact supporting span, URL, fetch time, content hash, backend/version, and an explicit abstain state when extraction or source quality is uncertain. I would also keep source authority separate from entailment; a page can perfectly support a claim and still be wrong. With n=15, the 100% result is best treated as a seed regression set. Adversarial cases around units, date scope, table headers, quoted contradictions, and stale pages would reveal where the heuristic boundary actually sits.

Collapse
 
hannune profile image
Tae Kim

The fetch ladder and verification pipeline are independent but their failures interact in a way worth tracking explicitly: a circuit-breaker trigger or negative cache hit on a domain means claims citing that domain are quietly unverifiable rather than verified-pass. Unless you surface a "verification-skipped-fetch-failed" state alongside supported/unsupported/contradicted, a fetch infrastructure failure on an unreliable source looks identical to a clean verification pass. The abstain state covers uncertainty in extraction and source quality, but fetch errors are a third confidence profile that sits before extraction even starts — and a malicious or unreliable domain is most likely to produce repeated fetch failures. The locate_claim_span tool already attaches failure provenance to the verification layer; the same pattern at the fetch layer would make the gap visible.