How I stopped my research agent from inventing its sources

Deepak Rode — Thu, 29 Oct 2026 14:20:48 +0000

I build research agents: you give one a topic, and it searches the web, writes a short brief, and puts a link after every claim. The output looks trustworthy. Every sentence has a source.

Then I started actually opening the links.

The source often said something slightly different. A different number. A different year. "Experimental" where the brief said "default". Not invented out of thin air, just drifted: the model had read the page, then filled in a detail from memory while writing.

Here is the example I now use as a test. A short piece about Python releases, six claims, every one linked to an official python.org page:

The text said	The linked page actually says
Python 3.12 runs about 25% faster than 3.11	"an estimated 5% overall performance improvement"
distutils was deprecated since Python 3.8	"In Python 3.10 and 3.11, distutils will be formally marked as deprecated"
Python 3.13 runs without the GIL by default	"experimental feature and therefore is not enabled by default"

Three of six claims misstate their own sources. The links are right. The details aren't.

Where the fact-checker sits

The agent is split into a coordinator and three specialists:

Researcher: search + fetch. Its only output is a citations file.
Writer: turns the citations into a draft. No web access at all.
Fact-Checker: can fetch pages but can't search.
Grader: scores the brief against a rubric and sends it back for revision, at most 2 rounds.

The Fact-Checker having no search tool is deliberate. If it could search, it would "verify" a claim by finding some page that agrees. Without search, it can only check what was actually cited.

The checker reads each linked page and marks every claim verified, partial or unverified, and it has to copy the supporting sentence from the page as evidence.

That sounded like enough. It wasn't.

Problem 1: the quote can be invented too

After I shared the first version, someone asked the obvious question: what stops the model from inventing a plausible quote? My script only checked that a quote existed in the report, not that it was really on the page.

He was right. Asking a model to "quote its source" isn't a safeguard; it's another thing the model can get wrong.

So in v1.1 the report script downloads every cited page itself and checks the quote against it. The matching ignores case, spacing and punctuation (curly quotes, dashes, line breaks), but every word and number has to match:

import re, unicodedata

def squash(text):
    """Lower-case and drop spaces and punctuation, so formatting differences
    don't matter, but every word and number still has to match."""
    text = unicodedata.normalize("NFKC", text).lower()
    text = re.sub(r"(?<=\d)\.(?=\d)", "DOTX", text)  # keep 3.12 and 31.2 different
    text = re.sub(r"[^\w%$]", "", text)
    return text.replace("DOTX", ".")

def quote_on_page(quote, page_text):
    """True only if every part of the quote is really on the page."""
    page = squash(page_text)
    parts = [squash(p) for p in re.split(r"…|\.\.\.", quote)]
    return all(p in page for p in parts if p)

If a quote isn't on the page, the report refuses to build. If a page can't be downloaded (paywall, JavaScript-only page, blocked site), the claim is marked not machine-checked instead of quietly passing.

Testing this on my own example immediately caught a bug of mine: python.org sends its pages gzip-compressed, and my first version couldn't read them. So every quote "failed". A check that has never failed on real data hasn't been tested.

Problem 2: a real quote with the wrong number

The same person found the next gap. The quote can be genuinely on the page and still not support the claim. The most common way: the number is different. A real quote saying "5%" attached to a claim of "25%".

So v1.2 adds a numbers check: every figure in a "verified" claim must also appear in its quote.

It took some care to make that useful rather than noisy:

commas are ignored (Indian-format 1,29,940 = 129,940)
rounding is allowed to the claim's precision (claim 2.2 matches quote 2.17)
small whole numbers are skipped ("3 companies", "2 October" caused false alarms)
labels like Q1 or FY27 are skipped for now, for the same reason
"525k" counts as 525,000.

Testing it on a real market brief taught me two more things. Many quotes came from company results PDFs, so the checker now reads PDFs. And one PDF had two quoted sentences in the opposite order from how the model quoted them (PDF layout scrambles order), so quote fragments are now accepted in any order.

The report now shows these as separate lines: "quote found on page" and "figures in quote". Matching figures can stop a claim from being verified, but they can never make it verified on their own.

What it still can't catch

When I posted this on Reddit, three different people pointed at the same remaining gap: the quote is real, the number matches, but it's about something else. Last year's figure sitting next to this year's. The right percentage, but for a different population.

Some of that is covered: a claim of "25% in 2025" fails against a quote saying "25% in 2024", because 2025 isn't in the quote. But "FY27" vs "FY26" passes (labels are skipped), and "25% of adults" vs a quote about teenagers passes too. Whether the quote supports the exact sentence is still the model's judgement, and the report says so openly.

One commenter shared a useful data point: a small model he measured scored about 80% on an open-ended review that reconciled stale or conflicting evidence, against 98–100% on narrow decisions with a fixed set of answers. That suggests the next step: don't ask one open question ("does this quote support the claim?"). Split it into narrow yes/no checks (same period? same metric? same population?) and let a "no" on any of them block "verified".

What I took from it

"The model must cite a quote" is not a safeguard until code checks the quote. Most of the reliability came from what the scripts refuse, not from the prompt.
Give each agent only the tools its job needs. The fact-checker without search was the single biggest improvement.
Say what isn't checked. "Not machine-checked" and "this part is the model's judgement" are more useful to a reader than a confident green tick.
Test on real documents, not just unit tests. Every real brief found a new failure: gzip, PDFs, reordered quotes.

Try it

I packaged the fact-checking part as a free Claude skill called claim-checker: give Claude any text with links (an AI-written report, a blog draft, a client deliverable) and it returns a claim-by-claim report with the evidence. The ideas above work in any framework; the skill is just one way to use them.

It's MIT licensed, with tests and the Python example above:
https://github.com/deeprdai21-ai/practical-claude-skills

I'd like to hear how others handle this in their own research agents. Do you verify quotes in code, use a second model as a checker, or something else?