DEV Community

TECHDEV
TECHDEV

Posted on Originally published at github.com

AI assistants cite fake stats with total confidence. I built a 10-second checker.

#ai #cli #opensource #python

Last week I read a Hacker News thread about someone publishing fake statistics on a throwaway site just to watch LLMs cite them as fact. 295 points, 168 comments — and it hit a nerve because I've lived the milder version of this: an AI assistant hands me a draft with a confident "73% of developers…" plus a link, and I have no fast way to check whether the link actually says 73%.

The existing citation checkers I found mostly answer a different question — "does this reference exist?" — aimed at academic bibliographies, usually as paid web services. I wanted something narrower and local: does the cited page contain the actual numbers? So I built it.

Meet factfence

factfence is a zero-dependency Python CLI. Point it at any markdown or text file — a blog draft, a doc, an AI-generated report — and it:

  1. Extracts every quantitative claim (percentages, money, large counts, multipliers, ratios), skipping code blocks so retries = 99 never becomes a "claim".
  2. Fetches each unique citation once (polite user-agent, 8s timeout, 1.5MB cap) and reduces the page to normalized plain text.
  3. Reports one of five verdicts per claim:
  • SUPPORTED — the cited page contains the numbers
  • UNVERIFIED — the page loads but doesn't contain them (the classic AI hallucination shape)
  • BROKEN — every cited link failed
  • UNCITED — a hard number with no citation at all
  • CITED — a "studies show…" assertion with a working link, nothing numeric to cross-check

Real output on a sample AI-written paragraph:

[✓ SUPPORTED]
  According to the 2026 State of DevTools Survey, 73% of developers say AI-
  generated code needs human review before merging
    ↳ http://127.0.0.1:18347/source-a.html
      HTTP 200, matched 73%

[? UNVERIFIED]
  Some teams claim 99.9% of AI suggestions are production-ready, citing an
  internal study (http://127.0.0.1:18347/source-b.html).
    ↳ http://127.0.0.1:18347/source-b.html
      HTTP 200
    cited page(s) fetched but contain none of the claim's numbers

[! UNCITED]
  It is often repeated that 60% of all code will be AI-written by 2027,
  though nobody cites a source.
    quantitative claim with no citation — verify by hand

6 claim(s) checked: 2 supported, 1 cited, 1 unverified, 1 broken, 1 uncited. 3 need review.
Enter fullscreen mode Exit fullscreen mode

Exit codes are CI-friendly (0 clean, 1 needs review, 2 error), and --json gives you machine-readable output for pipelines.

Try it 

git clone https://github.com/techdevfun/factfence.git
cd factfence
python3 -m factfence your-draft.md
Enter fullscreen mode Exit fullscreen mode

No pip install needed — it's stdlib only, Python 3.10+.

What it deliberately doesn't do

It checks citation support, not truth. A SUPPORTED claim can still be wrong if the source is wrong — the tool just catches the far more common failure of confident numbers with dead, irrelevant, or missing citations. Number matching is textual ("73%" vs "nearly three quarters" won't match), pages behind logins can't be fetched, and it's English-focused. All of this is documented as limitations in the README rather than hidden.

If you publish anything an LLM helped write, run your draft through it once. The three sentences it flags are exactly the three worth re-reading.

Repo: https://github.com/techdevfun/factfence — MIT licensed, issues and PRs welcome.

Top comments (0)