claimcheck: Gemma 4 fact-checks my AI blog drafts against their own sources

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

I run CoreMesh, a small tech blog where an AI pipeline writes the first draft of every article and I approve or reject it. That is how a sentence like this got into a draft about upgrading to Python 3.15:

The stdlib now returns frozendict from EnumType.__members__ and errno.errorcode instead of MappingProxyType or dict.

It reads like a fact, but it isn't one. PEP 814 lists those two only as possible candidates for frozendict in the stdlib. A reader who trusted that line would write code that breaks.

So I built claimcheck: a small CLI that checks an AI-written draft against the sources the draft itself links to, using an open-weight model (Gemma 4), before anything goes live.

Who would pay for it: anyone who publishes AI-assisted writing under their own name and can't afford to be wrong in public:

• solo bloggers and newsletter writers;
• small content agencies;
• docs and DevRel teams;
• people selling technical courses.

The tool itself is MIT. What people would pay for is the service around it: a pre-publish check per article, a CI gate on a docs repo, or a weekly re-check of old posts as the docs they cite change.

Demo

A real run on that Python 3.15 draft, with the bad sentence put back in:

$ python3 claimcheck.py examples/python315-draft.md --exclude coremesh.online
28 claims, 4 sources
  [1/28] supported    Python 3.15 uses UTF-8 as the default text encoding for files, stdio and pipes.
  ...
  [18/28] not_found    In Python 3.15, `EnumType.__members__` and `errno.errorcode` return `frozendict`.

The report puts problems first and shows the draft's words next to the source's words:

## 5. [CHECK] In Python 3.15, `EnumType.__members__` and `errno.errorcode` return `frozendict`.
> Draft: "The stdlib now returns `frozendict` from `EnumType.__members__` and `errno.errorcode` instead of `MappingProxyType` or `dict`"

## 7. [OK] Python 3.15 uses UTF-8 as the default text encoding for files, stdio and pipes.
> Draft: "Python now uses UTF-8 as the default text encoding for files, stdio and pipes"
> Source (https://peps.python.org/pep-0686/): "With this change, Python consistently uses UTF-8 for default encoding of files, stdio, and pipes."

The exit code is 1 when anything needs a human, so it can block a publish step or a CI job.

Code

indra112f / claimcheck

Fact-check AI-written drafts against their own sources with an open-weight model (Gemma). Evidence quotes are verified word for word.

claimcheck

Fact-check an AI-written draft against its own sources with an open-weight model, before you publish it.

$ claimcheck examples/python315-draft.md --exclude coremesh.online
28 claims, 4 sources
  [1/28] supported    Python 3.15 uses UTF-8 as the default text encoding for files, stdio and pipes
  [18/28] not_found    In Python 3.15, `EnumType.__members__` and `errno.errorcode` return `frozendict`.

That last line is a real mistake our blog's AI writer made: PEP 814 lists those two as possible candidates, not as shipped changes. claimcheck flagged it.

How it works

1. Claims. The model lists the checkable claims in the draft: numbers, versions, limits, defaults, dates, "X supports Y". Every claim must quote the draft. A claim whose quote isn't in the draft is dropped.
2. Retrieval. Every source page linked in the draft (plus any --source) is fetched and split into passages. A small IDF-weighted lexical index picks the closest passages for each claim, with no…

View on GitHub

It's one file, standard-library Python, with offline tests that use a fake model.

How I Built It

The pipeline has four steps. The interesting part is that the model is never trusted on its own word.

1. Extract claims. Gemma 4 lists the checkable claims in the draft (numbers, versions, limits, defaults, "X supports Y") and has to quote the draft for each one. If the quote isn't actually in the draft, the claim is dropped.
2. Retrieve. Every linked source page is fetched and cut into overlapping passages. A tiny IDF-weighted lexical index picks the closest six passages per claim. There are no embeddings and no vector DB. Rare tokens like load_module or 3.15 carry the weight, which is what you want for technical claims.
3. Judge. Gemma sees one claim and those passages only, and must answer supported, contradicted or not_found and copy one evidence sentence word for word.
4. Verify the verifier. This is plain code, not the model:
   • Quote check: the evidence sentence must exist in the fetched page (after normalizing quotes, dashes and whitespace). If it doesn't, the verdict is thrown away.
   • Number check: every number in the draft's sentence must appear in the evidence or in the text around it. I added this after a real run marked "you'll need LLVM 21" as supported with a quote that never mentions 21.

Model and infra: gemma-4-31B-it on DigitalOcean serverless inference, through the OpenAI-compatible endpoint, with temperature: 0. A full article took 3 to 5 minutes and about 70k input tokens. At DigitalOcean's listed Gemma 4 serverless rate ($0.18 per 1M input tokens and $0.50 per 1M output tokens, as of October 2026), that is about 1.4 cents per article.

I built it in one sitting with an AI agent (Claude) doing most of the typing while I steered. That fits the project: an AI wrote it, and the tests and the quote check are why I trust it.

Why Does Open Innovation Matter?

• The unit economics only work with cheap inference. A check is around 30 model calls. At about 1.4 cents per article, I can run it on every draft and every old post, every week, and still charge a few dollars a month. That's a business.
• The judge shouldn't be a black box I rent. A fact-checker whose model changes behind an API without notice is a liability. With open weights I can always run the exact same model myself, so the verdicts can be reproduced and audited.
• Client drafts are private until they're published. The tool talks to any OpenAI-compatible endpoint, so the same CLI can point at Gemma running locally with Ollama and the draft never leaves the machine. For an agency checking a client's unpublished piece, that's a hard requirement. I measured the numbers here on DigitalOcean only.
• No lock-in. Swapping Gemma for another open model is one flag. If a better open checker ships next month, I switch the same day.

My First User

My first user is my own blog, so here's what it found, good and bad.

• The planted mistake: flagged, as shown above.
• A false alarm, and why the report format matters. In one run it marked "all four models have a 1M-token context window" as contradicted, quoting Anthropic's pricing page: "Claude 4.6 and later models (except Claude Haiku 5.5) … include the full 1M token context window at standard pricing." I checked the models overview: all four list 1M tokens. The draft was right; that sentence is about price, not size. Because the report shows the draft's words next to the source's words, it took me about 30 seconds to see it. A later run left the same claim as "not found", so the model isn't fully deterministic even at temperature 0, which is one more reason a human makes the final call.
• My best-ranking article ("Claude Code usage limit reached", page 1 on Google) got 0 of 12 claims backed by its sources. It links only to documentation home pages. Re-run against the right support and docs pages, 3 of 10 were backed: the 5-hour and weekly limits, the limits being shared with chat, and /usage. Most of the rest are my own measurements (build and review times) and a description of my own product, which no doc can confirm, and flagging them is correct. The fix there is better links, not a better model.

Nobody has paid yet. The next step is to offer a free check of one article to three bloggers who publish AI-assisted posts, and ask whether they'd pay for it weekly.

Prize Categories

• Best Use of Gemma: Gemma 4 is both the claim extractor and the judge.
• Best Use of DigitalOcean: it runs on DigitalOcean serverless inference.

---

This post and claimcheck were written with an AI agent (Claude) working under my direction. Every number above comes from real runs, and the reports are in the repo.

Comments

[CoreMesh]

Before I built this tool, I was checking AI-written articles by hand, and it was exhausting. How do you usually check AI-written content?