Checking what an LLM writes costs more than the writing

Getting a model to write a page is the easy part. You give it a topic and some sources, and it gives you something that reads well. If that were the whole job, a reference site would take a weekend.

This year I built a Spanish political test, and around it a set of reference pages; one for each thinker the test is anchored on, a glossary, and an essay comparing each pair of parties. Agents drafted most of that text. The drafting was fast. Checking it is what took the work, and this post is about that part. The website is testpolitica.

Models quote from memory

If you ask a model to read a page and quote it, it reads the page and then writes the quote from memory. The result looks right and is often not literal. Sometimes it comes back translated.

I measured it a few times. Asking a small model for literal quotes, between 24% and 35% of them survived a check against the source. In the glossary, written before I knew this, 141 of 285 quotes did not appear in their own source. One entry had translated every quote into Spanish while its sources were in English, French and Russian, so nine of its ten failed.

That is, a big part of the LLM generated content on the Internet is completely wrong.

Download the sources first

What fixed it was taking the quoting away from the model:

1. Download the sources with curl before any model sees them.
2. Check that each file is what it says it is. I once had a book of Finnish poetry saved as Kant's essay on the Enlightenment, and three PDFs that were all the same 25 KB error page.
3. Give the writer the files and tell it not to quote anything that is not in one of them. For research claims the model does not quote at all: it returns the claim, the URL and a few keywords, and a script picks the sentence out of the downloaded HTML.

With that, 36 of 39 research quotes came out literal, and the thinker biographies ended with 95% of their quotes verifying against the source.

Before trusting a failure, run the verifier on quotes you know are good.

When the quote does not support the claim

A literal quote proves the text is in the source. It does not prove that it supports the sentence next to it.

The claim "the PP backed Sánchez's investiture in 2023" is false, and it still got a literal quote with a perfect score; that is, a sentence about Sánchez being appointed. So there is a second check, a model acting as judge that reads each claim with its quote and says whether it holds. A partial match only survives if the judge rewrites the claim into something the quote does support.

Later I ran a review with agents over the 26 riskiest pages. It caught things no script would... a page attributing a court case and a conviction to a living person with no source behind it, a quote about Putin whose source was about Kennedy, and an argument credited to Hayek that is Mises'. Reviewers also over-report, so every finding went to a second agent whose job was to refute it, and 45% of them fell.

Wording and context

Some fixes had nothing to do with facts. "Fabricated" became "started" in a sentence about a named journalist, and "the best ammunition against" became "the standard reference".

No check flags that kind of thing. Someone made of flesh has to read it.

So be prepared before starting a content webstie

You need to Budget for checking, as the writing is the cheap part now. Do not let the model produce what you cannot check and give it files instead of URLs, and let code extract the quotes.

I also recommende to test the verifier before you trust it and also keeping a threshold.

In many cases a shorter hand-written version should be chosen instead of an entry with fewer than four verified quotes.

Search engines are getting better at telling thin generated content apart, so this matters for ranking too, as Google might flag your site if it's basically poor AI-generated content.

Next time I start a content site, the verifier will be the first thing I write. Actually, most part of the site was heavily reviewed or just handwritten.