Stop LLM confabulations: make the model prove every claim with an exact quote

You're extracting structured facts from a long document: a contract, a support thread, a spec nobody on your team wrote. To keep the result checkable, you ask the model to return the exact line each fact came from.

Then one of those quotes doesn't survive a Ctrl-F against the source. Nothing else about the output looks wrong, which is how it got past review.

A line in the prompt saying don't invent anything can only point the model back at its input, which is no help when what you asked for was never in that input. That is what a confabulation usually turns out to be: the answer left over when the construction gives you no honest way to say the true thing.

What works instead of better wording is making every factual field a copy of something in the input, and putting code, rather than the model, in charge of checking that it is one. Proof-based grounding, if it needs a name.

TL;DR. Make every factual field a copy of something in the input. Tell the model the rule that will reject its answer. Put the limits in the schema instead of the prose. Give it a way to say "nothing found". Choose the temperature for copying rather than for writing. Verify in code and route what fails. All of it is packaged as a Claude Code skill:

git clone https://github.com/r-ms/prompt-construction.git ~/.claude/skills/prompt-construction

The six mechanisms below are in the order I'd apply them, each with what it is worth.

1. Classify every field: copied, or composed?

Walk your output schema field by field and ask, for each one, whether the model copies it out of the input or composes it.

Anything that has to be a fact must be printed in the input, in the form you ask back. Number the lines of the input and ask for the line number: [#42] the actual line, and the model returns {"line_id": 42, "quote": "..."}.

Then there are the fields it will invent whatever you write: character offsets, page numbers, ids that appear nowhere in the text. Don't ask for those at all. My own inputs are transcripts, and the pipeline hides word-level timestamps from the model on purpose, because it fabricates them fluently. Code reattaches them afterwards by matching the text the model copied.

Before blaming a model for a fact it didn't produce, print the assembled prompt and look for the thing you want back. One step of mine returned report headings as bare numbers until I printed the prompt and saw that the list I was passing in had ids in it and no titles.

2. Tell the model the rule that will reject its answer

If your code discards a quote that isn't an exact substring of the source, say exactly that in the prompt, in terms of the consequence:

quote: the wording verbatim as it appears in the source. A value not found by exact match will be discarded automatically.

Written that way it becomes a specification, and a model given the acceptance criterion optimises for it.

One line of this kind is worth +28.6 points of accuracy on my status field, the verdict recording whether an item was actually met in the document. It's now frozen byte-for-byte, with a comment naming the effect and the conditions it was measured under, because a reworded version loses the gain without any signal that it did.

3. Make constraints physical, not polite

"Return at most 25 items" was a line in my prompt. On the longest document in my corpus it returned 134.

The same limit written as maxItems stops generation at the boundary, because the decoder is never allowed to open a twenty-sixth element. In prose the number was one more thing the model had to weigh against everything else I asked for in the same prompt. So push whatever actually matters down into the schema: enum over a fixed set of values makes a wrong one unrepresentable, and maxLength will stop a looping decoder, which in my case once put a schema field name into a bullet of a customer-facing report.

Two limits worth knowing. Grammar can't tell a correct value from an incorrect one, only an absent one from a present one, so code validation stays. And a grammar guarantees a valid prefix, not a finished document: truncation by max_tokens gives you schema-conformant, unparseable output. Raising the ceiling doesn't fix that either, mine hit the new limit too.

4. Make "no evidence" representable

Under a strict schema every field must be filled. If "I found nothing" has no representation, the model fills the hole with an invention, and you've engineered the confabulation you're trying to stop.

Three conditions, all required: null or a dedicated value exists in the schema, the prompt names it an acceptable outcome, and code does something with it.

And never let uncertainty become a business status. "The model is unsure" and "this did not happen" look identical to whoever reads the output, and mean opposite things.

5. Temperature is part of grounding

Qwen's model card says, in capitals, "DO NOT use greedy decoding", and recommends 0.6 with thinking, 0.7 without.

My extraction pass runs at 0.1.

At 0.6 the model stops copying and starts paraphrasing. The quote stays plausible, stops being an exact substring, and the verification code drops it. Qwen's ban is aimed at degeneration and endless repetition, which are risks of generative work, and extraction has the opposite optimum. Against repetition I use frequency_penalty 0.3 instead, and the vendor names that cost too: "using a higher value may occasionally result in language mixing."

6. Verify in code, then route the disagreement

Never let the model validate itself, and don't let the verifier raise on the first unverifiable quote either, because then you lose the whole run and learn nothing from it.

def check_grounding(items, source_lines):
    ok, ungrounded = [], []
    for it in items:
        line = source_lines.get(str(it["line_id"]))       # keys are strings; assert this once
        if line and norm(it["quote"]) in norm(line):      # norm: NFKC, collapse ws, unify dashes
            ok.append(it)
        else:
            ungrounded.append(it)                         # route, don't raise
    return ok, ungrounded

Normalisation carries more weight here than it looks. Raw comparisons fail on a doubled space or a unicode dash more often than on an actual invention.

That leaves you a bucket of ungrounded claims to route, and the run survives. A second signal tells you which of the claims that did pass to still distrust: run the extraction more than once and watch where the runs disagree.

I run each one three times. When the three answers disagree, their error rate against my reference set is 57.1%, against 21.7% when they agree, so routing only the disagreements to a human catches about half of all errors.

How you know any of it worked

Two things fooled me here.

The first is noise. I score a change by running the pipeline over a fixed set of documents and comparing the output against reference answers I trust. That run scored 64.84%. I ran it again with nothing changed, same model, same server, same config, same documents, and got 57.45%. My pass threshold was 60.05%, sitting between the two, so the gate was reporting which way the noise leaned. Record the baseline twice before you believe any delta, because until you do you don't know what size of effect you are able to detect.

The second is a switch that does nothing. I enabled reasoning for an extraction step, measured no improvement, and nearly wrote reasoning off for that task. Then I printed how many reasoning tokens the run had actually produced and got 19 for the whole run, which is less than a sentence of thinking. vLLM's grammar backend checks whether reasoning content is present and skips the structured output when it is, so without the --reasoning-parser flag the two features collide and the reasoning is what gets dropped. An unfired mechanism and a useless one both report "no effect", so print a counter proving the thing you switched on is running before you read any result.

Audit checklist

1. Every schema field labelled copied or composed; composed facts deleted and rebuilt in code.
2. Input printed with line ids, and carrying every fact you ask back in the form you ask it back.
3. The rejection rule stated in the prompt, in terms of the consequence.
4. Limits in the schema (maxItems, enum, maxLength), code validation still present.
5. "No evidence" representable, permitted, and handled.
6. Sampling profile chosen for extraction rather than inherited from the model card.
7. A verifier that normalises before comparing, and routes what fails instead of raising.
8. Baseline recorded twice, so you know your minimum detectable effect before you claim one.

The effect sizes above belong to my stack, and the vendor quotes are quotes: I have not reproduced their experiments. What transfers is that every one of these mechanisms is checkable in an afternoon on your own stack, with a counter you print before you form an opinion.

All of this is a Claude Code skill. One command installs it:

git clone https://github.com/r-ms/prompt-construction.git ~/.claude/skills/prompt-construction

Claude then reads these rules before it writes your next prompt or response schema. The repo carries the vendor quotes, a symptom-to-mechanism table to debug against, and the pre-ship checklist.