Catching wrong LLM outputs when you have no labels

Tarun Agarwal — Tue, 04 Aug 2026 07:35:09 +0000

A shorter, code-first version of a piece I first published in Towards AI on Medium.

If you've pointed an LLM at a pile of documents to extract a field — a total, a date, an amount — you know the uncomfortable part: it returns an answer for every document, all equally confident, and some are wrong. You can't check them by hand, because not having the answer is the whole reason you reached for a model. So the wrong ones ship silently.

I spent a couple of weeks on this and landed on something that helps, borrowed from an old idea called metamorphic testing.

The trick: test what shouldn't change
You usually know things that shouldn't change the answer:

reordering a receipt's line items shouldn't change the total
adding an irrelevant footer shouldn't change it
stripping a currency symbol shouldn't change it
So run your system twice — once on the original input, once on a "shouldn't-matter" variant — and compare. If the outputs disagree, the system just contradicted itself. That's a bug, found with zero labels.

from wobbly import check, Relation, unchanged
import random

def extract_total(receipt):
    # your system under test — any input -> output callable
    ...

rng = random.Random(0)
reorder = Relation(
    name="reorder lines => total unchanged",
    transform=lambda r: {"lines": rng.sample(r["lines"], len(r["lines"]))},
    assertion=unchanged(),
)

report = check(extract_total, receipt, [reorder], samples=20)
print(report.summary())

# BROKE (1 of 5 trials): expected 7.95 to be preserved, got 7.5
No correct total was ever supplied — yet we know one of those answers is wrong.

Does it actually catch anything?
I ran it on 535 real scanned receipts (ICDAR-SROIE) in two deliberately separate stages: flag receipts blind (never reading the labels), then open the labels only to score the flags.

On held-out data, the receipts it flagged were 2.7× more likely to be genuinely wrong than the ones it didn't — decided with zero labels.

The honest part
It's a spot-check, not a safety net. Recall is low (~8% on this dataset) — it misses most bugs. But precision is high (~75%): about 3 in 4 of the things it flags are real errors. So it's a cheap, label-free way to surface bugs you'd otherwise ship silently — not a replacement for a real test set.

The one thing that's easy to get wrong
A check is only useful if a correct system passes it. My first "reorder lines" transform shuffled every line — and broke correct extractors too, because real extractors pair a cue like TOTAL with the value on the next line. 68 of 72 flags were artifacts of a bad transform, not real bugs. The fix was to permute independent blocks (header / items / totals) as units. Design the transform so only a real defect can fail it.

Try it
Small Python library, no dependencies, MIT:

pip install wobbly
Code: https://github.com/tarunagarwal1981/wobbly Full experiment (with the mistakes): https://medium.com/towards-artificial-intelligence/your-llm-extracted-10-000-numbers-which-ones-are-wrong-7a5d54050dd3

How are you checking LLM outputs when there's no ground truth? Genuinely curious what people are doing.

DEV Community: Tarun Agarwal

Catching wrong LLM outputs when you have no labels