Your LLM eval set is quietly certifying your bugs

Everyone eventually wants to build their LLM eval set from real traffic. It's the right instinct — hand-written test cases go stale, miss the weird inputs real users send, and quietly encode what you imagined the product does instead of what it actually gets asked. Your logs have the real distribution. So you export a few thousand rows and start turning them into a regression suite.

Then you hit a trap that's easy to miss and expensive to keep: your logs contain the model's past outputs, and those outputs may be wrong.

The circular ground-truth problem

A regression test needs two things: an input, and the expected output you'll assert against. Traffic gives you the input for free. The temptation is to grab the expected output for free too — it's right there in the log, it's what the model said last time.

But "what the model said last time" is exactly the thing you're trying to test. If last time it hallucinated a refund policy, and you freeze that answer as "gold," you've just written a test that passes only when your agent hallucinates the same way. Your suite now certifies the bug. Every future "regression" it catches is a regression away from being wrong, which is the opposite of what you want.

You cannot bootstrap gold from logs, because the logs were produced by the system under test. Ground truth has to come from somewhere the system can't contaminate.

There's a second, quieter trap layered on top: frequency sampling lies about what matters. If you sample traffic by how often things occur, 95% of your "eval set" is the happy path everyone already knows works, and the rare escalation, the tool-error branch, the adversarial input — the cases that actually break in production — get sampled into oblivion. The distribution that's most useful for testing is almost the inverse of the distribution by volume.

The principle: candidates are cheap, gold is earned

The way out is to stop conflating two different things:

• A candidate is a case mined from traffic. The tool can produce these all day — cluster the inputs, dedupe them, surface the rare and failing ones. Fully automatic.
• A golden case is a candidate where a human (or a genuinely trusted oracle) has confirmed the expected outcome. Never automatic.

Keep a hard wall between them. Nothing gets exported as "gold" until a human has set the expected outcome, and the exporter should refuse to lie about it. That single rule is what makes a traffic-derived suite trustworthy instead of a mirror of your model's mistakes.

I built a small open-source tool, goldset, around exactly this rule, so the rest of this post is concrete. But the principle stands on its own — you can steal it into whatever you already use.

What the workflow looks like

Install and run the self-checking demo first — it's keyless, no API key or model download, and it plants known problems (duplicates, a rare escalation, a coverage gap, an unconfirmed case) then shows the tool recovering them:

pip install git+https://github.com/AshwinUgale/goldset.git
goldset demo --html demo.html

The real pipeline is six steps, and every one of them is a design decision worth calling out:

goldset ingest  ./logs.jsonl --out ws.json    # 1. read logs — PII redacted BEFORE anything is stored
goldset cluster --workspace ws.json           # 2. embed + cluster inputs into candidate intents
goldset select  --workspace ws.json           # 3. medoid + boundary + MMR + mined failures, deduped
goldset label   --workspace ws.json           # 4. REQUIRED: a human confirms the expected outcome
goldset roles   --workspace ws.json --lock    # 5. deterministic dev / regression / holdout split
goldset export  --workspace ws.json --out suite.yaml   # 6. golden suite your runner can execute

A few things that fall out of taking the trap seriously:

• Redaction happens at ingest, before a single record is written to the workspace. If you're curating from real user traffic, PII scrubbing can't be an afterthought at export time — by then it's already on disk.
• Selection isn't random. Instead of frequency sampling, it picks cluster medoids (the representative case), boundary cases (the ambiguous ones near a cluster edge), an MMR pass for diversity, and mined failures — the escalations, tool errors, and safety flags that volume-based sampling would bury.
• label is not optional. This is the human gate. Each case carries a trust_tier: unconfirmed, human_confirmed, or oracle_confirmed.
• export fails closed. If nothing is confirmed, it exits non-zero and refuses to call the output golden. You can pass --include-candidates to emit the unconfirmed ones, but then the whole suite is marked a candidate set, not gold. The tool won't let you accidentally ship your model's guesses as ground truth.

Splits are locked and deterministic, so your holdout stays a holdout across runs, and there's coverage (relative to your traffic, or against an external --taxonomy if you want absolute coverage vs a spec), drift (new intents in fresh traffic not yet represented), and version --bump for evolving the set over time.

Two design choices worth stealing regardless

Cases are assertions, not exact strings. For generative systems, a frozen reference output is almost always the wrong contract — the model can be right in a hundred phrasings. The default case is an assertion about the output (it must contain / must not claim / must match), not a demand that it reproduce one blessed sentence.

Export to a standard format. goldset exports to promptfoo, which means the suite drops straight into an existing runner rather than living in a bespoke format only one tool understands. A curation tool shouldn't also try to be your test runner.

The one case where you can trust the log

There's an exception to "you can't bootstrap gold from logs," and it's a useful one: memory. Multi-turn memory failures — stale facts, forgotten context, cross-session leakage — can't be a single (input → expected) row, and you can't read the right answer off a transcript. But if you plant the fact yourself in an earlier turn, the expected outcome is trustworthy by construction — you know what the agent should remember because you told it. goldset's experimental scenarios module builds these as ordered turns with assertions that reference planted facts:

goldset scenarios build --out scenarios.yaml
# your runner replays each turn and writes a transcript...
goldset scenarios check scenarios.yaml --transcript transcript.json   # exit 2 if a check fails

Planting sidesteps the bootstrap problem entirely — which is the whole point of the human gate elsewhere, just enforced by construction instead of by a person.

Takeaway

If you're building evals from traffic, the tooling question ("how do I cluster and dedupe?") is the easy half. The half that decides whether your suite is worth anything is: where does the expected output come from, and can the system under test contaminate it? Mine candidates automatically; earn gold with a human; keep a hard wall between the two; and make your exporter refuse to blur it.

goldset is early and open source — the repo is here and the live demo report shows the planted-defect self-check end to end. If you're wrestling with this, run the demo and tell me where it breaks on your data — that's the most useful thing you could send me.

Comments

[Brian · AI News]

The circular ground truth problem is the eval trap teams underestimate. Keeping model outputs as candidates and earning gold through independent review makes the benchmark a test instead of a mirror. That separation should be standard for regression suites.

[Ashwin Ugale]

Exactly — and the thing that surprised me building this is that the separation only holds if you enforce it mechanically. "Confirm your gold before trusting it" is easy to agree with and easy to skip under a deadline, so the export has to fail closed until something's actually confirmed — otherwise the wall quietly erodes back into a mirror on the first busy week.

The other subtlety is who does the independent review. Reach for an LLM judge and you can walk the circularity right back in, unless that judge is genuinely independent of the system under test. The cheapest reliable oracle I've found is a human on just the boundary and failure cases — which are the ones you can't afford to get wrong anyway, so the review budget lands where it matters.