I got tired of hand-writing test cases for our agent, so I did the obvious 2026 thing: I pointed an LLM at our docs and asked it to generate them. In an afternoon I had a few thousand. I felt incredibly productive.
They passed CI on every release. Production kept breaking anyway.
When I finally looked closely, maybe a third of my generated tests were wrong, near-duplicates, or things no real user would ever ask. The set was too noisy to catch a real regression, so a green check meant nothing. The generating was the easy part. The part I had skipped, throwing most of them away, was the entire job.
Here is what I learned making auto-generated tests that actually catch things. The one line I would tattoo on this: generate ten times what you need, and keep only the tenth that survives a hard filter.
Why AI-generated test sets are usually useless
Two things go wrong, and I hit both.
The first is no real filter. You generate thousands, run a quick "is this well-formed" check, and ship. A big chunk are subtly wrong or duplicates, and the set quietly stops testing anything. It passes every release because it is too mushy to fail.
The second is sneakier, and it took me months to see. The generator has a house style, and everything it writes drifts toward that style. So your test set is really a test of how well your model imitates the model that wrote the tests, not how well it does the actual job. And if you ever fine-tune on that set, your model just learns to sound like the generator. The scores go up while nothing real gets better.
Both come from the same mistake: treating generation as a one-shot prompt instead of a generate, judge, and keep loop.
Generate along three axes, not one
If you generate one flavour of test, you get one flavour of blind spot. I now generate along three axes on purpose:
- Personas. Same request, different people. A rushed user on their phone and a careful expert on a laptop phrase the identical question in completely different ways, and your model has to handle both. One warning: build personas from real user data, not the model's imagination, or you get cartoons ("frustrated user" becomes a stereotype).
- Scenarios. Not one-line questions, but whole conversations with tool calls, dead ends, and recovery. This is the only way to catch a step that drops context on turn three or a tool that returns the wrong shape.
- Adversarial versions. Take a normal case and make it harder: add a step, make it ambiguous, or phrase it like someone trying to slip past your rules. Real traffic surfaces these too slowly to wait for, so you manufacture them.
Run one axis alone and it collapses onto the others. Run all three and you get coverage you can actually point at.
The filter is the whole game
This is the part everyone skips and the part that matters. After generating, I run everything through a few stages and expect to throw away most of it:
- Drop duplicates by meaning, not by text. Two questions can be worded differently and still be the same test. Matching on meaning catches the paraphrases that exact-text matching misses.
- Judge every candidate and reject the ones that fail. Is this something a real user would ask? Is the expected answer actually recoverable? Is it just a reworded copy of the seed?
- Check for that house-style collapse before you ship. If your synthetic set is far less varied than real traffic, that is a regenerate signal, not a green light.
- Hand-check a small sample. Label five percent yourself. If the quality there is bad, throw the batch out.
The first time I saw that this rejected 80 percent of my candidates, I flinched. Then it caught a batch that was a third wrong before it could poison the set, and I stopped flinching.
Use a different model family to judge than to generate
This is the single most useful thing I picked up, so it gets its own section.
If the same model both writes and grades the tests, it is blind to its own bad habits. It shares its own vocabulary quirks, its own stereotyped personas, its own shortcut reasoning, so it happily rubber-stamps exactly the junk it produced. A judge from a different family does not share those blind spots, and it agrees with human reviewers far more often.
So I generate with one model and judge with another, or rotate which one grades each batch. It is a small change and it is the difference between a filter you can trust and a filter that quietly approves its own mistakes.
Cover the space on purpose, or you will oversample the easy stuff
Even with three axes, unconstrained generation piles up on the easy, obvious cases. So I lay out a simple grid, something like intent by difficulty by mood by dialect, and set a small quota per cell.
The reason is dumb but important: an empty cell in a grid is visible, so you can see the gap and fill it. A gap in a random pile is invisible until it shows up as a production incident. I drop the cells that make no sense for my product and fill the rest.
Check the synthetic set against real data before you trust it
Synthetic tests are only worth anything if they resemble reality, so before I trust a set I run three quick checks:
- Does the synthetic set actually sit near real traffic, or is it off in its own corner of the space?
- If the model wrote the labels too, I re-label a small sample by hand to make sure the labels are not carrying the generator's bias.
- If the set is for fine-tuning, I train on synthetic and then test on real. If it only looks good on synthetic, the generator invented a world my model is now optimising for.
Different jobs need different filter strength
The same loop ships in three shapes, and the only thing that really changes is how hard you filter:
- An eval set gets the strictest filter. It has to be right or your CI gate is theatre. A few hundred great cases beat thousands of noisy ones.
- A fine-tuning set can take more volume and a slightly looser filter, because training averages over noise. But keep the filter, or your model inherits the generator's accent forever.
- A red-team set flips the filter around. Now "too hard" is the goal. You keep the nasty ones and only reject the attacks that are not realistic.
Same generator, same judge. You are just turning the strictness dial.
The mistakes that quietly wreck an auto-generated test set
- Letting one model both write and grade the tests.
- Never checking for style collapse, so it only shows up months later.
- Trusting one overall score that hides a weak high-risk slice.
- Not tracking where each test came from, so you cannot re-check it when your docs change.
- Treating generation as a replacement for human judgement. It is volume, not judgement. The high-stakes cases still need a human.
- Running it once. Real traffic drifts, and a set you generated in spring is stale by autumn.
The thing I keep coming back to is that generating the tests was never the hard part, and it is the part all the tooling loves to show off. The hard part is being willing to throw most of them away, and letting a different model be the one that decides.
If you want to go deeper on the filtering side, the judge rubric, the dedupe cutoffs, the checks that catch style collapse, this piece is a good one on it.
If you have generated tests this way, I would love to know your keep rate. Mine settled around one in ten, and I am genuinely curious whether other people land higher, or whether everyone quietly throws most of theirs away too.
Top comments (1)
The grid is the part I'd push on. Intent by difficulty by mood by dialect is still your mental model of the product, so the blind spot doesn't disappear, it moves up a level: you can only see empty cells you thought to draw. Clustering real traffic first and letting the clusters define the axes would at least make the gaps come from the data instead of from you. Also curious how rotating judges affects your keep rate, since a different family will have a different strictness baseline and one in ten stops meaning the same thing batch to batch.