DEV Community: Kartik N V J K

How I generate LLM test cases that actually catch bugs

Kartik N V J K — Mon, 27 Jul 2026 17:32:42 +0000

I got tired of hand-writing test cases for our agent, so I did the obvious 2026 thing: I pointed an LLM at our docs and asked it to generate them. In an afternoon I had a few thousand. I felt incredibly productive.

They passed CI on every release. Production kept breaking anyway.

When I finally looked closely, maybe a third of my generated tests were wrong, near-duplicates, or things no real user would ever ask. The set was too noisy to catch a real regression, so a green check meant nothing. The generating was the easy part. The part I had skipped, throwing most of them away, was the entire job.

Here is what I learned making auto-generated tests that actually catch things. The one line I would tattoo on this: generate ten times what you need, and keep only the tenth that survives a hard filter.

Why AI-generated test sets are usually useless

Two things go wrong, and I hit both.

The first is no real filter. You generate thousands, run a quick "is this well-formed" check, and ship. A big chunk are subtly wrong or duplicates, and the set quietly stops testing anything. It passes every release because it is too mushy to fail.

The second is sneakier, and it took me months to see. The generator has a house style, and everything it writes drifts toward that style. So your test set is really a test of how well your model imitates the model that wrote the tests, not how well it does the actual job. And if you ever fine-tune on that set, your model just learns to sound like the generator. The scores go up while nothing real gets better.

Both come from the same mistake: treating generation as a one-shot prompt instead of a generate, judge, and keep loop.

Generate along three axes, not one

If you generate one flavour of test, you get one flavour of blind spot. I now generate along three axes on purpose:

Personas. Same request, different people. A rushed user on their phone and a careful expert on a laptop phrase the identical question in completely different ways, and your model has to handle both. One warning: build personas from real user data, not the model's imagination, or you get cartoons ("frustrated user" becomes a stereotype).
Scenarios. Not one-line questions, but whole conversations with tool calls, dead ends, and recovery. This is the only way to catch a step that drops context on turn three or a tool that returns the wrong shape.
Adversarial versions. Take a normal case and make it harder: add a step, make it ambiguous, or phrase it like someone trying to slip past your rules. Real traffic surfaces these too slowly to wait for, so you manufacture them.

Run one axis alone and it collapses onto the others. Run all three and you get coverage you can actually point at.

The filter is the whole game

This is the part everyone skips and the part that matters. After generating, I run everything through a few stages and expect to throw away most of it:

Drop duplicates by meaning, not by text. Two questions can be worded differently and still be the same test. Matching on meaning catches the paraphrases that exact-text matching misses.
Judge every candidate and reject the ones that fail. Is this something a real user would ask? Is the expected answer actually recoverable? Is it just a reworded copy of the seed?
Check for that house-style collapse before you ship. If your synthetic set is far less varied than real traffic, that is a regenerate signal, not a green light.
Hand-check a small sample. Label five percent yourself. If the quality there is bad, throw the batch out.

The first time I saw that this rejected 80 percent of my candidates, I flinched. Then it caught a batch that was a third wrong before it could poison the set, and I stopped flinching.

Use a different model family to judge than to generate

This is the single most useful thing I picked up, so it gets its own section.

If the same model both writes and grades the tests, it is blind to its own bad habits. It shares its own vocabulary quirks, its own stereotyped personas, its own shortcut reasoning, so it happily rubber-stamps exactly the junk it produced. A judge from a different family does not share those blind spots, and it agrees with human reviewers far more often.

So I generate with one model and judge with another, or rotate which one grades each batch. It is a small change and it is the difference between a filter you can trust and a filter that quietly approves its own mistakes.

Cover the space on purpose, or you will oversample the easy stuff

Even with three axes, unconstrained generation piles up on the easy, obvious cases. So I lay out a simple grid, something like intent by difficulty by mood by dialect, and set a small quota per cell.

The reason is dumb but important: an empty cell in a grid is visible, so you can see the gap and fill it. A gap in a random pile is invisible until it shows up as a production incident. I drop the cells that make no sense for my product and fill the rest.

Check the synthetic set against real data before you trust it

Synthetic tests are only worth anything if they resemble reality, so before I trust a set I run three quick checks:

Does the synthetic set actually sit near real traffic, or is it off in its own corner of the space?
If the model wrote the labels too, I re-label a small sample by hand to make sure the labels are not carrying the generator's bias.
If the set is for fine-tuning, I train on synthetic and then test on real. If it only looks good on synthetic, the generator invented a world my model is now optimising for.

Different jobs need different filter strength

The same loop ships in three shapes, and the only thing that really changes is how hard you filter:

An eval set gets the strictest filter. It has to be right or your CI gate is theatre. A few hundred great cases beat thousands of noisy ones.
A fine-tuning set can take more volume and a slightly looser filter, because training averages over noise. But keep the filter, or your model inherits the generator's accent forever.
A red-team set flips the filter around. Now "too hard" is the goal. You keep the nasty ones and only reject the attacks that are not realistic.

Same generator, same judge. You are just turning the strictness dial.

The mistakes that quietly wreck an auto-generated test set

Letting one model both write and grade the tests.
Never checking for style collapse, so it only shows up months later.
Trusting one overall score that hides a weak high-risk slice.
Not tracking where each test came from, so you cannot re-check it when your docs change.
Treating generation as a replacement for human judgement. It is volume, not judgement. The high-stakes cases still need a human.
Running it once. Real traffic drifts, and a set you generated in spring is stale by autumn.

The thing I keep coming back to is that generating the tests was never the hard part, and it is the part all the tooling loves to show off. The hard part is being willing to throw most of them away, and letting a different model be the one that decides.

If you want to go deeper on the filtering side, the judge rubric, the dedupe cutoffs, the checks that catch style collapse, this piece is a good one on it.

If you have generated tests this way, I would love to know your keep rate. Mine settled around one in ten, and I am genuinely curious whether other people land higher, or whether everyone quietly throws most of theirs away too.

Evaluating an intent classifier: what I check beyond accuracy

Kartik N V J K — Wed, 22 Jul 2026 16:18:03 +0000

I built the intent router that sits at the front of our support agent. It reads what a user typed and decides which pipeline handles it. I tested it on a nice balanced set, got 92 percent accuracy, and shipped it feeling good.

Two weeks later the on-call engineer noticed cancellation requests were landing in the billing queue. The fraud-report queue was empty while actual fraud complaints sat in product feedback. And a newer request type that did not exist when we launched was being silently filed as a billing question most of the time.

The model had not broken. My test set had simply hidden the failures that actually happen in production. The 92 percent was real and useless at the same time.

Here is what I learned pulling that apart, in plain terms, so you can skip the two weeks I spent finding out the hard way.

Why the headline accuracy number lied to me

Real support traffic is lopsided. Five or six common intents (product questions, pricing, account access, shipping, refunds) make up most of the volume, and they are easy. The rare stuff (cancel subscription, fraud report, policy dispute, a brand-new request type) is a small slice of traffic and it is exactly where the money, the on-call pages, and the trust-and-safety incidents live.

A balanced test set gives every intent equal weight. So the model learns to do fine on all of them evenly, and my one big accuracy number was dominated by the easy, high-volume classes. The rare classes could quietly collapse to 30 percent and the headline barely moved, because they were a rounding error in the average.

And every one of those rare classes matters more than its volume suggests. Whatever the router picks decides which knowledge base, which tools, and which escalation path runs next. Get the routing wrong and everything downstream is confidently wrong.

Two test sets, not one

The fix that changed everything for me was keeping two separate test sets:

One that mirrors real traffic. Same lopsided mix as production. This is the number I report, because it is what users actually experience.
One that oversamples every intent evenly, a hundred or so examples per class no matter how rare. This is the one I debug against, because it is the only way to see whether the model can even do the rare classes at all.

The gap between those two is the warning sign. If the model is great on the real-traffic set but weak on a rare class in the even set, that class is one traffic shift away from becoming my next incident.

Look at each intent, not the average

Once I stopped looking at one number and started looking at each intent separately, the problems were obvious. For every intent I now look at two things:

How often is it right when it fires? If this is low, the model is over-using that intent as a dumping ground when it is unsure.
How often does it catch the real cases? If this is low, the model is quietly sending those cases to some neighbouring intent instead.

The single most useful view is the grid of what got confused with what. That grid told me plainly that "cancel" requests were mostly being read as "billing." Once you can see the exact pair that is colliding, the fix is usually two or three edits to how you describe those two intents to the model, not some big model change.

When one intent is both rarely right and rarely caught, that is the danger zone: either the description is ambiguous, the label itself is badly defined, or the model genuinely cannot tell it apart from its neighbour. The grid tells you which.

The confidence score means nothing until you calibrate it

My router also spat out a confidence number, and I had wired the escalation logic to trust it: below some cutoff, hand off to a human. The problem is that the raw confidence was close to meaningless.

Two things surprised me. The scores clustered at the extremes, either very high or very low, with almost nothing in the middle to set a cutoff against. And the same number meant different things for different intents. A 0.7 on a common product question was right almost every time, while a 0.7 on a fraud report was barely better than a coin flip.

So one global cutoff was shipping a totally different level of caution per intent. What actually works:

For irreversible actions like cancelling a subscription or flagging fraud, set a high bar and send anything below it to a clarifying question or a human.
For catch-all safety intents, set a low bar so the system over-routes to the safe path instead of guessing.

And treat a change to those cutoffs like a change to the model itself. Version them together.

Give the model a "none of these" option

This was the one I had completely missed. If you do not give the model an explicit "none of these" class, it is forced to jam every input into some known label. Gibberish, off-topic questions, and outright jailbreak attempts all get filed as a real intent and routed into a real pipeline.

So I added an explicit "does not fit any intent" class and trained it on the messy stuff: adversarial prompts, vague multi-topic messages, plain nonsense. Two rules made it work:

If nothing clears its confidence bar, fall through to this class instead of picking the best guess.
Never let this class dead-end. Route it to a "which of these did you mean" question or a human, not into a pipeline.

I also watch this class's catch rate closely and never let it get worse. A drop there means real garbage is leaking into real flows again.

The taxonomy you ship goes stale

The list of intents I launched with was not the list production needed three months later. User language drifts, a product launch creates a brand-new request type, a policy change creates a new kind of complaint. That new request type being misfiled as billing was exactly this.

Two things I watch for now:

An existing intent slipping. If its numbers move week over week, the model is fine but the incoming language shifted under it. Refresh its examples and re-baseline.
A new intent forming. I group together the stuff that landed in the "none of these" bucket and the low-confidence misses, and look for a cluster that keeps showing up. If the same new pattern is there two weeks running, it has earned its own intent: label some examples, write a definition, add it.

Two things I wish I had known on day one

Define intents by what the user wants, not what your system does. "User wants a refund" is an intent. "Run the refund flow" is a tool. I had mixed the two and it muddied the labels.
Check that humans agree on the labels before blaming the model. Have a few people label the same hundred examples. If they cannot agree on an intent, the model has no chance, and that is a taxonomy problem to fix first, not a model problem.

The thing I keep coming back to is that the intent router is the first decision the whole agent makes, and I had been grading it with the one number least likely to show me its real failures. The moment I looked per intent, on real traffic, with a "none of these" escape hatch, the bugs that had hidden for two weeks were sitting right there in the grid.

If you run a router like this, I would love to hear which intent quietly fell apart on you. For me it will always be cancellations pretending to be billing questions.

Evaluating LangGraph agents: the four things I score per step

Kartik N V J K — Mon, 20 Jul 2026 12:11:30 +0000

I had a LangGraph agent running in production and a regression set of fifty queries it passed comfortably. Task completion sat at 0.93. I spot-checked the answers and they looked right. I genuinely thought I had this covered.

Then someone mentioned our cost per query had roughly doubled.

I spent an afternoon in the traces and found three things wrong at once. The router was taking the expensive path on every query that happened to mention a date. A retry loop was firing three times before the search tool gave up, and at that point the model just invented an answer. And a step near the end was overwriting a constraint that an earlier step had carefully set.

Here is the part that stung. Every single one of those still produced a correct-looking final answer. My evaluation only looked at final answers. So it saw nothing, and told me everything was fine, for a week.

That was when I realised I had been evaluating the wrong thing the entire time. Took me a while to work out what to do instead, so here is the short version.

Why the final answer hid all of it

If your agent runs in a straight line, checking the output is mostly fine. But the reason you reach for LangGraph in the first place is that your agent is not a straight line.

And everything that breaks lives in the parts a straight line does not have:

Branches. Something reads the state and picks a path. A right answer down the wrong path is still a bug, just an expensive one.
Retry loops. A step can loop back on itself. Five retries and a correct answer costs five times one try and a correct answer.
Shared state. Steps read and write to a shared bag of data. One step overwriting where it should have added is silent until it is not.
Nested graphs. A step can be a whole graph itself. Flatten it when you look and you lose where the problem was.
Parallel work that merges. Fan out, merge back, and a collision quietly drops data.
Memory across turns. A bug in how state gets saved might not show up until turn three.

The final answer averages over all of that. It moves when things are catastrophically broken and sits perfectly still while the agent slowly rots. Which is exactly what mine did.

The four things I score per step

The change is easy to describe and took me a while to actually do. Stop treating the agent as a conversation with an output. Start treating it as a sequence of steps, and grade the steps.

Four questions per step:

Did it get what it needed? The right data, the right shape, nothing missing. A planning step that runs before the user's intent is filled in is broken before it starts, and everything after inherits that.
Did it write back correctly? Did it update the shared state the way it was meant to. This is where I finally caught the overwrite-instead-of-append bug that had cost me a day.
Did it pick the right branch? At a decision point, given the state right then, was that the correct path.
Does it do the same thing twice? Same starting state, same behaviour after. If not, something in there is quietly random.

First two catch correctness. Third catches expensive-but-correct, which is the one that got me. Fourth catches the bugs that become flaky incidents six months later.

If you only add one check, add the branch one

This is the one I would go back and add first.

A wrong branch still produces an answer, so nothing looks broken to anyone. But a five percent drop in how often one decision point chooses correctly shows up in your cost and latency weeks before quality metrics move at all. It is a real early warning, and I had nothing like it.

It is also so much more actionable. Overall score drops a point and you are in for a two-day hunt. One decision point drops five points and you know exactly which instruction to go rewrite. On a Friday afternoon that difference matters.

Grading everything gets expensive, so I stopped

I tried scoring every step at first and the bill got silly fast. A graph of any real size does dozens of state changes per run.

What works: pick the five to ten steps whose output lots of other steps depend on. Planners, routers, the merge points after parallel work. Score those every run and sample the rest. Most of the signal, a fraction of the cost.

The checkpoint trick I had been ignoring

This is my favourite thing I learned, and it was sitting in the framework the whole time.

LangGraph can snapshot the agent's state at every step so a long job can pause and resume. That is what it is advertised for. But those snapshots are also the best debugging tool you have.

Save one at every step and every production failure becomes a test you can rerun whenever you like. Grab the snapshot ID off the failing trace, rerun from that exact point, and watch what happens. Want to check a fix? Replay the same snapshot against the old version and the new one and compare.

Before this I was guessing whether a bug would reproduce. After it I was running an actual experiment. On an incident call that is a completely different feeling.

My test set was testing my imagination

My original fifty queries were fifty things I had thought of, which means they tested my assumptions rather than my agent.

Now I pull the runs that scored badly, group the similar ones, and promote a representative snapshot from each group into the set with a note on what should have happened. Then replay all of them on every pull request.

Every incident becomes a permanent test. The set gets stronger instead of going stale, which is the opposite of what my first one did.

When this is overkill

Being honest: if your graph is five steps in a straight line with no branching, one overall score really is enough and all of this is a waste of your time.

It starts paying off once you have real branching, retries, nested graphs, or state that survives across turns. In my experience that describes most agents by about month three, whether you planned for it or not.

What I keep coming back to is that I had built a state machine and I was grading it like a chatbot. The moment I started scoring the steps, bugs that had hidden for a month were obvious in an afternoon.

If you run agents like this, I would like to hear which check caught the most for you. For me it was the branch one, and it was not close.

I picked a coding agent off a leaderboard. It flopped on our codebase.

Kartik N V J K — Tue, 14 Jul 2026 21:51:14 +0000

Last year my team had to pick a coding agent, and I volunteered to run the evaluation. I felt good about it. I pulled up the public benchmark scores, lined up the contenders, took the one at the top, and told everyone we had a winner.

Then we actually pointed it at our repo.

It did not blow up dramatically. It just kept being slightly wrong in ways that ate our time. It wrote diffs our reviewers would not approve. It renamed a function and broke three files it had never opened. The tests it ran passed, and the repo was still broken. I had confidently recommended a tool based on a number that turned out to say almost nothing about our situation.

That was embarrassing enough that I went and figured out why. It took a few weeks of reading and a couple more bad calls before I landed on something that works. This is that, written plainly, and I hope it saves you the meeting where you have to walk your recommendation back.

Why the benchmark score lied to me

The score was not fake. It was just measuring somebody else's code. Once I looked properly, four gaps explained the whole thing:

The agent might have already seen the answers. The problems in these public benchmarks are old. Models were very likely trained on the actual fixes used to grade them. So the score partly measures memory, not problem-solving.
The setup is nothing like real work. A benchmark gives the agent a clean repo, one clear issue, and one command to run the tests. My engineers give it a half-open editor, a messy branch, a Slack thread, and a reviewer comment. Completely different job.
Our codebase has its own habits. Our internal libraries, our wrappers, our test style, the imports we ban. No benchmark knows any of that, so an agent can write textbook-perfect code that our reviewers still reject on sight.
The bar for passing is way lower. A benchmark passes a patch if the broken test now passes. My team passes a patch if it does that, and does not break unrelated tests, does not reformat the whole file, does not add stray imports, and is small enough to actually review. Benchmarks grade the first thing and none of the rest.

So I stopped treating the leaderboard as a decision and started treating it as a sniff test. Fine for narrowing the field. Useless for the actual call.

The test set was in our git history all along

Here is the bit I wish I had thought of on day one. The best possible test data for a coding agent is the work we had already merged.

Our last fifty merged pull requests came with everything a benchmark cannot fake: the real intent, the reviewer's comments, a passing test suite, and a known-good final diff. So I replayed them:

Take the last fifty merges, skip version bumps and generated files, keep the mix close to what we really ship.
For each one, note the commit it started from and the issue text.
Roll the repo back to that starting point, hand the agent the issue, let it work in a sandbox.
Compare what it produced against what we actually merged.

I got this wrong on my first attempt, so learn from it: I scored on how closely the text matched. That is both unfair and useless. A human reviewer would happily merge a differently-worded solution that behaves the same way. The question is not "does this look like our diff." It is "would this have passed the same review our diff passed."

I also swap in a freshly merged PR every week and drop the oldest, otherwise the set slowly stops resembling what we ship.

The five things I ended up scoring

After running enough replays I stopped guessing and settled on five things. Each one catches something a benchmark never sees.

Would the diff pass our review? Behavior, not text. I mark it down for reformatting files nobody asked it to touch, for breaking a test the merged version passed, and for taking twice the diff to reach the same outcome.

Did it use its tools in a sane order? A coding agent is a loop, not a text generator. It reads, edits, runs tests, reacts, commits. I check four things: did it read a file before editing it, did the edit land where it said it would, did it actually run the tests afterward or just commit blind, and did it commit only what it changed. Editing a file it never opened is just guessing with confidence.

Did it update everything the change touched? This is the one that burned us. The agent renames a helper in one file and leaves broken call sites scattered across others. The tests in the file it touched pass. The repo is wrecked. So now I follow the imports outward from every changed file and check that the downstream files still line up.

Did the plan survive the work? Modern agents write a plan before touching anything, and that plan is worth grading. Two questions: did the plan match what was actually asked, and did the final diff still match the plan. Agents drift halfway through and quietly contradict the original request while the tests keep passing. This turned out to be the best predictor of which agent our reviewers ended up trusting, which I did not expect at all.

Does it know when to give up on a bad idea? This was the sleeper, and it is now the one I care about most. When tests fail after an edit, a disciplined agent reverts and tries another approach. An undisciplined one piles more code on top of the failure until the suite is in pieces. You can see it plainly in the trace: how many edits does it make per failed test run, and does it ever actually revert. An agent averaging several edits per failure with no reverts will ship regressions. Count on it.

The first three catch correctness. The fourth catches intent. The fifth catches whether you can trust it unsupervised.

One more check I added after a scare

Coding agents invent credentials and echo risky shell patterns more often than I expected. So now every generated diff gets scanned for made-up secrets, dangerous constructs like raw eval or shell interpolation, and whatever patterns our own repo has banned. It takes milliseconds and it has already caught things. Cheap insurance.

What actually changed

The agent that won our replay was not the one at the top of the leaderboard. It was the one whose plans stayed honest and that knew when to back out of a bad edit. Neither of those shows up in a public score, and both were exactly what our reviewers cared about.

The line I settled on: the vendor owns the model and the prompt. I own the evaluation and the decision. And my evaluation is the only one that knows what our code actually looks like.

If you have run something like this on your own repo, I would love to hear which dimension caught you off guard. For me it was rollback. It never occurred to me to measure whether an agent knows how to abandon a bad idea, and it turned out to matter more than almost anything else.

I built an LLM eval framework from scratch. Here is what I wish I had bought instead.

Kartik N V J K — Tue, 14 Jul 2026 21:50:54 +0000

One weekend I wrote an LLM eval framework in about two hundred lines of Python. It demoed beautifully. I felt clever.

Six months later that same framework was a mess. Three different judge models with three different parsing hacks. A test dataset nobody had touched since November. A CI gate that kept failing because a vendor nudged their model, not because anyone broke a prompt. And the second engineer on rotation asking me, fairly, "how does this even work?"

The framework did not fail. The eighty percent of the work the weekend tutorial skipped is what failed. That gap is the whole story, and this is what I would tell myself before starting again.

The one line I wish someone had told me: build the rubric, buy the runner

Here is the split that took me six months to see.

Some parts of an eval setup are yours and only yours. The rubric that decides what "good" means for your product. The dataset built from your real failures. The rules for when a change is bad enough to block a release. Nobody else can write these, because they encode your domain.

The rest is the same at every company. The thing that calls the judge model, parses its answer, retries, and caches. The machinery to run thousands of checks in parallel. The plumbing that scores live traffic. The system that groups failing calls together. Every team rebuilds these, hits the same bugs, and gains nothing by writing them twice.

So build the first list. Do not hand-build the second. I rebuilt the second, and it cost me most of a year.

Two questions I now ask about every single piece

Before writing any part of this, I ask two things:

Is it specific to me, or generic? A rubric for my domain is specific and worth owning. A retry-and-cache loop around a model call is generic. Everyone writes the same one.
Does it compound, or does it rot? A dataset that grows from real production failures compounds. A year in, it is a regression suite no competitor can copy. A hand-built tracing layer rots. The moment a vendor changes something, you are behind and patching.

Build the pieces that are specific and compound. Reuse everything else. That single filter would have saved me the whole mess.

What is actually worth building yourself

Four things earned their place, and all four live in the same git repo as the agent code, reviewed in the same pull request as the prompt they score:

The rubric. Keep it in git, not a database, so changes show up in diffs. And make it specific. "Rate the quality" is noise. "Score 5 if every claim is supported by the source, 1 if it is largely made up" is a signal you can act on. Broken-down rubrics like that agree with human reviewers noticeably more often.
The dataset. A hundred real cases from production beat two hundred you invented. Pull the hardest failures in, because most of the signal lives in the worst ten percent of calls. Tag each case by area so a change to one feature does not rerun everything.
The gating rules. The exact thresholds and policy for blocking a release are yours. The math behind them is not, but the decision of where the bar sits is.
Your labels. The human judgement about what a correct score looks like. Only your team can produce that.

What I should not have built

The judge engine is the one that hurt the most, so start there.

Every judge model hands back its score in a slightly different shape. One wraps it in tags, another writes "Score: 4," another buries the number after a paragraph of rambling. My parser caught almost all of it on day one. On day ninety a vendor shipped an update that formatted numbers differently, my parser quietly fell back to a middle score on the cases it missed, and my dashboard showed evals gently trending up. They were not. Middle scores just drag everything toward the average.

On top of that came drift, where a judge scores the same answer differently this month than last after a model update, so the gate starts failing on prompts nobody touched. And cost, where running a top-tier judge across a big nightly set becomes a real budget line. Getting all of that right is months of work for an output that looks identical at every company.

The same goes for running checks at scale, scoring live traffic, and grouping failing calls into themes. All generic, all a long slog, none of it your edge. I would reuse an existing runner for every one of these now.

Gate on the change, not the average

This was my most expensive small mistake. I set the gate to "fail if the average score drops below a line."

The trouble is that a small dataset is noisy. With only thirty test cases, the average wobbles by a few points on its own, just from which cases you picked. So my gate failed on random noise and passed on real regressions, and within a month everyone had learned to ignore it, which is worse than having no gate at all.

What actually works is two checks together:

A hard floor that catches a catastrophic drop.
A comparison against a rolling recent baseline that only fires when the drop is bigger than the normal wobble.

The second one is the difference between a gate people trust and a gate people mute.

Run the same check in CI and in production

I kept my checks in CI only, and production drifted past my dataset within a quarter.

The fix is to run the exact same rubric in both places: against your test set before release, and against live traffic after. When they disagree, that disagreement is useful. If CI is green but production is failing, your dataset has stopped looking like reality. If CI fails on something production never shows, your dataset is carrying dead weight. Either way you learn something, but only if it is the same check on both sides.

The cost nobody warned me about

The weekend version is genuinely a weekend. The real version is not. Once you are shipping a couple of products, keeping all of this alive is close to one to one and a half full-time engineers.

And the cost never shows up when you expect it. It shows up the week a judge model updates and your parser breaks. The week a vendor changes their tracing format. The month your dataset quietly stops covering your product's top failure. None of that is on the weekend roadmap, and all of it is the job.

When building the whole thing yourself is actually right

To be fair, sometimes it is the correct call:

You are a research lab and the eval method itself is the product.
You have a strict isolation requirement where every dependency needs a long security review.
Your domain is so narrow that a general setup is just dead weight.

And a few signs you have crossed the line and should stop hand-building:

A second engineer joined and the framework now lives in one person's head.
You are maintaining more than one parser for more than one judge.
The eval bill has become a quarterly conversation.

The mistakes I would warn a friend about

Treating the eval setup as a side project instead of code that ships with the prompt.
Keeping rubric prompts in a database where changes never show up in a diff.
Letting the judge model float instead of pinning a version.
Gating on the average of a tiny dataset.
Running the full expensive judge on every single pull request.
Freezing the dataset at launch, so a two-year-old test set is grading a brand-new product.

The lesson I keep coming back to is that the hard, valuable part was never the code. It was the rubric and the dataset, the two things only my team could write, and I spent my first six months building everything except those.

If you have built one of these, I would love to hear which piece quietly ate the most of your time. For me it will always be that parser.

How I caught the voice-agent failures my eval dashboard kept missing

Kartik N V J K — Tue, 30 Jun 2026 06:59:33 +0000

I was working on a retail support voice agent, and on paper it looked great. Transcription quality, green. Conversation coherence, green. Task completion, green. Every standard metric I had was happy.

The problem was that it kept getting order numbers wrong. It would confidently read back the wrong number, the call would still "complete," and my dashboard would still show green. The checks I had simply did not know what an order number was, or why getting it wrong was the whole ballgame.

That is the gap I want to write about. The default metrics catch the failures every voice agent shares. They cannot catch the failures that belong to your specific domain. Here is what I learned about writing checks that do.

Built-in metrics catch the universal failures, not yours

Every voice agent, in any business, can fail in the same handful of ways:

Did it transcribe what the user said correctly?
Did the conversation stay coherent across turns?
Did it actually finish the task by the end of the call?
Did it stay safe, no leaked personal data, no obvious prompt injection?

Off-the-shelf checks handle all of that well. What they cannot know is your taxonomy. A retail agent lives or dies on order numbers. A clinical scribe lives or dies on drug names. An insurance agent lives or dies on coverage codes. None of that is in a generic metric, so none of it shows up on a generic dashboard.

Know when to reach for a custom check

The rule I settled on is simple:

If the failure is universal, use a built-in check.
If the failure is specific to your domain, write a custom one.
In production, run both on every call.

The last point matters more than it sounds. The two layers catch completely different kinds of errors. A call can pass the universal checks and still fail the domain one, which is exactly what was happening to me. You want both signals, not one.

What a custom check actually looks like

It helps to see a few real ones, in plain terms, no code:

Drug names for a medical scribe. Not just "was the word spelled right," but did the drug, the dose, the frequency, and the route all survive together. Getting three of four right is still a dangerous note.
Brand voice for a marketing line. Did it open with the approved greeting, avoid the banned phrases, and keep the right tone. A coherent call can still sound nothing like the brand.
Quote correctness for an insurance claim. Check each field, the deductible, the limit, the exclusions, against the actual policy, and flag any mismatch for a human to look at.

In every case the check encodes something only my team knows. That is the whole point of writing your own.

Pick the shape of the score before you write it

Before writing the check, decide what kind of answer it gives back:

Yes or no, for clean pass-or-fail rules ("did it read the right order number").
A category label, when you want to sort calls into buckets.
A score from 0 to 1, when quality is a gradient, like how closely the brand voice matched.

Pick the one that matches what you will do with it. A pass-rate dashboard wants yes or no. A trend line wants the gradient. Choosing the shape first saves you from rewriting the check later.

Two ways to actually write one

There are two honest paths, and they suit different people.

The first is to write the rule yourself, in plain English: spell out exactly what counts as a pass, what counts as a fail, and where the grey area is. Best when you, the engineer, already know the rule cold.

The second is to let a tool propose one. Some tooling reads your failed calls, groups the similar ones together, and drafts a candidate rule with example pass and fail cases pulled from real calls. You then edit and accept it. This is faster when the person reviewing is a domain expert rather than a coder.

Whichever path you take, keep one rule: a human approves the check before it goes live. Do not let anything auto-promote its own scoring into production.

The part that actually makes it work: calibrate against humans

This is the step I underestimated. Your first version of a check will score some calls wrong. Calibration is how you close that gap.

Build a small set of calls and have a domain expert label each one with the "true" score.
Run your check against that set and see where it disagrees with the expert.
Fix the wording or the examples, run it again, and repeat until it mostly agrees.

Plan for a few rounds, not one. And keep a human approving every change, with a record of who approved what. In anything regulated, that paper trail is the thing that makes the whole setup usable at all.

Pair the judge with a hard check whenever you can

A check that leans on a language model to judge will drift, and it quietly lets false passes pile up over time. So wherever a real, verifiable test exists, run it alongside the judge:

a number that has to fall in a valid range,
a format or schema that has to validate,
an exact entity that has to match.

The judge handles the fuzzy part, the hard check handles the part that has a right answer, and together they catch what either one alone would miss.

Re-run new versions on old calls

Every time I tweaked a check, I asked one question: how does the new version score the last month of calls compared to the old version? Re-running a check over your call history gives you that before-and-after diff directly. It is the easiest way to catch a "small" wording change that secretly moved everyone's scores.

Two tradeoffs I made on purpose

Calibration stays manual. A human signs off on every change to a check. It is slower, but it is the only version of this I actually trusted in a regulated workflow.
I only sample a slice of calls for the heavier scoring, to keep cost and privacy in check. Rare failures still surface, because grouping failed calls together brings the uncommon ones up even when you are not scoring everything.

The metric that finally caught my order-number problem was never going to be on a default dashboard. I had to write it, and then spend more time calibrating it against a human than writing it in the first place. That ratio surprised me, but it is also the reason the check ended up trustworthy.

If you run voice agents, I would love to hear which domain failure your standard metrics never caught. For me it will always be that confidently wrong order number.

How I A/B test LLM prompts without fooling myself

Kartik N V J K — Tue, 23 Jun 2026 19:22:48 +0000

A while back I was building a support assistant and hit a simple-sounding question: is this new version of the prompt actually better than the old one? So I did the obvious thing. I wrote thirty test cases, ran both prompts, saw the new one score a little higher, and shipped it.

I felt great about it for half a day, until the support queue filled with complaints and I was rolling the change back from a Slack thread that evening.

The bump in the score was never real. The test was far too small to tell a tiny genuine improvement apart from luck, so the number was noise the whole time. That one lesson is behind almost everything below. This is the guide I swear by now, in plain language, and I hope it saves you the same evening.

First, figure out how small a change your test can even see

A small test can only catch large differences. If the real improvement is small, thirty examples cannot tell it apart from the random wobble of which examples you happened to pick. Run it again and it often flips.

So before testing anything, I now answer two questions:

What is the smallest improvement that would actually be worth shipping?
Do I have enough examples to see an improvement that small?

The catch is how fast the numbers grow. To catch an improvement half as big, you need about four times as many examples. In practice, thirty examples can barely see anything under a ten-point swing. A four-point improvement took me a few hundred examples. A two-point one took over a thousand. Below a hundred, you are measuring your scorer's mood, not your prompt.

Test both versions on the very same inputs

My next mistake looked harmless: I gave version A one batch of questions and version B a different batch, then compared averages. But some questions are just harder. If B drew the easier batch, it looks better even when it is worse.

The fix is simple. Run both versions through the exact same questions and compare them one question at a time. The difficulty cancels out, because both faced the same hard ones, and all that is left is the real gap between the prompts.

This let me trust results from a few hundred examples that would otherwise have needed four times as many. The only cost is running every question twice, which is far cheaper than collecting four times the questions.

Put a range on the improvement, not a single average

A single average never told me enough, even with a "statistically significant" stamp. That stamp only says the difference probably is not exactly zero. It says nothing about how big it is, and the size is what decides whether I ship.

So I report a range: the smallest and largest the true improvement is likely to be.

If the range still includes zero, I do not have a real win, however nice the average looks.
If the whole range sits above the smallest improvement worth shipping, I ship.

One easy mistake to avoid: build the range from the per-question differences, keeping each question's two scores paired together. If you instead compare the two overall averages as if they were unrelated, you throw away the benefit of testing on the same inputs and the range balloons.

Score each answer alone, or judge the two side by side

There are two ways to score, and they answer different questions.

Score each answer on its own against a checklist (did it stay grounded, did it fully answer). Use this when you need an absolute number to hold against a bar you set.
Put both answers side by side and ask which is better. Use this when the quality is fuzzy, like tone or helpfulness, or when your scorer drifts run to run. Comparing two things directly is steadier than scoring each in a vacuum.

Simple rule: if the question is "is B better than A," judge side by side. If it is "does B clear the bar," score each one alone.

When you cannot wait for a full test, let the traffic pick the winner

Once I had three versions to compare, on a low-traffic feature, a normal test would have taken weeks. And the whole time, two of the three were probably worse and still going out to real people. Waiting a month while users got worse answers did not sit right.

So I used something with an odd name: a bandit. A normal test gives every version an equal share of traffic and waits until the end to call a winner, so half your users keep getting the weaker prompt the whole time. A bandit does not wait. It watches results as they come in and quietly sends more people to whatever is pulling ahead, and fewer to the rest. The longer it runs, the fewer users ever see the weak versions.

The name is just a leftover from gambling. A slot machine was a one-armed bandit, and this is the old puzzle of picking between machines when you do not yet know which pays out best. Swap machines for prompt versions and that was my situation.

The catch is that it gives up a clean number. Because it starves the weaker versions of traffic on purpose, you cannot say at the end that B beat A by exactly this much. So I only reach for a bandit when I have three or more versions, when even the weak ones are still usable, and when waiting on a full test costs more than a fuzzy final verdict. When I need a result I can defend to leadership, or it is just two versions head to head, I go back to the normal test.

Sometimes it is better to generate the variants, not hand-write one

For a long time my loop was: think hard, write one new version, test it, keep it if it won. The blind spot is that one hand-written version is a tiny slice of all the ways you could phrase the thing.

Once I had a scorer I trusted and labeled examples, I let a tool do the exploring. It generates many variations and searches for the ones that score well, far more than I would write by hand. Then I take its best candidate and run the same careful comparison against what is already in production. The tool covers ground I could not, and the A/B still makes the final call.

The mistakes that quietly ruin these tests

A few traps I fell into, so you do not have to:

An average with no range. A small bump on thirty examples is noise wearing a result's clothes.
A different batch of questions per version. You cannot tell a better prompt from an easier batch.
Swapping the scorer mid-test. That is a different test, not the same one. Lock it down first.
Peeking and stopping the moment it looks good. That manufactures wins that were never there. Decide when you will look before you run it.
Watching five "main" numbers at once. With five, there is about a one in four chance one looks like a win by pure luck. Pick one number and one bar up front.
Trusting the scorer before checking the scorer. If your automatic judge does not match careful human judgement, it can flip the result on its own noise.

That is the whole guide, and none of it is clever, which is sort of the point. The hard part was never running the test. It was learning when the result is real, and that comes down almost every time to whether I had enough examples to see what I was looking for. I hope having it in one place saves you an evening or two. If you have run these comparisons yourself, I would love to hear where your offline results and your live numbers parted ways. For me it is almost always how often the new prompt starts refusing things it should have answered.

How I set up RAG evals in CI/CD so they actually catch regressions

Kartik N V J K — Mon, 15 Jun 2026 17:40:15 +0000

I have hit this a few times.A PR lands late in the day, the RAG eval runs in under a minute, green check, merge.Twelve hours later support tickets start coming in.

The trace shows the retriever switched its top-1 chunk on a class of queries the 30-example dataset never covered. Suite Groundedness stayed at 0.91. Production Groundedness on the affected traffic was 0.62.

The gate passed because it was not checking the right thing.Most CI eval gates I have seen for RAG are smoke tests. Small dataset, mean compared against a fixed floor, pass unless something is badly broken.

The dataset is not representative, the floor is not tied to baseline variance, and the threshold does not separate a real regression from judge noise.So a green check does not tell you much.

The way I think about it now: a gate wants three things at once -cheap, fast, and statistically significant and you usually get two.
A 30-example dataset at 12 cents per PR fails on significance.A 2,000-example sweep at 9 dollars per PR fails on cost and speed.
The work is holding all three.

I split the gate into three tiers

Running the full LLM-judge sweep on every push was my first mistake.That belongs on nightly main, not an in-progress branch.

Every push:

Cheap classifier rubrics (NLI faithfulness, claim_support) plus deterministic floors (citation validity, schema, latency).
Under three minutes against 100 to 200 examples.
Blocks the merge.

Nightly main:

The full LLM-judge stack against the versioned dataset.
15 to 30 minutes.
Blocks the next promotion to canary.

Canary:

The same rubrics scoring 5 to 10 percent of live traffic.
Alarms on rolling-mean drift.

The dataset is the gate's worldview

A 2,000-example set built from my own head loses to a 200-example set sampled from production.If it misses the failure modes that show up at 2 am, so does the gate.
Below 100 examples per route the variance drowns the signal.
Above 500 the per-PR judge bill grows faster than detection improves.My PR range is 100 to 200 per route, covering happy paths, edge cases, refusals, and the hardest 10 percent of past incidents.
The field most people skip is expected_chunks, the ground-truth doc IDs.
Without it you can score generation but not retrieval, and the bisect takes a day instead of an hour.

Five rubrics cover most regressions

The five rubrics that gate most RAG regressions in CI:

Groundedness
Context Relevance
Answer Relevance
Citation Validity
Retrieval Recall

I split them by layer so the bisect is easy:

ContextRelevance drops while Groundedness holds → retriever regressed.
Groundedness drops while ContextRelevance holds → generator regressed.

Citation validity is just a string match, so I run it on 100 percent of responses and keep the judge bill on the rubrics that need semantic scoring.

Gating on the delta, not the mean

This is the part most setups skip and the one that matters.

A green check should mean the PR did not introduce a statistically significant regression, not that the mean sat above some floor.

I use two thresholds.

An absolute floor per rubric catches the obvious break:

Groundedness ≥ 0.85
ContextRelevance ≥ 0.80
Citation validity ≥ 0.99

A delta gate against the trailing 7-day rolling baseline catches the slow drift, using Welch's t-test on per-example scores.

import statistics
from scipy import stats

def regression_gate(current, baseline, alpha=0.05, min_effect=0.03):
    """Fail only if the mean dropped, the change is significant, and the effect is big enough."""
    delta = statistics.mean(current) - statistics.mean(baseline)

    if delta >= 0:
        return True, f"no regression (delta=+{delta:.3f})"

    _, p = stats.ttest_ind(current, baseline, equal_var=False)

    if p >= alpha:
        return True, f"delta={delta:.3f}, p={p:.3f} (not significant)"

    if abs(delta) < min_effect:
        return True, f"delta={delta:.3f} below effect floor {min_effect}"

    return False, f"regression: delta={delta:.3f}, p={p:.3f}"

A 30-example dataset gives a confidence interval of about ±0.07 on a 1–5 mean, so a 2-point drop sits inside the noise.

Gate on that and you are wrong half the time, and after two false alarms people stop trusting the gate.

For long-tail failures that hide in averages, gate on percentiles instead.

The one rule that keeps this honest:

The baseline is a rolling production window, not a frozen number.

The GitHub Actions wiring

The workflow itself is path-scoped triggers, a cache, parallel pytest, and the cheap-rubric assertions.

This is roughly what I run on PRs.

name: RAG Evals

on:
  pull_request:
    paths:
      - "rag/**"
      - "evals/**"
      - "fi-evaluation.yaml"

concurrency:
  group: rag-evals-${{ github.head_ref }}
  cancel-in-progress: true

jobs:
  pr-gate:
    runs-on: ubuntu-latest
    timeout-minutes: 8

    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 2

      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
          cache: pip

      - uses: actions/cache@v4
        with:
          path: .eval_cache
          key: evals-${{ hashFiles('evals/rubrics/**', 'evals/datasets/**') }}

      - run: pip install -r requirements.txt

      - id: routes
        run: echo "routes=$(python evals/affected_routes.py)" >> "$GITHUB_OUTPUT"

      - name: Cheap-rubric gate
        if: steps.routes.outputs.routes != '[]'
        env:
          FI_API_KEY: ${{ secrets.FI_API_KEY }}
          FI_SECRET_KEY: ${{ secrets.FI_SECRET_KEY }}
        run: fi run --check --strict --parallel 16 -c evals/fi-evaluation.yaml

      - name: Statistical delta gate
        if: steps.routes.outputs.routes != '[]'
        env:
          FI_API_KEY: ${{ secrets.FI_API_KEY }}
          FI_SECRET_KEY: ${{ secrets.FI_SECRET_KEY }}
          BASELINE_WINDOW_DAYS: "7"
        run: pytest evals/test_rag.py -n auto --routes='${{ steps.routes.outputs.routes }}'

      - if: always()
        uses: actions/upload-artifact@v4
        with:
          name: eval-report-${{ github.sha }}
          path: eval-report.json

A separate nightly cron workflow runs the full sweep across all routes and posts the daily baseline back to the observer.

The same shape works on GitLab CI, Buildkite, Jenkins, or CircleCI, since it is just pytest, a CLI, and a cache action.

Three things paid back in the first week:

Path-scoped triggers
Concurrency cancel-in-progress
Eval report artifacts with failing examples

Those turned borderline regressions into short conversations instead of arguments.

Running the same rubric in production

Offline CI catches the regressions I can think of.Production catches the rest.So I run the same rubric definition in both places.The CI version lives in code.

The production version runs against live OTel spans as a score attached to the trace.

from fi_instrumentation import register
from fi_instrumentation.fi_types import (
    EvalTag,
    EvalTagType,
    EvalSpanKind,
    EvalName,
    ProjectType,
)

eval_tags = [
    EvalTag(
        eval_name=EvalName.GROUNDEDNESS,
        type=EvalTagType.OBSERVATION_SPAN,
        value=EvalSpanKind.LLM,
        config={},
        mapping={
            "context": "retrieval.documents",
            "output": "output.value",
        },
    ),
    EvalTag(
        eval_name=EvalName.CONTEXT_RELEVANCE,
        type=EvalTagType.OBSERVATION_SPAN,
        value=EvalSpanKind.RETRIEVER,
        config={},
        mapping={
            "input": "input.value",
            "context": "retrieval.documents",
        },
    ),
]

register(
    project_type=ProjectType.OBSERVE,
    project_name="legal-rag",
    eval_tags=eval_tags,
)

The score lands as a span attribute, so a failing trace shows up with its rubric score next to latency and chunk IDs.

I sample 5 to 10 percent of production traffic for the LLM-judge rubrics and run the cheap ones on 100 percent.

Then I alarm on a sustained 2 to 5 point drop in a rubric's rolling mean over a 15 to 60 minute window.

The gap between the CI baseline and the production rolling mean is its own signal.

When it widens, the dataset has stopped being representative.

Closing the loop so the dataset keeps up

A dataset stops being a regression suite the moment production drifts past it.
The loop is what keeps the gate honest.
Failing production traces get clustered into named issues.
An analysis pass writes up the root cause and a fix for each one.
The worst representative traces get promoted into the eval set with rubric labels.
The next PR touching that path either clears the new entries or fails on them.
Over a few weeks the gate gets stronger instead of decaying because it learns the failures that actually happened rather than the ones I guessed at during setup.

Pitfalls I watch for

Scoring only Groundedness

Catches hallucinations.
Misses retrieval regressions.
Run all five core rubrics.

No expected_chunks in the dataset

You cannot score retrieval recall without ground truth.
The labeling pays back the first time retrieval breaks.

A floating judge model

Scores drift across runs of the same eval.
Pin and version the judge alongside the rubric.

Floors with no delta gate

Slow regressions stay under the floor for months.
The delta gate is what catches them.

A 30-example dataset with a mean gate
Variance wider than the regressions you are chasing.
Grow the set or gate on percentiles.

The full LLM-judge sweep on every PR

Too slow and too expensive.
Cheap rubrics on PRs.
Heavy sweeps nightly.

A dataset frozen at launch

A 2024 set scoring a 2026 product is a benchmark, not a regression suite.

No cache

Cost climbs and flaky network calls block PRs.
Cache verdicts.
Invalidate on rubric or judge version change.

A production observer using a different rubric than CI

Then people argue about which number is real instead of fixing the bug.
One definition.
Run it in both places.

Tradeoffs I made on purpose

Statistical gating slows the first week

Building the baseline window takes a few nightly runs before the delta gate means anything.
The payoff is that a red PR starts meaning a real regression, which is the only reason people leave the gate on.

Sharded, path-scoped runs cost orchestration

Route-aware test selection and a distributed runner are extra wiring.
The return is the difference between a four-minute and a thirty-second PR gate.

The classifier cascade needs rubric discipline

Cascades work where the classifier has a clean target.
They fall apart on subjective axes.

- Deciding which rubrics cascade is a one-time design call you have to actually make.

If you have built a gate like this, I am curious which of the three corners gave you the most trouble.

For me it was significance, every time.