DEV Community: Kartik N V J K

How I caught the voice-agent failures my eval dashboard kept missing

Kartik N V J K — Tue, 30 Jun 2026 06:59:33 +0000

I was working on a retail support voice agent, and on paper it looked great. Transcription quality, green. Conversation coherence, green. Task completion, green. Every standard metric I had was happy.

The problem was that it kept getting order numbers wrong. It would confidently read back the wrong number, the call would still "complete," and my dashboard would still show green. The checks I had simply did not know what an order number was, or why getting it wrong was the whole ballgame.

That is the gap I want to write about. The default metrics catch the failures every voice agent shares. They cannot catch the failures that belong to your specific domain. Here is what I learned about writing checks that do.

Built-in metrics catch the universal failures, not yours

Every voice agent, in any business, can fail in the same handful of ways:

Did it transcribe what the user said correctly?
Did the conversation stay coherent across turns?
Did it actually finish the task by the end of the call?
Did it stay safe, no leaked personal data, no obvious prompt injection?

Off-the-shelf checks handle all of that well. What they cannot know is your taxonomy. A retail agent lives or dies on order numbers. A clinical scribe lives or dies on drug names. An insurance agent lives or dies on coverage codes. None of that is in a generic metric, so none of it shows up on a generic dashboard.

Know when to reach for a custom check

The rule I settled on is simple:

If the failure is universal, use a built-in check.
If the failure is specific to your domain, write a custom one.
In production, run both on every call.

The last point matters more than it sounds. The two layers catch completely different kinds of errors. A call can pass the universal checks and still fail the domain one, which is exactly what was happening to me. You want both signals, not one.

What a custom check actually looks like

It helps to see a few real ones, in plain terms, no code:

Drug names for a medical scribe. Not just "was the word spelled right," but did the drug, the dose, the frequency, and the route all survive together. Getting three of four right is still a dangerous note.
Brand voice for a marketing line. Did it open with the approved greeting, avoid the banned phrases, and keep the right tone. A coherent call can still sound nothing like the brand.
Quote correctness for an insurance claim. Check each field, the deductible, the limit, the exclusions, against the actual policy, and flag any mismatch for a human to look at.

In every case the check encodes something only my team knows. That is the whole point of writing your own.

Pick the shape of the score before you write it

Before writing the check, decide what kind of answer it gives back:

Yes or no, for clean pass-or-fail rules ("did it read the right order number").
A category label, when you want to sort calls into buckets.
A score from 0 to 1, when quality is a gradient, like how closely the brand voice matched.

Pick the one that matches what you will do with it. A pass-rate dashboard wants yes or no. A trend line wants the gradient. Choosing the shape first saves you from rewriting the check later.

Two ways to actually write one

There are two honest paths, and they suit different people.

The first is to write the rule yourself, in plain English: spell out exactly what counts as a pass, what counts as a fail, and where the grey area is. Best when you, the engineer, already know the rule cold.

The second is to let a tool propose one. Some tooling reads your failed calls, groups the similar ones together, and drafts a candidate rule with example pass and fail cases pulled from real calls. You then edit and accept it. This is faster when the person reviewing is a domain expert rather than a coder.

Whichever path you take, keep one rule: a human approves the check before it goes live. Do not let anything auto-promote its own scoring into production.

The part that actually makes it work: calibrate against humans

This is the step I underestimated. Your first version of a check will score some calls wrong. Calibration is how you close that gap.

Build a small set of calls and have a domain expert label each one with the "true" score.
Run your check against that set and see where it disagrees with the expert.
Fix the wording or the examples, run it again, and repeat until it mostly agrees.

Plan for a few rounds, not one. And keep a human approving every change, with a record of who approved what. In anything regulated, that paper trail is the thing that makes the whole setup usable at all.

Pair the judge with a hard check whenever you can

A check that leans on a language model to judge will drift, and it quietly lets false passes pile up over time. So wherever a real, verifiable test exists, run it alongside the judge:

a number that has to fall in a valid range,
a format or schema that has to validate,
an exact entity that has to match.

The judge handles the fuzzy part, the hard check handles the part that has a right answer, and together they catch what either one alone would miss.

Re-run new versions on old calls

Every time I tweaked a check, I asked one question: how does the new version score the last month of calls compared to the old version? Re-running a check over your call history gives you that before-and-after diff directly. It is the easiest way to catch a "small" wording change that secretly moved everyone's scores.

Two tradeoffs I made on purpose

Calibration stays manual. A human signs off on every change to a check. It is slower, but it is the only version of this I actually trusted in a regulated workflow.
I only sample a slice of calls for the heavier scoring, to keep cost and privacy in check. Rare failures still surface, because grouping failed calls together brings the uncommon ones up even when you are not scoring everything.

The metric that finally caught my order-number problem was never going to be on a default dashboard. I had to write it, and then spend more time calibrating it against a human than writing it in the first place. That ratio surprised me, but it is also the reason the check ended up trustworthy.

If you run voice agents, I would love to hear which domain failure your standard metrics never caught. For me it will always be that confidently wrong order number.

How I A/B test LLM prompts without fooling myself

Kartik N V J K — Tue, 23 Jun 2026 19:22:48 +0000

A while back I was building a support assistant and hit a simple-sounding question: is this new version of the prompt actually better than the old one? So I did the obvious thing. I wrote thirty test cases, ran both prompts, saw the new one score a little higher, and shipped it.

I felt great about it for half a day, until the support queue filled with complaints and I was rolling the change back from a Slack thread that evening.

The bump in the score was never real. The test was far too small to tell a tiny genuine improvement apart from luck, so the number was noise the whole time. That one lesson is behind almost everything below. This is the guide I swear by now, in plain language, and I hope it saves you the same evening.

First, figure out how small a change your test can even see

A small test can only catch large differences. If the real improvement is small, thirty examples cannot tell it apart from the random wobble of which examples you happened to pick. Run it again and it often flips.

So before testing anything, I now answer two questions:

What is the smallest improvement that would actually be worth shipping?
Do I have enough examples to see an improvement that small?

The catch is how fast the numbers grow. To catch an improvement half as big, you need about four times as many examples. In practice, thirty examples can barely see anything under a ten-point swing. A four-point improvement took me a few hundred examples. A two-point one took over a thousand. Below a hundred, you are measuring your scorer's mood, not your prompt.

Test both versions on the very same inputs

My next mistake looked harmless: I gave version A one batch of questions and version B a different batch, then compared averages. But some questions are just harder. If B drew the easier batch, it looks better even when it is worse.

The fix is simple. Run both versions through the exact same questions and compare them one question at a time. The difficulty cancels out, because both faced the same hard ones, and all that is left is the real gap between the prompts.

This let me trust results from a few hundred examples that would otherwise have needed four times as many. The only cost is running every question twice, which is far cheaper than collecting four times the questions.

Put a range on the improvement, not a single average

A single average never told me enough, even with a "statistically significant" stamp. That stamp only says the difference probably is not exactly zero. It says nothing about how big it is, and the size is what decides whether I ship.

So I report a range: the smallest and largest the true improvement is likely to be.

If the range still includes zero, I do not have a real win, however nice the average looks.
If the whole range sits above the smallest improvement worth shipping, I ship.

One easy mistake to avoid: build the range from the per-question differences, keeping each question's two scores paired together. If you instead compare the two overall averages as if they were unrelated, you throw away the benefit of testing on the same inputs and the range balloons.

Score each answer alone, or judge the two side by side

There are two ways to score, and they answer different questions.

Score each answer on its own against a checklist (did it stay grounded, did it fully answer). Use this when you need an absolute number to hold against a bar you set.
Put both answers side by side and ask which is better. Use this when the quality is fuzzy, like tone or helpfulness, or when your scorer drifts run to run. Comparing two things directly is steadier than scoring each in a vacuum.

Simple rule: if the question is "is B better than A," judge side by side. If it is "does B clear the bar," score each one alone.

When you cannot wait for a full test, let the traffic pick the winner

Once I had three versions to compare, on a low-traffic feature, a normal test would have taken weeks. And the whole time, two of the three were probably worse and still going out to real people. Waiting a month while users got worse answers did not sit right.

So I used something with an odd name: a bandit. A normal test gives every version an equal share of traffic and waits until the end to call a winner, so half your users keep getting the weaker prompt the whole time. A bandit does not wait. It watches results as they come in and quietly sends more people to whatever is pulling ahead, and fewer to the rest. The longer it runs, the fewer users ever see the weak versions.

The name is just a leftover from gambling. A slot machine was a one-armed bandit, and this is the old puzzle of picking between machines when you do not yet know which pays out best. Swap machines for prompt versions and that was my situation.

The catch is that it gives up a clean number. Because it starves the weaker versions of traffic on purpose, you cannot say at the end that B beat A by exactly this much. So I only reach for a bandit when I have three or more versions, when even the weak ones are still usable, and when waiting on a full test costs more than a fuzzy final verdict. When I need a result I can defend to leadership, or it is just two versions head to head, I go back to the normal test.

Sometimes it is better to generate the variants, not hand-write one

For a long time my loop was: think hard, write one new version, test it, keep it if it won. The blind spot is that one hand-written version is a tiny slice of all the ways you could phrase the thing.

Once I had a scorer I trusted and labeled examples, I let a tool do the exploring. It generates many variations and searches for the ones that score well, far more than I would write by hand. Then I take its best candidate and run the same careful comparison against what is already in production. The tool covers ground I could not, and the A/B still makes the final call.

The mistakes that quietly ruin these tests

A few traps I fell into, so you do not have to:

An average with no range. A small bump on thirty examples is noise wearing a result's clothes.
A different batch of questions per version. You cannot tell a better prompt from an easier batch.
Swapping the scorer mid-test. That is a different test, not the same one. Lock it down first.
Peeking and stopping the moment it looks good. That manufactures wins that were never there. Decide when you will look before you run it.
Watching five "main" numbers at once. With five, there is about a one in four chance one looks like a win by pure luck. Pick one number and one bar up front.
Trusting the scorer before checking the scorer. If your automatic judge does not match careful human judgement, it can flip the result on its own noise.

That is the whole guide, and none of it is clever, which is sort of the point. The hard part was never running the test. It was learning when the result is real, and that comes down almost every time to whether I had enough examples to see what I was looking for. I hope having it in one place saves you an evening or two. If you have run these comparisons yourself, I would love to hear where your offline results and your live numbers parted ways. For me it is almost always how often the new prompt starts refusing things it should have answered.

How I set up RAG evals in CI/CD so they actually catch regressions

Kartik N V J K — Mon, 15 Jun 2026 17:40:15 +0000

I have hit this a few times.A PR lands late in the day, the RAG eval runs in under a minute, green check, merge.Twelve hours later support tickets start coming in.

The trace shows the retriever switched its top-1 chunk on a class of queries the 30-example dataset never covered. Suite Groundedness stayed at 0.91. Production Groundedness on the affected traffic was 0.62.

The gate passed because it was not checking the right thing.Most CI eval gates I have seen for RAG are smoke tests. Small dataset, mean compared against a fixed floor, pass unless something is badly broken.

The dataset is not representative, the floor is not tied to baseline variance, and the threshold does not separate a real regression from judge noise.So a green check does not tell you much.

The way I think about it now: a gate wants three things at once -cheap, fast, and statistically significant and you usually get two.
A 30-example dataset at 12 cents per PR fails on significance.A 2,000-example sweep at 9 dollars per PR fails on cost and speed.
The work is holding all three.

I split the gate into three tiers

Running the full LLM-judge sweep on every push was my first mistake.That belongs on nightly main, not an in-progress branch.

Every push:

Cheap classifier rubrics (NLI faithfulness, claim_support) plus deterministic floors (citation validity, schema, latency).
Under three minutes against 100 to 200 examples.
Blocks the merge.

Nightly main:

The full LLM-judge stack against the versioned dataset.
15 to 30 minutes.
Blocks the next promotion to canary.

Canary:

The same rubrics scoring 5 to 10 percent of live traffic.
Alarms on rolling-mean drift.

The dataset is the gate's worldview

A 2,000-example set built from my own head loses to a 200-example set sampled from production.If it misses the failure modes that show up at 2 am, so does the gate.
Below 100 examples per route the variance drowns the signal.
Above 500 the per-PR judge bill grows faster than detection improves.My PR range is 100 to 200 per route, covering happy paths, edge cases, refusals, and the hardest 10 percent of past incidents.
The field most people skip is expected_chunks, the ground-truth doc IDs.
Without it you can score generation but not retrieval, and the bisect takes a day instead of an hour.

Five rubrics cover most regressions

The five rubrics that gate most RAG regressions in CI:

Groundedness
Context Relevance
Answer Relevance
Citation Validity
Retrieval Recall

I split them by layer so the bisect is easy:

ContextRelevance drops while Groundedness holds → retriever regressed.
Groundedness drops while ContextRelevance holds → generator regressed.

Citation validity is just a string match, so I run it on 100 percent of responses and keep the judge bill on the rubrics that need semantic scoring.

Gating on the delta, not the mean

This is the part most setups skip and the one that matters.

A green check should mean the PR did not introduce a statistically significant regression, not that the mean sat above some floor.

I use two thresholds.

An absolute floor per rubric catches the obvious break:

Groundedness ≥ 0.85
ContextRelevance ≥ 0.80
Citation validity ≥ 0.99

A delta gate against the trailing 7-day rolling baseline catches the slow drift, using Welch's t-test on per-example scores.

import statistics
from scipy import stats

def regression_gate(current, baseline, alpha=0.05, min_effect=0.03):
    """Fail only if the mean dropped, the change is significant, and the effect is big enough."""
    delta = statistics.mean(current) - statistics.mean(baseline)

    if delta >= 0:
        return True, f"no regression (delta=+{delta:.3f})"

    _, p = stats.ttest_ind(current, baseline, equal_var=False)

    if p >= alpha:
        return True, f"delta={delta:.3f}, p={p:.3f} (not significant)"

    if abs(delta) < min_effect:
        return True, f"delta={delta:.3f} below effect floor {min_effect}"

    return False, f"regression: delta={delta:.3f}, p={p:.3f}"

A 30-example dataset gives a confidence interval of about ±0.07 on a 1–5 mean, so a 2-point drop sits inside the noise.

Gate on that and you are wrong half the time, and after two false alarms people stop trusting the gate.

For long-tail failures that hide in averages, gate on percentiles instead.

The one rule that keeps this honest:

The baseline is a rolling production window, not a frozen number.

The GitHub Actions wiring

The workflow itself is path-scoped triggers, a cache, parallel pytest, and the cheap-rubric assertions.

This is roughly what I run on PRs.

name: RAG Evals

on:
  pull_request:
    paths:
      - "rag/**"
      - "evals/**"
      - "fi-evaluation.yaml"

concurrency:
  group: rag-evals-${{ github.head_ref }}
  cancel-in-progress: true

jobs:
  pr-gate:
    runs-on: ubuntu-latest
    timeout-minutes: 8

    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 2

      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
          cache: pip

      - uses: actions/cache@v4
        with:
          path: .eval_cache
          key: evals-${{ hashFiles('evals/rubrics/**', 'evals/datasets/**') }}

      - run: pip install -r requirements.txt

      - id: routes
        run: echo "routes=$(python evals/affected_routes.py)" >> "$GITHUB_OUTPUT"

      - name: Cheap-rubric gate
        if: steps.routes.outputs.routes != '[]'
        env:
          FI_API_KEY: ${{ secrets.FI_API_KEY }}
          FI_SECRET_KEY: ${{ secrets.FI_SECRET_KEY }}
        run: fi run --check --strict --parallel 16 -c evals/fi-evaluation.yaml

      - name: Statistical delta gate
        if: steps.routes.outputs.routes != '[]'
        env:
          FI_API_KEY: ${{ secrets.FI_API_KEY }}
          FI_SECRET_KEY: ${{ secrets.FI_SECRET_KEY }}
          BASELINE_WINDOW_DAYS: "7"
        run: pytest evals/test_rag.py -n auto --routes='${{ steps.routes.outputs.routes }}'

      - if: always()
        uses: actions/upload-artifact@v4
        with:
          name: eval-report-${{ github.sha }}
          path: eval-report.json

A separate nightly cron workflow runs the full sweep across all routes and posts the daily baseline back to the observer.

The same shape works on GitLab CI, Buildkite, Jenkins, or CircleCI, since it is just pytest, a CLI, and a cache action.

Three things paid back in the first week:

Path-scoped triggers
Concurrency cancel-in-progress
Eval report artifacts with failing examples

Those turned borderline regressions into short conversations instead of arguments.

Running the same rubric in production

Offline CI catches the regressions I can think of.Production catches the rest.So I run the same rubric definition in both places.The CI version lives in code.

The production version runs against live OTel spans as a score attached to the trace.

from fi_instrumentation import register
from fi_instrumentation.fi_types import (
    EvalTag,
    EvalTagType,
    EvalSpanKind,
    EvalName,
    ProjectType,
)

eval_tags = [
    EvalTag(
        eval_name=EvalName.GROUNDEDNESS,
        type=EvalTagType.OBSERVATION_SPAN,
        value=EvalSpanKind.LLM,
        config={},
        mapping={
            "context": "retrieval.documents",
            "output": "output.value",
        },
    ),
    EvalTag(
        eval_name=EvalName.CONTEXT_RELEVANCE,
        type=EvalTagType.OBSERVATION_SPAN,
        value=EvalSpanKind.RETRIEVER,
        config={},
        mapping={
            "input": "input.value",
            "context": "retrieval.documents",
        },
    ),
]

register(
    project_type=ProjectType.OBSERVE,
    project_name="legal-rag",
    eval_tags=eval_tags,
)

The score lands as a span attribute, so a failing trace shows up with its rubric score next to latency and chunk IDs.

I sample 5 to 10 percent of production traffic for the LLM-judge rubrics and run the cheap ones on 100 percent.

Then I alarm on a sustained 2 to 5 point drop in a rubric's rolling mean over a 15 to 60 minute window.

The gap between the CI baseline and the production rolling mean is its own signal.

When it widens, the dataset has stopped being representative.

Closing the loop so the dataset keeps up

A dataset stops being a regression suite the moment production drifts past it.
The loop is what keeps the gate honest.
Failing production traces get clustered into named issues.
An analysis pass writes up the root cause and a fix for each one.
The worst representative traces get promoted into the eval set with rubric labels.
The next PR touching that path either clears the new entries or fails on them.
Over a few weeks the gate gets stronger instead of decaying because it learns the failures that actually happened rather than the ones I guessed at during setup.

Pitfalls I watch for

Scoring only Groundedness

Catches hallucinations.
Misses retrieval regressions.
Run all five core rubrics.

No expected_chunks in the dataset

You cannot score retrieval recall without ground truth.
The labeling pays back the first time retrieval breaks.

A floating judge model

Scores drift across runs of the same eval.
Pin and version the judge alongside the rubric.

Floors with no delta gate

Slow regressions stay under the floor for months.
The delta gate is what catches them.

A 30-example dataset with a mean gate
Variance wider than the regressions you are chasing.
Grow the set or gate on percentiles.

The full LLM-judge sweep on every PR

Too slow and too expensive.
Cheap rubrics on PRs.
Heavy sweeps nightly.

A dataset frozen at launch

A 2024 set scoring a 2026 product is a benchmark, not a regression suite.

No cache

Cost climbs and flaky network calls block PRs.
Cache verdicts.
Invalidate on rubric or judge version change.

A production observer using a different rubric than CI

Then people argue about which number is real instead of fixing the bug.
One definition.
Run it in both places.

Tradeoffs I made on purpose

Statistical gating slows the first week

Building the baseline window takes a few nightly runs before the delta gate means anything.
The payoff is that a red PR starts meaning a real regression, which is the only reason people leave the gate on.

Sharded, path-scoped runs cost orchestration

Route-aware test selection and a distributed runner are extra wiring.
The return is the difference between a four-minute and a thirty-second PR gate.

The classifier cascade needs rubric discipline

Cascades work where the classifier has a clean target.
They fall apart on subjective axes.

- Deciding which rubrics cascade is a one-time design call you have to actually make.

If you have built a gate like this, I am curious which of the three corners gave you the most trouble.

For me it was significance, every time.