DEV Community: Ethan Walker

356 of the 528 eval cases we could judge have never failed. Our pass rate could only move 21 points.

Ethan Walker — Thu, 13 Aug 2026 19:28:20 +0000

TL;DR: I broke out per-case results for the incident-harvested part of our eval suite, 611 of our 1,400 cases, and asked which cases have ever discriminated between two shipped versions. Of the 528 with enough history to judge, 356 had passed every version and 59 had failed every version. 113 cases were carrying all of the signal. That is not a bug in harvesting, it is what harvesting does, and it is measurable.

Yesterday I published that our eval gate caught 23 of 41 known regressions when we back-tested it (post). This is the other half of that measurement. That one asked what the gate misses. This one asks which cases were ever going to fire at all.

The breakdown I did not have

Our suite is about 1,400 cases. 611 of them are the behavioural set, the ones harvested from production incidents rather than written by hand, which I wrote about starting in June. That set has grown fast, because it grows every time something breaks.

We stored a pass rate per run. We did not store a result per case per version, so I could not answer the question a colleague asked me in a review: when the number moves two points, which cases moved?

Adding it was an afternoon. Then a second pass, because the first one was wrong: I bucketed every case including ones added last month, and a case that has only ever seen three versions has passed all of them for uninteresting reasons. Requiring at least five shipped versions of history drops 83 cases as too new to judge.

Of the 528 that remain, across 23 versions shipped since April:

356 passed every version. Never once failed.
59 failed every version. Known-failing, mostly things we decided not to fix.
113 actually varied.

So 21 percent of the judged set was carrying the signal. A case that every version passes has no discriminating power, because power here just means the ability to come out differently on two runs.

Why a harvested set rots this way

This is a consequence of the harvesting idea, and the mechanism is dull.

You harvest a case from an incident. At the moment you write it, the case fails, because the bug is live. You fix the bug. From the next version on, the case passes. Forever.

The case did its job once, on the day it was written. After that it is a regression guard, and a regression guard is worth its runtime only if the regression can plausibly come back. Some can. Most of ours could not, because the fix was a schema change or a prompt restructure that nothing was going to walk back.

So the set accumulates cases at the rate you have incidents, and it accumulates cases that can no longer discriminate at very nearly the same rate, one fix behind.

The 356: six minutes and no headroom

Runtime. Those 356 cases are roughly 6 minutes of the batched suite's 22. Not fatal on its own, but it is 6 minutes spent re-confirming things that have not changed since April.

A reported number with almost nowhere to go. This is the one that bothers me, and it needs a denominator stated out loud, because I have quoted three different ones in past posts.

One 94 was 481 of 512 cases, on the run that shipped a PII leak. A second 94 was our ticket-routing eval over 600 rows, and that one turned out to be manufactured, because the few-shot index was built from the eval set. The 96 to 97 was an intent-classification eval sitting above a 90 percent gate, one eval, not a suite. None of the three is this number, and the suite has roughly tripled since. The number in this section is the behavioural set's own pass rate, over the 528 cases with enough version history to judge. Different denominators, different scales, and mixing them is how you end up comparing a 96 to an 89 and concluding something regressed.

On that 528: with 356 guaranteed passes the floor was 67.4 percent before any model did anything, and with 59 guaranteed failures the ceiling was 88.8. The entire achievable range was 21.4 points, and we spent the quarter inside a 4 point window near the top of it, between 84 and 88.

That 21.4 is not a coincidence: the achievable range in points is the discriminating share, 113 of 528 either way. Every case you add that everything passes shortens it by one.

That holds whatever your numbers look like. I have argued before that aggregates hide individual regressions, and this is the prior question, which is how much room the aggregate had to begin with.

What I got wrong first

My first move was to delete the 356. My colleague caught it in review and was right.

Some of those cases guard behaviour that is currently correct because the guard exists. No version failing them since April is evidence the guard works, not evidence the case is useless. Deleting a PII redaction case because it has always passed is how you find out in November that it stopped passing in September.

What we did instead was use discrimination as a tiering criterion, for the tiers we already had rather than a new structure:

113 discriminating plus 59 known-failing stay in the reporting tier from yesterday's back-test post, the one that scores but does not block.
356 always-pass move to the nightly tier, minus those of the 61 already in the blocking tier, which stay where they are: that tier blocks on a case flipping pass to fail and does not care whether the case has ever flipped before.
Re-score discrimination monthly, because a dead case revives when the code around it changes. Three did in the first month.

The batched merge suite lost about 6 minutes, on the rough assumption that per-case cost is uniform. It is not: the 356 skew deterministic and the judge-scored cases dominate wall clock, so treat 6 as an upper bound rather than a measurement. Nothing was deleted.

What happened to the reported number

We now report the pass rate over the 113 discriminating cases, so the constant block is out of the denominator.

The number did not get lower. It got wider. On the old denominator our quarter sat between 84 and 88 percent; recomputed over the 113, the same quarter swings between roughly 77 and 96. The fixed point is around 86, so it moved down about as often as up.

That is correct, and it took some explaining. The wider number moves when the model moves, which the old one could not do.

The script

Feed it per-case results per version. If you do not store those, that is the actual first task.

import collections

# (version, case_id, passed): substitute your own rows
results = [
    ("v1", "case_a", True),  ("v2", "case_a", True),  ("v3", "case_a", True),
    ("v4", "case_a", True),  ("v5", "case_a", True),
    ("v1", "case_b", True),  ("v2", "case_b", False), ("v3", "case_b", True),
    ("v4", "case_b", True),  ("v5", "case_b", False),
    ("v1", "case_c", False), ("v2", "case_c", False), ("v3", "case_c", False),
    ("v4", "case_c", False), ("v5", "case_c", False),
    # too new to judge: only 3 versions of history, must land in neither bucket
    ("v3", "case_d", True),  ("v4", "case_d", True),  ("v5", "case_d", False),
]

MIN_HISTORY = 5   # versions a case must have seen before we judge it

by_case = collections.defaultdict(list)
for _, case_id, passed in results:
    by_case[case_id].append(passed)

always_pass, always_fail, discriminating, too_new = [], [], [], []
for case_id, outcomes in by_case.items():
    if len(outcomes) < MIN_HISTORY:
        too_new.append(case_id)
    elif all(outcomes):
        always_pass.append(case_id)
    elif not any(outcomes):
        always_fail.append(case_id)
    else:
        discriminating.append(case_id)

judged = len(always_pass) + len(always_fail) + len(discriminating)
if not judged:
    raise SystemExit("no case has enough history yet; lower MIN_HISTORY or wait")

for name, bucket in [("always pass", always_pass), ("always fail", always_fail),
                     ("discriminating", discriminating)]:
    print(f"{name:15s} {len(bucket):4d}  ({len(bucket)/judged:.0%} of judged)")
print(f"{'too new':15s} {len(too_new):4d}  (excluded)")
print(f"achievable range: {len(discriminating)/judged:.1%} of the scale")

MIN_HISTORY is the line that matters, which is why case_d is in the sample: three versions of history, so it lands in too_new and in none of the three buckets. Without that guard every case added last month reads as dead, and the number that comes back is flattering rather than true. That was my first run.

One thing I have not resolved: this treats all 23 versions as equally informative and they are not. Several were small prompt edits that were never going to move most cases, so a case can look non-discriminating because nothing asked it a hard question. Weighting by how much each version actually changed is the right fix and I do not have a defensible way to do it yet.

What I'd check first

Do you store a result per case per version, or only an aggregate per run? If it is the second, none of this is answerable, and it is an afternoon to fix.
What fraction of your set has ever failed, counting only cases with real version history? That fraction is your metric's entire range.
What is your floor, the number you would report if every discriminating case failed? If it is high, your headline has less room than it looks.

We back-tested our eval gate against 41 known regressions. It caught 23.

Ethan Walker — Tue, 11 Aug 2026 16:49:15 +0000

The gate had been green for eleven weeks. In the same eleven weeks we rolled back two prompt changes, hotfixed a retrieval config on a Saturday, and shipped a summarizer that started dropping the second half of long documents. Nobody was confused about whether the gate was working. Everyone assumed it was, because it was the thing standing between a pull request and production, and it kept saying yes.

The question I couldn't answer, when someone finally asked it in a retro, was how often the gate says yes to something it should have caught. I'd never measured it. I'd tuned it, twice, both times to make it stop paging people. That's a different activity.

An eval gate is a classifier. Its input is a diff, its output is one bit, and like every classifier it has a false negative rate and a false positive rate. Ours had both. Neither had a number attached.

Building a labeled set out of history

You can't measure recall without known positives, and the useful thing about having shipped regressions is that you have a supply of them.

I went back fourteen months and pulled every change that we knew, after the fact, had degraded model behavior in production. Three sources: incident tickets with a root cause pointing at a prompt, model, retrieval or tool change; reverts in git where the revert message named a quality problem; and bug reports that a later commit fixed by changing model-facing code. Deduped, that gave 41 merges. For each one I recorded the commit and one sentence on what got worse, because the sentence turns out to matter more than the commit.

The control set was 60 merges from the same window that touched model-facing code and had no incident, revert or quality bug within thirty days after. That thirty-day window is the weakest part of the whole exercise, and I want to be honest about which way it bends. A regression nobody noticed looks exactly like no regression. Some unknown number of my 60 controls are bad merges we never caught, so my false alarm rate is, if anything, overstated. Recall is worse: an unnoticed regression is disproportionately one the gate let through, so it never made it into the 41 at all, and the 56 percent below is an optimistic reading. Both numbers are soft in the direction of making the gate look better than it is. I decided I could live with that rather than build nothing.

Then I replayed the gate at each of those 101 commits.

# replay.py: run the eval suite as it existed at a commit, against that
# commit's model-facing code, and record the gate's verdict.
import subprocess, sys, json, pathlib
from collections import Counter

def replay_once(sha: str, suite_ref: str = "eval-suite-at-commit") -> dict:
    # suite_ref="eval-suite-at-commit" replays the suite as it was THEN.
    # suite_ref="HEAD" replays today's suite against the old code, which
    # answers a different and also useful question. Run both.
    subprocess.run(["git", "worktree", "add", "-f", "/tmp/replay", sha], check=True)
    try:
        if suite_ref != "eval-suite-at-commit":
            subprocess.run(["git", "-C", "/tmp/replay", "checkout", suite_ref, "--", "evals/"], check=True)
        out = subprocess.run(
            [sys.executable, "-m", "evals.run", "--json", "--baseline", f"{sha}~1"],
            cwd="/tmp/replay", capture_output=True, text=True,
        )
    finally:
        # without this, one exception leaks the worktree and every later
        # iteration dies on "fatal: '/tmp/replay' already exists"
        subprocess.run(["git", "worktree", "remove", "-f", "/tmp/replay"], check=False)
    if out.returncode != 0:
        # otherwise stdout is empty, json.loads raises, and the loop dies
        # 40 commits in without telling you which one broke
        raise RuntimeError(f"{sha}: {out.stderr[:400]}")
    return json.loads(out.stdout)

def replay(sha: str, runs: int = 5) -> dict:
    # judge-graded cases are not deterministic. majority verdict over 5.
    rs = [replay_once(sha) for _ in range(runs)]
    votes = Counter(r["blocked"] for r in rs)
    blocked, n = votes.most_common(1)[0]
    return {"sha": sha, "blocked": blocked, "votes": f"{n}/{runs}",
            "delta": sum(r["mean_delta"] for r in rs) / runs}

for sha in pathlib.Path("labeled/bad.txt").read_text().split():
    print(json.dumps(replay(sha)))

Replaying the suite as it existed at the commit is the honest version and it's the one that gave me the number below. Replaying today's suite against old code is the kinder version, and the gap between the two is a decent measure of how much your eval set has learned since.

Two practical notes, because both cost me a day. Pinned model versions matter: two of the 41 replays came back green in a way that had nothing to do with the gate, because the provider had since retired the model the commit ran against and our fallback silently picked a newer one. Both are in the 18 misses below, re-run against the nearest surviving pinned version, and both are misses I hold loosely. And three commits came back 3 to 2 on the majority vote rather than unanimous. A stricter rule would have demanded all five agree. I didn't use one, which again bends toward the gate.

56 percent

The gate blocked 23 of the 41 known regressions. It fired on 9 of the 60 control merges.

Recall of 56 percent was lower than anyone on the team guessed. I asked four people to write their guess down before I showed the number, which I recommend, because it converts an argument about methodology into an argument about who was closest. The guesses were 85, 90, 80 and 75.

The 15 percent false alarm rate surprised no one, because false alarms are the only part of a gate's behavior that anybody experiences. That asymmetry is the whole problem. A gate that wrongly blocks you interrupts your afternoon and you file it under "the eval thing is flaky again". A gate that wrongly passes you does nothing at all, which feels identical to working.

Where the 18 went

I read every miss. They sorted into three piles, and the sizes changed what we did next.

Eight had no case in the eval set that exercised the behavior that broke. The long-document summarizer is the clearest one: our longest eval input was about 2,800 tokens and the failure started somewhere past 6,000. The gate didn't miss this. It was never asked.

Six had a case, the case got worse, and the aggregate didn't move enough to cross the threshold. Our scores run 0 to 100 and our threshold was a two-point drop in the mean across 180 cases, so one case going from pass to fail moves it by about half a point. Six regressions were smaller than the resolution of the thing measuring them.

Four ran against a stubbed dependency, so the failure couldn't occur inside the harness at all. Our retrieval stub returned a fixed document set. Three of these four were retrieval regressions.

Those three piles want three different fixes, and only the first one is "write more evals", which is the thing everybody reaches for by default.

The threshold was set by whoever was most annoyed

Then I swept the threshold across the same labeled set, which took an afternoon because the replays were already cached.

1-point mean drop: regressions caught (of 41) 29, false alarms (of 60) 21
2-point (what we shipped): regressions caught (of 41) 23, false alarms (of 60) 9
3-point: regressions caught (of 41) 18, false alarms (of 60) 5
4-point: regressions caught (of 41) 14, false alarms (of 60) 3

I remembered both times we moved that threshold. Neither time did anyone say "we are trading recall for quiet". The first move followed a week with four false alarms. The second followed a single loud one during a launch. Both were reasonable in the moment and both were made by whoever was on call, with no visibility into the column on the left, because that column didn't exist.

This is the part I'd put in front of anyone who runs a gate on a continuous score. You have an operating point whether or not you chose it, and if you have never measured recall then every threshold change you have ever made has been a one-sided negotiation.

What we changed

We split the gate in two.

The blocking tier holds cases that are cheap, deterministic, and tied to something that actually broke once. Exact-match and structural assertions, no judge. It blocks on any case flipping from pass to fail, not on an aggregate drop, which kills the entire second pile above. It's small on purpose: 61 cases at the time of writing.

A case earns a place in it by meeting three conditions, and we wrote them down because otherwise the tier grows until it is the old suite again. It has to be traceable to a specific production failure, by ticket number, which is what keeps it from being somebody's hunch. It has to be deterministic, meaning the same input produces a byte-identical assertion result across five runs on the same model pin, which is what excludes the judge. And it has to run in under 400 milliseconds, because the whole tier runs on every pull request and a slow gate gets bypassed. Cases that fail the last two conditions aren't thrown away, they go to the reporting tier.

The reporting tier holds everything else, including all the judge-graded scoring. It posts a comment on the pull request and blocks nothing.

Moving the judge out of the blocking path is not free, and this is where I have to show the whole ledger rather than the good half. It removed five of the nine false alarms, because judge variance was most of our false alarm budget. It also cost us eight catches: eight of the original 23 were judge-graded cases that moved the aggregate past the threshold, and those now only leave a comment. Against that, per-case blocking recovers the six in pile two, all six of which were assertion cases and so qualify for the tier, unstubbing retrieval recovers two of the four in pile three, and the eight incident-derived cases we added cover pile one. Twenty-three, minus eight, plus six, plus two, plus eight, is 31.

We also stopped stubbing retrieval in the blocking tier and pinned a frozen corpus snapshot instead. Slower, and it caught two of the four stub misses on replay.

Every incident that traces back to model-facing code now owes the blocking tier a case, written from the incident's own reproduction, before the incident closes. That rule is the only reason pile one shrinks over time, and it works because writing the case is easiest on the day you understand the failure.

So: 31 of 41 caught, 4 of 60 false alarms. I don't fully believe the 31. Eight of those cases exist because these specific regressions are in my labeled set, so the back-test is partly grading a suite built from its own answer key. The honest number arrives in a year, from incidents that had no chance to influence the set. What I do believe is the 4, because nothing on the false alarm side was tuned against this data.

Is 76 percent good? I have no idea what good looks like here, and I haven't found anyone publishing theirs. What I know is that 56 wasn't the number anyone was operating on, and that the fourteen months of history I replayed are all months in which nobody had measured the thing we were trusting. Calling it a control was the mistake.

What I'd check first

Pull your last ten quality incidents and ask, for each one, whether the eval set contains a case that would have gone from pass to fail. Not "a case about that area". A case that flips.

Look at when your blocking threshold last changed, and what happened in the week before. If the answer is a false alarm, you moved your operating point along a curve you have never plotted.

Check what your eval harness stubs out. Anything you replaced with a fixture is a class of regression your gate is structurally incapable of seeing, and retrieval is usually the expensive one.

Our eval gate runs 22 minutes. The queue behind it hit three hours.

Ethan Walker — Thu, 06 Aug 2026 17:03:21 +0000

Nobody complained about the gate. They complained about Tuesday.

Our merge queue runs the full eval suite before anything lands: 1,400 cases, about 240 of them scored by an LLM judge, the rest deterministic. Wall clock, 22 minutes. One PR at a time, because the suite pins the same eval dataset and the judge budget. Every engineer on the team would tell you 22 minutes is fine. Go get coffee.

Then one Tuesday I counted nine PRs sitting in the queue at 11:04. The last one merged at 14:22. Three hours and eighteen minutes for a change that took 40 minutes to write, and the gate itself never ran slow, never flaked, never failed. Every single run took its normal 22 minutes. The queue did the rest.

The math nobody ran

A serial gate is a single-server queue. What sets the wait is the ratio between the gate's duration and how fast work arrives. That ratio has a name, utilisation, and the wait it produces is not linear in it. It has a knee.

For a fixed 22-minute service time and randomly arriving merges, the steady-state average wait in queue is:

def gate_queue_wait(s_min: float, arrivals_per_hr: float) -> float:
    """Average queue wait (minutes) before a serial s_min-minute gate.
    M/D/1: Wq = rho * s / (2 * (1 - rho)), rho = arrival rate * s."""
    rho = arrivals_per_hr * s_min / 60
    if rho >= 1:
        return float("inf")  # the queue never drains
    return rho * s_min / (2 * (1 - rho))

for rate in (1.0, 1.5, 2.0, 2.25, 2.5, 2.6):
    print(f"{rate:>4}/hr  wait {gate_queue_wait(22, rate):6.1f} min")

Run it for our 22-minute gate and the knee is right there:

1.0/hr: utilisation 37%, avg wait in queue 6.4 min, wait + gate 28.4 min
1.5/hr: utilisation 55%, avg wait in queue 13.4 min, wait + gate 35.4 min
2.0/hr: utilisation 73%, avg wait in queue 30.2 min, wait + gate 52.2 min
2.25/hr: utilisation 82%, avg wait in queue 51.9 min, wait + gate 73.9 min
2.5/hr: utilisation 92%, avg wait in queue 121.0 min, wait + gate 143.0 min
2.6/hr: utilisation 95%, avg wait in queue 224.7 min, wait + gate 246.7 min
Capacity is 2.73 merges per hour. That is the whole budget a 22-minute serial gate gives you, ever.

At one merge an hour the gate is invisible. At two an hour, the average PR waits longer in the queue than it spends being tested. At two and a half, the average experience is over two hours, and that is the average on a quiet, evenly spaced day. The formula assumes arrivals sprinkled at random. Real teams merge in bursts, after standup, before the sprint cutoff, and a burst is strictly worse than the formula. Our nine-PR Tuesday was not an anomaly. It was nine arrivals hitting a server that clears 2.7 an hour, and 9 times 22 minutes is 3 hours 18. The math was never going to do anything else.

The part that stung: we had spent a month optimising the suite from 26 minutes down to 22, and we only understood what we had bought after running these numbers. At two merges an hour, the old gate held the average wait at 85 minutes; the new one holds it at 30. And on Tuesday-heavy stretches at two and a half, the 26-minute suite was past capacity entirely (60 over 26 is 2.3 an hour): that queue was not slow, it was diverging, and we had fixed it by accident. Service time matters exactly as much as the ratio says it does, and none of that is visible from the gate's runtime alone.

What we changed

Three tiers, nothing clever.

Tier 0, per commit, 90 seconds. Every deterministic check that does not need the full dataset: schema conformance, tool-call shape, regex and exact-match cases, token-budget ceilings. About 400 of the 1,400 cases, and historically the tier that catches most honest mistakes. Runs on push, before review, outside the merge queue entirely.

Tier 1, the merge queue, batched. The full 22-minute suite still gates every merge, but it gates batches, not PRs. As soon as the runner is free it takes whatever is queued, up to four PRs, and runs once against the batch head. Green, all four land. Red, the batch splits into pairs and reruns, then the failing pair splits again: standard bisection, four extra runs worst case to isolate one offender. A batch of four cuts effective service per PR to five and a half minutes, which moves us from 92 percent utilisation back to about 23 at the same merge rate; the average wait falls off the bottom of the table, into single-digit minutes. Red batches cost us the bisection, and at our failure rate the trade wins by a large margin; if your gate is red a third of the time, batching will hurt, and your problem is the failure rate anyway.

Tier 2, nightly. The expensive sweeps that never belonged in a merge path: full judge calibration against the human-labelled set, cross-model regression, the long-tail scenarios. Nightly, with a report in the morning, and a rule that a red nightly blocks the next day's releases rather than the next engineer's merge.

Did we lose per-PR attribution inside a green batch? Yes, and we decided we do not care: a green batch means every PR in it shipped against the full suite. Attribution only matters on red, and red is exactly when bisection runs.

What it cost

One week of one engineer, most of it spent classifying the 1,400 cases into deterministic versus judge-scored, and deciding which of the heavyweight sweeps belong in the new nightly tier. The queue infrastructure was our existing merge queue with batching turned on. The 22-minute suite itself did not change at all, which is the point: we stopped paying for it per PR without deleting a single case. The nightly sweeps run on top of the merge gate, not instead of it.

Median wait in the queue, the time before your run even starts, went from 74 minutes that Tuesday-heavy week to under six the week after. Add the 22 minutes the suite still takes and a typical merge now lands in under half an hour.

What I'd check first

Your gate's service time against your busiest hour's arrival rate. Multiply them, service time in hours. Past 0.7, you have a queueing problem, not a test-speed problem, and shaving suite minutes will not save you.
What fraction of the gate is deterministic checks that could run per commit. Ours was 29 percent of cases and they catch the majority of failures; they were queueing behind an LLM judge for no reason.
Whether your queue serialises PRs that could gate as a batch. Serial-per-PR is the default in most setups. It is also the strictest possible service discipline, and the table above is the bill for it.

Half my eval regressions never touched the prompt file

Ethan Walker — Tue, 04 Aug 2026 18:22:03 +0000

I keep a tally. Over the last two quarters, a little more than half of the output regressions I traced in one RAG service came from a change that never went near the prompt directory. The eval gate was green the whole time. It was green because it never ran, or it ran against a dataset that did not exercise the thing that moved.

That is the bug I want to talk about. Not a specific incident. The shape of the coverage.

The trigger

Most eval gates are wired to fire on two things: a prompt change or a model change. Someone edits prompts/answer.md, CI runs the eval suite, the suite scores the new prompt against a dataset, and the merge is blocked if the score drops. That is a good gate. It catches the failure mode it was built for.

Look at the trigger, though. paths: ["prompts/**"]. The gate's entire theory of risk is "output changes when the prompt changes." That theory is incomplete, and the gap is not small.

The hole

Your model's output is a function of a lot of inputs. The prompt is one. Here are others, none of which touch the prompt file:

The tokenizer. A dependency bump can swap it. The same text then becomes a different number of tokens, your context budget shifts, and long inputs get truncated at a different boundary. Anthropic's own model docs note that the tokenizer introduced with a recent model generation produces "roughly 30% more tokens" for the same text than earlier models (Anthropic models overview). A 30% swing in token accounting is not a rounding error. It moves what fits in the window.
The retrieval index. Rebuild it, re-embed with a new model, or change the chunker, and the same query returns the same documents in a different order, or returns different documents. The prompt template is byte-for-byte identical. The context inside it is not.
The tool schema. Rename a parameter, change an enum, tighten a JSON schema, and the model's tool calls change. The prompt that describes the tool may not have changed at all.
A config default. Temperature, top_p, max_tokens, a response_format flag, a system-vs-developer role default in an SDK. Bump the SDK minor version and a default can flip under you. Nothing in your repo shows a diff.
The provider model behind an alias. You pinned a moving alias (a classic gpt-4o-style pointer) instead of a dated snapshot. The provider points that alias at a newer build. Your code did not change. Your outputs did.

Every one of those can regress the output. Not one of them trips a paths: ["prompts/**"] filter.

Two failure modes, not one

Separate them, because the fix differs.

Mode one: the gate does not run. The change lives in requirements.txt or services/retrieval/index.py, the path filter does not match, CI skips the eval job entirely. Silent.

Mode two: the gate runs and passes anyway. This one is sneakier. Say the tool schema changed. The eval suite runs, but the eval dataset is thirty prompt-quality cases that never invoke that tool. Full marks. Green check. The regression ships behind a passing gate, which is worse than no gate, because now you trust it.

Both modes have the same root cause. Gate coverage was defined by "what changed in the prompt directory" instead of "what can move the output."

The input list

Here is the check I actually run. It is boring and it works.

List every input that can affect the output. Prompt, model id, model version behind the alias, tokenizer, decoding params, retrieval index, embedding model, chunker, tool schemas, SDK version, and any upstream service that shapes context. For each one, answer a single question: if this changes, does an eval run get triggered, and does the eval dataset exercise it?

Two columns. Triggered yes/no. Exercised yes/no. Anything that is no in either column is a hole. You will find holes. The first time I did this for the RAG service, seven of the eleven rows were "no" on at least one column. The prompt row was the only one that was solidly "yes" on both, which is exactly why the prompt regressions were the ones we always caught and the rest were the ones that bit us.

You do not need tooling to make this list. A text file is fine. The value is in forcing yourself to name the inputs, because the ones you forget to name are the ones with no gate.

The canary

The lightweight fix is a canary eval that runs on any merge to the service, not just prompt merges. Widen the trigger from the prompt directory to the whole service path. Keep the eval small and fast so it is cheap enough to run every time.

# .github/workflows/canary-eval.yml
name: canary-eval
on:
  push:
    branches: [main]
    paths:
      - "services/answer/**"   # the whole service, not just prompts/**
jobs:
  canary:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: python -m eval.canary --n 40 --fail-under 0.9

The canary itself is deliberately small. Forty fixed cases, chosen to exercise the surfaces that the big prompt-quality suite ignores: at least a handful that force a retrieval hop, a handful that force a tool call, and a couple with long inputs that live near the context boundary. Pass rate in, threshold out.

# eval/canary.py
import sys, argparse
from statistics import mean
from myservice import answer      # the real service entrypoint
from eval.dataset import CANARY   # 40 fixed cases: retrieval + tools + long inputs

def score(case) -> float:
    out = answer(case.query)
    return float(case.check(out))  # 1.0 pass, 0.0 fail

def main() -> int:
    p = argparse.ArgumentParser()
    p.add_argument("--n", type=int, default=40)
    p.add_argument("--fail-under", type=float, default=0.9)
    args = p.parse_args()
    rate = mean(score(c) for c in CANARY[: args.n])
    print(f"canary pass rate: {rate:.2f}")
    return 0 if rate >= args.fail_under else 1

if __name__ == "__main__":
    sys.exit(main())

This is not your full eval suite. It is the smoke test that runs when someone bumps a dependency or rebuilds the index and does not think of it as a model change. Forty cases run in well under a minute for most services, which is the whole point: cheap enough that you never argue about whether to run it.

Pinning

The canary catches the changes that live in your repo. Two changes do not live in your repo, and you have to handle them separately.

First, pin the provider model to a dated snapshot rather than a moving alias, where the provider offers one. Anthropic's model docs are explicit that for models before their 4.6 generation, the alias entries are "convenience pointers that resolve to a dated model ID" (models overview), and OpenAI's classic aliases have historically moved to newer builds under the same name. A moving pointer is convenient right up until it moves during a release you did not tag as a model change. Pin the dated id in config, and treat bumping it as a change that runs the full suite.

Second, alert when the alias moves. If you must run against an alias, record the resolved model version (most providers return it in the response metadata) and diff it on a schedule. When the alias starts resolving to a new build, that is a model change that happened with zero commits on your side. You want a page, not a surprise in next week's quality numbers.

Neither of these is heavy. A pinned string in a config file and a cron job that compares a recorded version field. The cost is minutes. The thing it protects against is a class of regression you otherwise cannot see, because there is no diff to review.

What I'd check first

Read your eval workflow's trigger. If it says paths: prompts/** and nothing wider, every non-prompt input to your model is ungated. That is the hole, in one line.
Grep your service for the model string. If it is a moving alias and not a dated snapshot, you are trusting the provider not to move it between your releases. Log the resolved version so at least you would notice.
Open your eval dataset and count how many cases force a tool call and a retrieval hop. If the answer is near zero, your gate can pass while the tool and retrieval surfaces regress. Green does not mean covered.

A one-word prompt edit dropped our accuracy 9 points. Nothing caught it.

Ethan Walker — Tue, 28 Jul 2026 17:27:44 +0000

We version our code. We test our code. We gate merges on the tests. Then someone edits the system prompt in a text box, clicks save, and ships it with none of that.

The edit that got me was small. Someone changed "Answer concisely" to "Answer concisely and only from the context provided." Reasonable. It shipped on a Tuesday. By Thursday our extraction accuracy on the eval set had dropped from 0.87 to 0.78, and the only reason we noticed was a support ticket, not a gate. In most LLM apps I've seen, the prompt is the least-tested artifact even though it's the one that moves the output the most.

The gap

Here is what a normal code change goes through: PR, diff, review, CI runs the test suite, a red check blocks the merge. Here is what a prompt change goes through in most teams: someone edits a string. Sometimes it lives in a prompt-management UI, sometimes in a YAML file, sometimes hardcoded. Either way there's usually no dataset scored before and after, and no threshold that stops a bad edit.

The usual excuse is that prompts count as content rather than code. I think that gets it backwards. A prompt is the highest-leverage line in the whole system. One token changes the output distribution for every request. If you would not ship a one-line change to your ranking function without a test, you should not ship a one-line prompt edit without one either.

What a prompt regression gate actually is

Three pieces, none of them exotic:

A pinned eval set. A few dozen to a few hundred labeled examples that represent real traffic. Frozen, versioned next to the prompt.
A scorer. Whatever metric matches the task: exact-match, an F1 on extracted fields, a rubric score, an LLM-as-judge if the task needs it. It has to be the same scorer run before and after.
A delta threshold in CI. The gate compares against the committed baseline and fires on the drop itself: "this prompt change may not drop score more than X against the pinned baseline." That's the check that would have caught my Tuesday edit.

The absolute-versus-delta distinction is the part people get wrong. An absolute gate ("score must be above 0.85") drifts: it goes green on a change that quietly dropped you from 0.91 to 0.86. A delta gate compares against the committed baseline and fires on the drop itself.

Pseudocode, because it's easier to argue with than prose:

baseline = load_scores("baseline.json")      # {example_id: score}, committed with the prompt
current  = run_eval(prompt=new_prompt, dataset="pinned_v3")

delta = mean(e.score for e in current) - mean(baseline.values())
if delta < -0.02:                            # tune per task; ours is 2 points
    fail(f"prompt regressed {delta:+.3f} vs baseline on pinned_v3")

# also check the tail, not just the mean
regressed = [e for e in current if e.score < baseline[e.id] - 0.10]
if len(regressed) > 0.05 * len(current):
    fail(f"{len(regressed)} examples dropped >10 pts; mean hid it")

That second check matters more than the first. A mean can stay flat while a specific slice falls off a cliff. My 0.87-to-0.78 drop was concentrated in one document type; the aggregate looked survivable until you split it.

Where the tools sit (and where they don't help)

The landscape splits into three jobs, and no single tool is the whole answer as of July 2026:

Prompt versioning and management. Langfuse and LangSmith both do this well: a prompt registry, versions, the ability to roll back. This is the "diff and review" half. It is real and useful. Version history shows you what changed. It does not score whether the change was good.
Eval in CI. Promptfoo and DeepEval both let you define assertions or metrics and run them in a pipeline, which is the "red check blocks the merge" half. This is where the delta gate above lives. Both are open source.
Prompt optimization. This is the part most teams skip. Instead of hand-editing the prompt and hoping, you treat the prompt as a search problem: define the metric, let a search procedure propose and score variants. DSPy is the best-known open-source take on this. Future AGI's open-source platform bundles several such search procedures behind a single optimizer call (its agent-opt library). The reason I mention both in the same breath is that they are the two I have actually seen treat prompting as measurable search rather than a text box. If you know others, I want the list.

The honest read: Langfuse still owns pure versioning ergonomics, DSPy has the deepest optimization research behind it, and the platforms that bundle eval plus optimization (Future AGI is one) buy you the CI wiring in one place at the cost of adopting more of one stack. Pick by which half you're missing. If you have zero gate today, the eval-in-CI half is the one that stops the Tuesday edit.

The part that surprised me

I assumed the optimizer angle was overkill for us. It wasn't the optimization that paid off first, it was the discipline it forced: to run an optimizer at all you need a pinned dataset and a scorer, which are exactly the two things a regression gate needs. Building the gate and building toward optimization are the same first two steps. We got the gate as a byproduct of trying to do the fancier thing.

What I'd check first

Does a prompt edit in your app touch a pinned eval set before it ships, or does it go straight to prod from a text box?
Is your gate an absolute threshold (drifts, hides slow drops) or a delta against a committed baseline (catches the drop)?
Do you check per-example regressions, or only the mean, which is where a one-slice cliff hides?

Retry-until-green turns a 70 percent eval gate into a 34 percent one

Ethan Walker — Fri, 24 Jul 2026 06:19:13 +0000

TL;DR. We let developers re-run a failed eval gate, up to three total runs, and merge on any green. Felt harmless. It is not. If a judge-scored gate catches a real regression with probability 0.7 per run, merge-on-any-green across three runs drops the catch rate to 0.7^3 = 0.343, because the regression ships the moment any single run passes. The same policy pushed our false-block rate near zero, which is why everyone loved it and nobody measured the other side. A retry is a second sample from the same noisy scorer, so aggregate the samples instead of letting the luckiest one decide.

The button

Our eval gate runs a judge-scored suite on every PR that touches a prompt or a retrieval config. I have written before about judge scores drifting between runs on unchanged output; the short version is same diff, 0.83 Friday, 0.78 Monday.

So we added the button. "Re-run eval gate." Up to three runs total, merge on any green. The team's false-block complaints stopped within a week. I counted that as a win and moved on.

The part I did not count: what the button does to a real regression.

The arithmetic

Two numbers describe any noisy gate. The chance it fails a clean PR (false block, call it f). The chance it fails a PR that genuinely regressed (catch rate, call it d).

Both react to retries the same way, and that is the problem. Under merge-on-any-green with k allowed runs:

P(clean PR gets blocked) = f^k
P(regressed PR gets blocked) = d^k

With f = 0.10 and d = 0.70 and k = 3, the false-block rate falls from 10 percent to 0.1 percent. Great. The catch rate falls from 70 percent to 34 percent. It used to catch seven regressions in ten. Now it catches about three, and the dashboard shows nothing, because a merged PR looks identical whether it merged on run one or run three.

Run it yourself:

def blocked(fail_prob: float, k: int) -> float:
    """P(all k runs fail) under merge-on-any-green."""
    return fail_prob ** k

for k in (1, 2, 3, 5):
    print(k, round(blocked(0.70, k), 3), round(blocked(0.10, k), 6))

Output for k = 1, 2, 3, 5: catch 0.7, 0.49, 0.343, 0.168. False blocks 0.1, 0.01, 0.001, 1e-05. Every extra allowed run trades a chunk of your remaining detection power for false-block reduction you mostly already had.

Why the policy works for tests and fails for judge scores

Google's testing blog documented the strongest version of retry-as-policy in their 2016 flaky-test writeup (testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html): a test can be marked flaky so that it reports "a failure only if it fails 3 times in a row." The same post is honest about the cost, noting that the mechanism trains developers to ignore flakiness in their own tests until the triple-fail threshold trips. They also report that roughly 1.5 percent of all test runs across their corpus come back flaky. A decade old, still the clearest thing written on the subject.

Here is why the policy is defensible for tests and not for judge scores. A conventional test is deterministic in intent; flakiness comes from the environment around it. Ports, clocks, race conditions. When a flaky test passes, the pass carries real information, because the assertion itself is exact and the code path demonstrably works when the environment cooperates.

A judge-scored eval inverts this. The scorer is the random variable, so a passing run is just another draw from the distribution that produced the failing run, and it deserves exactly the same trust. Taking the best of three draws and calling it the verdict is not noise handling. It is selecting the sample you liked.

One question worth asking your own CI today: how many of your merged PRs had at least one red eval run before the green one? If your system cannot answer that, you have no idea what your effective catch rate is.

What we run now

Three changes, in the order we made them.

Deterministic checks stay blocking and get no retry button. Schema validation, regex assertions, golden-string diffs, exit codes. A fail is a fail. This is most of the gate and it is the part that was never flaky to begin with.
The judge suite runs a fixed panel of n = 5 samples and gates on the aggregate, computed once. I argued nine days ago that a single judge score cannot hold a merge gate, and that has not changed. An aggregate over a fixed panel is a different object: the run-to-run variance that makes one score untrustworthy is exactly what the averaging shrinks. No re-run path exists for it. If someone wants another run, the new samples join the panel and the aggregate recomputes over all of them. Averaging shrinks the noise by a factor of sqrt(n). Best-of-n converts the noise into bias in the direction you wanted.
Retries exist only for infrastructure failures, and the runner distinguishes them by exit code. A timeout or a 429 re-runs automatically and silently. A low score never does.

The retry-rate metric came last and taught us the most. Retries per PR per week is now on the same dashboard as the pass rate. It is our flakiness number. The week it spikes, something in the judge path drifted, and we know before anyone starts arguing with the gate.

What I'd check first

Count merged PRs in the last 30 days with a red-then-green eval history. That number times your per-run catch rate is roughly what you are leaking.
Find every place a human can re-trigger a scored check. Each one lets the luckiest sample overrule the rest. Replace it with an aggregate or delete it.
Split your runner's exit codes: infra failure, score failure, harness bug. Only the first class earns an automatic retry.

Our CI eval gate sent us a token bill. The deterministic one sent nothing.

Ethan Walker — Tue, 21 Jul 2026 15:02:56 +0000

TL;DR. A quality gate that grades every pull request with an LLM-as-judge metric is buying a judge call per PR, per metric, forever. At forty PRs a day and a dozen judge-graded metrics, that is real money and, worse, a merge queue now coupled to a paid API's price, latency, and rate limits. I wired six open-source eval frameworks into the same GitHub Actions gate and watched the invoice, not the feature list. The split that matters for cost is simple: does the check run in-process and return an exit code, or does it call a model. Promptfoo assertions, MLflow's heuristic metrics, and Future AGI's deterministic metrics can gate without a single judge call. RAGAS ships deterministic metrics too (BLEU, ROUGE, string checks), but the RAG metrics you actually adopt it for are judge calls. DeepEval and Phoenix sit in the middle, judge-first by default but drivable in a cheaper mode. Rankings and the arithmetic below.

The invoice

I was not auditing cost. Finance was. Someone forwarded me a line item, a few hundred dollars against an OpenAI project key I did not recognize, tagged ci. It was the eval gate. We had added an LLM-as-judge relevance metric to the merge queue four months earlier, felt good about the coverage, and moved on. Nobody connected "we grade every PR with a model" to "we pay for every PR we grade."

A few hundred a month is noise against an engineering payroll. What bothered me was the shape of it: the bill grew with our merge volume, which means the better the quarter, the more the gate cost, and the gate itself did nothing you could not have paid for once. And the second I looked, I realized the merge queue now had a dependency nobody had reviewed. If that API rate-limited us at 9am on a busy day, PRs would stall on a gate that had nothing to do with the code in them.

So I did the boring thing and measured it.

The cost formula

The number that drives cost is not metric quality or accuracy. It is how many model calls one gate run makes.

A deterministic check makes zero. contains, equals, regex, is-json, a schema validator, a required-field assert, a golden-file diff. These run in-process, return in milliseconds, and cost nothing per PR. An LLM-as-judge metric makes one model call each, every run, and bills for the tokens.

So the monthly cost of a gate is close to:

PRs/day  x  judge calls per run  x  price per judge call  x  workdays

Everything else is a rounding error. The chart is that formula, nothing more.

[DIAGRAM: https://lh3.googleusercontent.com/d/1bVKW6DCpbvxco0nlIh6O06aJZLe_ttMh]

At forty PRs a day, a gate running a dozen judge-graded metrics adds roughly twenty dollars a month by this arithmetic (a stated ~$0.002 per call; check current token prices before you quote it). A single-judge gate is a few dollars. A deterministic gate is flat at zero, no matter how many PRs you merge. The absolute numbers are small. The point is the slope, and the fact that only one of these lines is flat.

The ranking

All licenses, metric counts, and defaults are as of mid-2026. Check each repo before you rely on any of this, because all six move.

Promptfoo (MIT). The cheapest to gate on, because its assertion library is deterministic by design: contains, equals, regex, is-json, starts-with, plus cost and latency asserts. An LLM-judge assertion is available but opt-in. You can build a real blocking gate that never makes a model call. CLI exit code, JSON output, maintained Action.
MLflow evaluate (Apache-2.0). mlflow.evaluate() ships heuristic metrics (exact match, token overlap, and similar) alongside optional genai judge metrics. Gate on the heuristic set and you pay nothing per run. The eval is logged next to the experiment, which is the reason to reach for it.
Future AGI (Apache-2.0). Its ai-evaluation SDK runs through one evaluate() call from Python or TypeScript and mixes deterministic checks with judge-based metrics. Gate on the deterministic ones and, like Promptfoo and MLflow, it makes no model call, so it lands on the cheap end. Gate on the judge metrics and it bills like the rest.
DeepEval (Apache-2.0). Its determinism comes from the runner. You inherit pytest exit codes and JUnit XML for free, which is why it is pleasant in CI, but most of its metric catalog is judge calls. Run it on plain pytest asserts and it is cheap. Run its default metrics on every PR and you are paying per metric, per PR.
Arize Phoenix (Elastic License 2.0). run_evals() in a script, a handful of evaluators, judge-based. The draw is tracing plus eval in one tool, not a deterministic gate. As a pure cost line it bills like any judge-first tool.
RAGAS (Apache-2.0). It does ship deterministic metrics (BLEU, ROUGE, chrF, string and tool-call checks), so a gate built on those makes no model call. But nobody adds RAGAS for BLEU. The reason it is on your stack is its RAG metrics (faithfulness, answer relevancy, context precision), and those are judge calls almost by definition. Gate on what you came to RAGAS for, and every check is a model call.

Beyond the dollars

The money was the small problem. Two bigger ones showed up once the gate depended on an API.

Rate limits became merge-queue outages. A judge-graded gate that 429s does not fail gracefully, it fails your PR, and the fix is retry-with-backoff logic you now own inside CI. A deterministic assert never 429s.

Price is not yours to control. We had modeled the gate at one token price. Model prices move, sometimes down, sometimes up, and a gate you pay per-run for is a line item that reprices without asking you. A gate that runs in-process is priced once, in compute you already have.

None of this means judge metrics are useless. I still run them. I just stopped putting them in the blocking path. Everything deterministic gates the merge. Everything judge-graded runs on the same PR, posts as a non-blocking comment, and a human reads it. The gate got cheaper and the signal did not get worse, because the judge was never a good binary gate anyway.

What I'd check first

If your CI bill has a mystery line item, or you are about to add an eval gate:

Count the model calls in one gate run (metrics x PRs/day). If that number is not zero and you expected a fixed cost, that is your line item.
Move every deterministic check you have (schema, required fields, forbidden strings, one golden diff) into the blocking path, and demote every judge metric to a non-blocking PR comment.
Before you gate on any judge metric, ask whether it survives a 429 at 9am. If a rate limit would block a clean PR, it is a coupling you did not mean to buy.

Our few-shot examples came from the eval set. The 0.94 was fiction.

Ethan Walker — Mon, 20 Jul 2026 07:29:57 +0000

TL;DR. Our ticket-routing eval scored 0.94 for five weeks. The number was manufactured. We had built a dynamic few-shot selector that retrieved the eight nearest labeled examples for each input, and we built its index out of the same labeled_tickets.jsonl the eval set was sampled from. So for every eval case, the nearest neighbor in that index was the eval case itself, gold label attached, pasted into the prompt directly above the question we were about to ask. The model was not answering. It was copying. Measured against tickets the index had never seen, real accuracy was about 0.79. The reframing that stuck for me: contamination is not just a training-time problem you inherit from a model vendor. Any pool you draw prompt content from is part of your eval's input. If your eval set and your few-shot pool share a parent file, the leak is yours, in your repo, shipped by someone on your team last Tuesday. Split the pools by content hash at the source, then check the prompt and not only the training data.

The trace that ruined a good number

I was not looking for this. I was chasing p99 latency on the routing endpoint, which had crept past two seconds and was making the queue back up. So I pulled a slow trace and started reading the prompt we actually send, top to bottom, the way you do when you suspect somebody stuffed too much into the context window.

The prompt had a system block, then eight few-shot examples, then the ticket to classify. We rendered them nearest-last, so the closest match sat directly above the question.

Example eight was the ticket to classify. Same text. Same customer. And sitting under it, formatted as the demonstration answer, was the label we were about to grade the model on.

I read it three times. Then I checked four more traces. Same shape every time: the last example before the question was the question.

Our eval had been reporting 0.94 since the selector shipped. Nobody questioned it because 0.94 is a believable number. Five weeks. Nobody asked once. If the suite had printed 1.00, someone would have opened it within the hour, because a perfect score reads as a bug. A 0.94 reads as a good quarter. A total leak still did not mean a perfect score: the other seven demonstrations pulled against the exact match often enough to cost a few points, which is precisely what made 0.94 look earned. It went in a deck. Somebody put it on a slide with an arrow pointing up. The score was high enough to celebrate and low enough to trust, which is the worst place a wrong number can sit.

select_examples()

The mechanism is boring, which is why it survived review.

We started with static few-shot: eight hand-picked examples, hardcoded, same eight for every request. It worked fine and it was obviously fine, because you could read the eight in the diff.

Then someone (me, partly, in a design review I do not get to distance myself from) pointed out that a fixed eight cannot cover billing and abuse reports and integration bugs at once. Retrieve the examples instead. For each incoming ticket, embed it, pull the eight nearest labeled tickets out of the index, put those in the prompt. Better coverage per token. This is standard practice and I still think it is right.

The index got built from labeled_tickets.jsonl, which was where every labeled ticket lived. Roughly 1,900 rows at the time.

The eval set was 600 rows sampled from labeled_tickets.jsonl.

Those two sentences are the whole bug. Both artifacts were correct in isolation. Both were reviewed. Nobody put them side by side, because they lived in different files, owned by different people, merged six weeks apart.

That gap is the part worth generalizing. Neither diff was wrong. A reviewer on the selector PR sees a sensible retrieval change and asks about latency and index freshness. A reviewer on the eval PR sees a defensible sample size and asks whether 600 cases is enough. Both are good questions. Neither reviewer gets asked the only one that mattered, which is whether these two things read the same rows, because that question is not visible in either diff. It lives in the space between them. We had no review process for the space between two files, and I am not sure most teams do.

Here is what the selector did, reduced to the part that matters:

def select_examples(query, index, k=8):
    # index was built from every row in labeled_tickets.jsonl
    return index.search(embed(query), k=k)

Then the eval harness:

cases = random.sample(load("labeled_tickets.jsonl"), 600)

Read them together and the failure is arithmetic, not machine learning. The eval case is in the index. The eval case's own embedding is its nearest neighbor, at distance zero. Every single one of the 600 eval cases retrieved itself as its own nearest neighbor and carried its gold label into the prompt. Not some of them. All of them. A retriever asked to find the most similar labeled example to a text that is sitting in its own index will return that text, because nothing is more similar to a string than the string.

We also had about 40 exact duplicate tickets in the pool, because customers paste the same complaint twice and support macros generate identical bodies. Those would have leaked across a naive random split even without a retriever. The retriever just made the leak total instead of partial.

Why the retriever hands over the answer

Worth being precise about what broke, because "contamination" gets used loosely enough to stop meaning anything.

The model did not memorize our tickets during pre-training. The vendor did not train on our data. Our fine-tune was clean. Every version of contamination I had read about was about the training corpus, and every one of those was genuinely not our problem.

The leak happened at prompt construction time, in our code, at inference, on every request. The model saw the answer in the line directly above the question, in a block explicitly labeled as examples of correct behavior. It did what any competent few-shot learner does with a demonstration that exactly matches the query. It copied the label.

So the eval was measuring copy fidelity. That is a real capability. It is not the capability we were shipping, and it is not the one the number claimed.

The published work here is almost entirely about training data, and it is worth knowing even though it does not describe this exact bug. Zhou et al., in Don't Make Your LLM an Evaluation Benchmark Cheater, work through how benchmark leakage into training "can dramatically boost the evaluation results," producing an unreliable read on what a model can do. Their setting is pre-training and fine-tuning corpora. Mine was a JSONL file and a vector index.

The mechanism ports anyway. The model cannot tell where in its input a leaked answer came from, and neither can your score. Weights or context window, the arithmetic is the same: if the answer reached the model before the question, the number measures retrieval, not reasoning. The difference is that training contamination is mostly somebody else's to fix, and prompt contamination is entirely yours. I find that clarifying rather than comforting.

contamination_check.py

The check we should have had. Standard library only, so it runs anywhere Python does and there is nothing to install in CI.

Two passes. A normalized hash catches verbatim and cosmetically edited duplicates. An n-gram containment score catches the case where an eval item sits inside a longer example. Containment rather than Jaccard on purpose: a short eval case buried in a long few-shot example still scores 1.0, and that is exactly the leak worth failing on.

"""contamination_check.py: does the few-shot pool already contain the eval answer?"""
import hashlib
import re
import unicodedata

_PUNCT = re.compile(r"[^\w\s]", flags=re.UNICODE)
_WS = re.compile(r"\s+")


def normalize(text):
    """Casefold, strip punctuation, collapse whitespace. Defeats cosmetic edits."""
    text = unicodedata.normalize("NFKC", text).casefold()
    return _WS.sub(" ", _PUNCT.sub(" ", text)).strip()


def fingerprint(text):
    return hashlib.sha256(normalize(text).encode("utf-8")).hexdigest()


def ngrams(text, n):
    toks = normalize(text).split()
    if len(toks) < n:
        return {" ".join(toks)} if toks else set()
    return {" ".join(toks[i:i + n]) for i in range(len(toks) - n + 1)}


def containment(case, example, n):
    """Fraction of the eval case's n-grams that also appear in the example."""
    gc = ngrams(case, n)
    if not gc:
        return 0.0
    return len(gc & ngrams(example, n)) / len(gc)


def audit(eval_set, fewshot_pool, n=5, threshold=0.6):
    """Return (eval_idx, pool_idx, score, kind) for every eval case the pool leaks."""
    by_hash = {}
    for i, ex in enumerate(fewshot_pool):
        by_hash.setdefault(fingerprint(ex), i)

    hits = []
    for j, case in enumerate(eval_set):
        i = by_hash.get(fingerprint(case))
        if i is not None:
            hits.append((j, i, 1.0, "exact"))
            continue
        scored = ((containment(case, ex, n), i) for i, ex in enumerate(fewshot_pool))
        score, i = max(scored, default=(0.0, -1))
        if score >= threshold:
            hits.append((j, i, round(score, 3), f"{n}-gram"))
    return hits


if __name__ == "__main__":
    fewshot_pool = [
        "Escalate to a human when the customer mentions legal action.",
        "Refund the order automatically if it shipped more than 30 days ago.",
        "When a customer asks to escalate to a human because they mention legal "
        "action or a lawsuit, hand the ticket to the on-call agent immediately.",
    ]

    eval_set = [
        "Escalate to a human when the customer mentions legal action.",         # verbatim
        "escalate to a human when the customer mentions LEGAL ACTION!!",        # cosmetic edit
        "Escalate to a human because they mention legal action or a lawsuit.",  # fragment
        "Hand off to a person if the buyer threatens to sue.",                  # paraphrase
        "Ask for the order number before checking shipment status.",            # clean
    ]

    hits = audit(eval_set, fewshot_pool)
    leaked = {j for j, _, _, _ in hits}
    for j, i, score, kind in hits:
        print(f"LEAK eval[{j}] <- pool[{i}]  score={score:<5} via {kind}")
    print(f"\n{len(leaked)}/{len(eval_set)} eval cases leaked by the few-shot pool")
    for j, case in enumerate(eval_set):
        if j not in leaked:
            print(f"  clean: eval[{j}] {case!r}")

Output:

LEAK eval[0] <- pool[0]  score=1.0   via exact
LEAK eval[1] <- pool[0]  score=1.0   via exact
LEAK eval[2] <- pool[2]  score=1.0   via 5-gram

3/5 eval cases leaked by the few-shot pool
  clean: eval[3] 'Hand off to a person if the buyer threatens to sue.'
  clean: eval[4] 'Ask for the order number before checking shipment status.'

Three of five caught. The cosmetic edit gets normalized into an exact hit, and the fragment gets caught by containment against the longer pool entry. Both of those would have slipped past a naive set(eval) & set(pool).

Now look at what the script calls clean.

The paraphrase the check calls clean

eval[3] is "Hand off to a person if the buyer threatens to sue." The pool contains "Escalate to a human when the customer mentions legal action." Same rule. Same decision. Zero shared 5-grams, so my check reports it as clean and moves on.

It is not clean. It is the same eval case wearing a different coat, and a model given the pool entry will get eval[3] right for reasons that have nothing to do with understanding your routing policy.

This is a known and documented hole, not something I discovered. Yang, Chiang, Zheng, Gonzalez and Stoica went at it directly in Rethinking Benchmark and Contamination for Language Models with Rephrased Samples. Their finding is that "simple variations of test data (e.g., paraphrasing, translation) can easily bypass these decontamination measures," where the measures in question are exactly the string and n-gram matching my script does. They report that a 13B model can overfit a benchmark and reach performance on par with GPT-4 when rephrased test data is left in, and they found 8% to 18% overlap with HumanEval sitting in pre-training sets like RedPajama-Data-1T and StarCoder-Data. Their decontamination tool is public at lm-sys/llm-decontaminator, Apache-2.0, and it uses an LLM to catch what string matching cannot.

Again their setting is training corpora and mine is a prompt. The blind spot transfers cleanly, because n-grams do not know what a sentence means in either setting.

I have not run their tool against our pool yet, so I am not going to tell you what it would find. What I will say is that the cheap check is worth shipping anyway. It caught three of five in a toy example and it caught our real leak on the first run, because our leak was verbatim. Verbatim is the common case. Paraphrase contamination is real, and my honest position is that I do not currently know how much of it we have, which is a different sentence from "we don't have any."

Where else the same pool leaks

Once I had the shape of it I went looking elsewhere and found three more instances in our own repo within a day. Listing them because the retriever version is the flashiest and probably the least common.

Static few-shot, curated out of the labeled pool. Before the retriever we had eight hardcoded examples, and I had filed them as safe precisely because they were hardcoded. They came from the same file. Two of the eight were in the eval set. A fixed leak is smaller than a total leak. It is still a leak, and it sat in that prompt for months while I told myself static few-shot was the conservative option.

RAG evals where the corpus contains the eval documents. Same arithmetic, different index. If your eval questions were written from documents that live in the retrieval corpus, the retriever will hand the model the exact paragraph each question was written from. This one is arguably fine, because production does the same thing. It stops being fine the second you report the score as evidence about the model rather than about your retriever.

Synthetic eval cases generated from the seed examples. The easiest to walk into and the one I would bet is most widespread right now. You ask a model for 500 eval cases. You seed it with your best labeled examples so the output looks like your domain. Those same examples are in your few-shot block. Your eval set is now a paraphrase of your prompt. No file is shared, no hash collides, and contamination_check.py finds nothing, because there is nothing verbatim left to find. It is the paraphrase case from the Yang et al. paper, arriving through a door we built ourselves and held open.

The common factor is not retrieval. It is one pool of text doing two jobs: teaching the model and grading it. Whenever those roles drink from the same well, the score is compromised, and how badly is not knowable from the score.

Disjoint by construction, not by discipline

Detection is a smoke alarm. It tells you the kitchen is already on fire. The actual fix is to make the overlap impossible to express.

We stopped sampling the eval set and the few-shot index separately from one parent file. Instead, every record gets assigned to exactly one side by hashing its normalized content:

def assign(text, eval_frac=0.35, salt="fewshot-v3"):
    h = hashlib.sha256((salt + normalize(text)).encode("utf-8")).hexdigest()
    return "eval" if int(h[:8], 16) / 0xFFFFFFFF < eval_frac else "fewshot"


def split_pool(pool, eval_frac=0.35, salt="fewshot-v3"):
    buckets = {"eval": [], "fewshot": []}
    for rec in pool:
        buckets[assign(rec, eval_frac, salt)].append(rec)
    return buckets["eval"], buckets["fewshot"]

Three properties earn this over random.sample, and the third is the one that actually mattered to us.

It is deterministic. No seed to forget, no ordering dependency. The same record lands in the same bucket on my laptop, in CI, and in the batch job that rebuilds the index at 4am.

It is stable as the pool grows. Adding 500 new tickets does not reshuffle the existing split, so last month's eval numbers stay comparable to this month's. A reshuffling split quietly moves your baseline and you get to spend a day proving the model did not regress.

It puts duplicates on the same side. This is the part random.sample cannot do. Because the hash runs over the normalized text, our 40-odd exact duplicates and their cosmetic variants all resolve to one bucket. Under random sampling, a duplicated ticket had a real chance of landing one copy in eval and one in few-shot, which is the leak reappearing through the back door after you thought you had closed it.

Run it over 2,000 synthetic records and the realized eval fraction lands within about a point of the 0.35 target, which is the usual hash-bucket wobble and does not matter at this size. The exact counts depend on your strings, so do not pattern-match mine. The part that does not wobble: feed it the escalation rule plus its shouty and double-spaced variants and all three land in the same bucket, every run, on every machine.

The salt is there so you can rotate the split deliberately. Bump fewshot-v3 to fewshot-v4 and you get a fresh partition, on purpose, in a diff, with a name someone has to review.

Then it goes in CI as a gate, not a dashboard. The audit runs on every PR that touches either artifact, and a nonzero hit count fails the build with the offending indices printed. It takes about a second on our pool. I have opinions about slow eval gates, but a hash join over a few thousand strings is not where your merge queue goes to die.

The number after the fix

Rebuilt the index from the few-shot side only. Re-ran the 600-case eval.

0.79.

That is roughly a 15 point drop, and it is the first number that suite ever produced that meant anything. Nobody enjoyed the meeting. The version I would defend now is that we did not lose 15 points, we found out we never had them, and we found out from a trace instead of from a customer.

The follow-on was more interesting than the drop. With a real number, the error slices were legible for the first time. Abuse reports were dragging well below the mean while billing sat above it, and that gap had been invisible at 0.94 because copying the label works equally well for every category. Contamination does not just inflate your score. It flattens it, and a flat score hides the shape of the problem you were trying to see. We spent the next sprint on abuse-report examples and got some of the 15 points back honestly, which took actual work.

What I'd check first

Print one real prompt from a failing eval run and read it end to end. Not the template. The rendered string, with the retrieved examples in it. If the eval case appears in its own prompt, stop and go fix that before you interpret another score.
Diff the provenance of the eval set against the few-shot pool. Not the contents, the source. If both trace back to the same file, table, or index, assume overlap until a hash join says otherwise, and treat any shared parent as a leak that has not been found yet.
Suspect the believable number, not just the perfect one. 1.00 gets investigated. 0.94 gets a slide. Ask what score you would have accepted without checking, and go check that one first.

Our few-shot examples came from the eval set. The 0.94 was fiction.

Ethan Walker — Thu, 16 Jul 2026 17:42:16 +0000

The trace that ruined a good number

The prompt had a system block, then eight few-shot examples, then the ticket to classify. We rendered them nearest-last, so the closest match sat directly above the question.

Example eight was the ticket to classify. Same text. Same customer. And sitting under it, formatted as the demonstration answer, was the label we were about to grade the model on.

I read it three times. Then I checked four more traces. Same shape every time: the last example before the question was the question.

select_examples()

The mechanism is boring, which is why it survived review.

We started with static few-shot: eight hand-picked examples, hardcoded, same eight for every request. It worked fine and it was obviously fine, because you could read the eight in the diff.

The index got built from labeled_tickets.jsonl, which was where every labeled ticket lived. Roughly 1,900 rows at the time.

The eval set was 600 rows sampled from labeled_tickets.jsonl.

Here is what the selector did, reduced to the part that matters:

def select_examples(query, index, k=8):
    # index was built from every row in labeled_tickets.jsonl
    return index.search(embed(query), k=k)

Then the eval harness:

cases = random.sample(load("labeled_tickets.jsonl"), 600)

Why the retriever hands over the answer

Worth being precise about what broke, because "contamination" gets used loosely enough to stop meaning anything.

So the eval was measuring copy fidelity. That is a real capability. It is not the capability we were shipping, and it is not the one the number claimed.

contamination_check.py

The check we should have had. Standard library only, so it runs anywhere Python does and there is nothing to install in CI.

"""contamination_check.py: does the few-shot pool already contain the eval answer?"""
import hashlib
import re
import unicodedata

_PUNCT = re.compile(r"[^\w\s]", flags=re.UNICODE)
_WS = re.compile(r"\s+")


def normalize(text):
    """Casefold, strip punctuation, collapse whitespace. Defeats cosmetic edits."""
    text = unicodedata.normalize("NFKC", text).casefold()
    return _WS.sub(" ", _PUNCT.sub(" ", text)).strip()


def fingerprint(text):
    return hashlib.sha256(normalize(text).encode("utf-8")).hexdigest()


def ngrams(text, n):
    toks = normalize(text).split()
    if len(toks) < n:
        return {" ".join(toks)} if toks else set()
    return {" ".join(toks[i:i + n]) for i in range(len(toks) - n + 1)}


def containment(case, example, n):
    """Fraction of the eval case's n-grams that also appear in the example."""
    gc = ngrams(case, n)
    if not gc:
        return 0.0
    return len(gc & ngrams(example, n)) / len(gc)


def audit(eval_set, fewshot_pool, n=5, threshold=0.6):
    """Return (eval_idx, pool_idx, score, kind) for every eval case the pool leaks."""
    by_hash = {}
    for i, ex in enumerate(fewshot_pool):
        by_hash.setdefault(fingerprint(ex), i)

    hits = []
    for j, case in enumerate(eval_set):
        i = by_hash.get(fingerprint(case))
        if i is not None:
            hits.append((j, i, 1.0, "exact"))
            continue
        scored = ((containment(case, ex, n), i) for i, ex in enumerate(fewshot_pool))
        score, i = max(scored, default=(0.0, -1))
        if score >= threshold:
            hits.append((j, i, round(score, 3), f"{n}-gram"))
    return hits


if __name__ == "__main__":
    fewshot_pool = [
        "Escalate to a human when the customer mentions legal action.",
        "Refund the order automatically if it shipped more than 30 days ago.",
        "When a customer asks to escalate to a human because they mention legal "
        "action or a lawsuit, hand the ticket to the on-call agent immediately.",
    ]

    eval_set = [
        "Escalate to a human when the customer mentions legal action.",         # verbatim
        "escalate to a human when the customer mentions LEGAL ACTION!!",        # cosmetic edit
        "Escalate to a human because they mention legal action or a lawsuit.",  # fragment
        "Hand off to a person if the buyer threatens to sue.",                  # paraphrase
        "Ask for the order number before checking shipment status.",            # clean
    ]

    hits = audit(eval_set, fewshot_pool)
    leaked = {j for j, _, _, _ in hits}
    for j, i, score, kind in hits:
        print(f"LEAK eval[{j}] <- pool[{i}]  score={score:<5} via {kind}")
    print(f"\n{len(leaked)}/{len(eval_set)} eval cases leaked by the few-shot pool")
    for j, case in enumerate(eval_set):
        if j not in leaked:
            print(f"  clean: eval[{j}] {case!r}")

Output:

LEAK eval[0] <- pool[0]  score=1.0   via exact
LEAK eval[1] <- pool[0]  score=1.0   via exact
LEAK eval[2] <- pool[2]  score=1.0   via 5-gram

3/5 eval cases leaked by the few-shot pool
  clean: eval[3] 'Hand off to a person if the buyer threatens to sue.'
  clean: eval[4] 'Ask for the order number before checking shipment status.'

Now look at what the script calls clean.

The paraphrase the check calls clean

It is not clean. It is the same eval case wearing a different coat, and a model given the pool entry will get eval[3] right for reasons that have nothing to do with understanding your routing policy.

Again their setting is training corpora and mine is a prompt. The blind spot transfers cleanly, because n-grams do not know what a sentence means in either setting.

Where else the same pool leaks

Disjoint by construction, not by discipline

Detection is a smoke alarm. It tells you the kitchen is already on fire. The actual fix is to make the overlap impossible to express.

We stopped sampling the eval set and the few-shot index separately from one parent file. Instead, every record gets assigned to exactly one side by hashing its normalized content:

def assign(text, eval_frac=0.35, salt="fewshot-v3"):
    h = hashlib.sha256((salt + normalize(text)).encode("utf-8")).hexdigest()
    return "eval" if int(h[:8], 16) / 0xFFFFFFFF < eval_frac else "fewshot"


def split_pool(pool, eval_frac=0.35, salt="fewshot-v3"):
    buckets = {"eval": [], "fewshot": []}
    for rec in pool:
        buckets[assign(rec, eval_frac, salt)].append(rec)
    return buckets["eval"], buckets["fewshot"]

Three properties earn this over random.sample, and the third is the one that actually mattered to us.

It is deterministic. No seed to forget, no ordering dependency. The same record lands in the same bucket on my laptop, in CI, and in the batch job that rebuilds the index at 4am.

The salt is there so you can rotate the split deliberately. Bump fewshot-v3 to fewshot-v4 and you get a fresh partition, on purpose, in a diff, with a name someone has to review.

The number after the fix

Rebuilt the index from the few-shot side only. Re-ran the 600-case eval.

0.79.

What I'd check first

Print one real prompt from a failing eval run and read it end to end. Not the template. The rendered string, with the retrieved examples in it. If the eval case appears in its own prompt, stop and go fix that before you interpret another score.
Diff the provenance of the eval set against the few-shot pool. Not the contents, the source. If both trace back to the same file, table, or index, assume overlap until a hash join says otherwise, and treat any shared parent as a leak that has not been found yet.
Suspect the believable number, not just the perfect one. 1.00 gets investigated. 0.94 gets a slide. Ask what score you would have accepted without checking, and go check that one first.

We gated CI on six open-source LLM eval frameworks. Only two survived the merge queue.

Ethan Walker — Tue, 14 Jul 2026 16:20:26 +0000

TL;DR. Most "top open-source LLM eval framework" roundups rank features. None of them ask the one question a merge queue cares about: does this gate pass or fail the same way twice. I wired six of these frameworks into a real GitHub Actions merge queue and ran them against production PRs for about eight months. The ones that gate cleanly share a single property: deterministic checks that return an exit code in seconds, with LLM-as-judge scores kept as non-blocking signals. The ones that flake share the opposite: nearly every metric is a judge call, so the queue blocks on a number that drifts. Ranked by "survived our merge queue," Promptfoo and DeepEval came out ahead. The short list first, then per-tool notes, then when you should not gate on any of them.

The outage that set the ranking

Two years ago I put an LLM-as-judge metric on our merge queue with a 0.8 threshold. It looked clean in the demo. Three weeks later it blocked fourteen PRs and a release over a weekend, because the judge scored the same unchanged output 0.83 on Friday and 0.78 on Monday. Same prompt. Same model. No seed. I killed the gate at 1am from my phone and we shipped fine. The regression it was "protecting" us from never existed.

That is the lens for this whole piece. A CI gate has one job: fail when something broke, pass when it did not, and do it the same way every time. An eval framework can carry the best metrics in the world and still be a bad gate if those metrics wobble. The feature-ranked listicles miss this because a notebook never punishes you for nondeterminism. A merge queue does, at 1am, in front of the whole team.

I am not arguing that quality evals are useless. I run plenty of them. I am arguing that a merge queue is a specific, unforgiving place, and the tool that belongs there is not always the tool with the longest metric list. A gate that blocks a clean PR trains your team to force-merge past it, and once the team routinely force-merges past a gate, it has stopped protecting anything. It only adds a click everyone has learned to ignore. That is the failure mode I now rank against first.

What "survived the merge queue" means

Five things, in the order they bit me:

Determinism. Same input, same verdict. A judge call with no fixed seed is not deterministic, and no threshold tuning will fully save you.
Speed. If one eval run adds four minutes and you merge forty PRs a day, you have bought a queue backup.
Cost. Judge-graded metrics burn tokens per run. Multiply by PR count. Some months that is a real invoice nobody budgeted for.
Wiring effort. Does the tool return an exit code, or do I hand-build the pass/fail logic around it.
Signal quality. When it fails, does it point at the actual regression, or just report a lower number and shrug.

Everything below is graded on that, not on how good the metrics look in a demo. Two of those five are about determinism and its side effects, because that is what cost me the most sleep. The other three (speed, wiring, signal) are what decide whether the gate is worth keeping once it works.

How I ran it: each tool guarded the same small golden set (about 60 input/expected pairs for a support-answer feature) inside the same GitHub Actions workflow, blocking on merge. I rotated them one at a time and watched three numbers: the flake rate on unchanged inputs, the added minutes per run, and the token bill at the end of the month. I kept a metric in the blocking path only if it never flipped a verdict on an input that had not changed. That rule alone reshuffled the list.

All versions, licenses, and metric counts are as of mid-2026. Check each repo before you rely on any of it, because all of this moves.

At a glance

1. Promptfoo (MIT). CI hook: a CLI exit code, JSON output, and a maintained GitHub Action. Metrics: dozens of assertions, deterministic and graded. LLM-judge: optional. Best for CLI gating in any stack.
2. DeepEval (Apache-2.0). CI hook: a pytest wrapper (deepeval test run). Metrics: 20-plus. LLM-judge: default for most. Best for pytest-based Python gates.
3. Future AGI (Apache-2.0). CI hook: call evaluate() in your own harness. Metrics: 50-plus (local plus hybrid judge). LLM-judge: optional. Best for an eval SDK you drive from Python or TypeScript.
4. RAGAS (Apache-2.0). CI hook: call evaluate() in a script. Metrics: around a dozen RAG-specific. LLM-judge: yes, most. Best for RAG quality measurement.
5. Arize Phoenix (Elastic License 2.0). CI hook: run_evals() in a script. Metrics: a handful of evaluators. LLM-judge: yes. Best for tracing plus eval in one tool.
6. MLflow evaluate (Apache-2.0). CI hook: mlflow.evaluate() in a script. Metrics: a dozen-plus (heuristic plus genai). LLM-judge: optional. Best for eval logged alongside experiments.

Repo links and install lines are in each section. Now the details.

The pattern that decided it

Once all six were in the same workflow, the ranking stopped being about metric quality. It came down to one split: does the blocking check call a model, or not. Deterministic checks (string match, JSON-schema, regex, exact match) passed and failed the same way every run, finished in under a second, and cost nothing. Judge checks did not. Every judge-based gate I ran drifted near its threshold at least once over the eight months, and two of them blocked a clean PR at least once. So the tools that let me put deterministic assertions in the blocking path, and push judge scores into an advisory lane, came out ahead. The tools built around a judge-first metric set fell behind, not because the metrics are weak, but because a score that moves between identical runs cannot hold a merge gate. Keep that split in mind as you read the six. It explains the whole order, including why a well-known RAG library sits below a younger tool, and why the tracking and observability tools sit at the bottom even though their metrics are fine.

1. Promptfoo

Repo: github.com/promptfoo/promptfoo. License: MIT. Install: npm install -g promptfoo (or run it with npx promptfoo).

What it is. A command-line eval and red-teaming tool. You describe prompts, providers, and test cases in a YAML file, run one command, and get a pass/fail table. As of mid-2026 it ships dozens of built-in assertions split into two camps: deterministic ones (contains, equals, regex, is-json, starts-with, cost, latency) and model-graded ones (llm-rubric, factuality, answer-relevance, similarity by embedding).

How it gates CI. promptfoo eval returns a nonzero exit code the moment an assertion fails. That is the whole ballgame for a merge queue: a nonzero exit is a red check, no glue code required. It writes JSON, CSV, or HTML you can archive as a build artifact, and there is a maintained GitHub Action that comments the eval diff on the PR so a reviewer sees exactly what changed. In my run the deterministic assertions never flaked across the eight months. The only week the queue wobbled was one where I let an llm-rubric assertion sit in the blocking path, and the fix was to move it back out.

# promptfooconfig.yaml  ->  run in CI with: npx promptfoo eval -c promptfooconfig.yaml
prompts:
  - "Answer the support question: {{question}}"
providers:
  - openai:gpt-4o-mini
tests:
  - vars:
      question: "What is the refund window?"
    assert:
      - type: contains
        value: "30 days"        # deterministic: no judge, no flake, sub-second
      - type: llm-rubric
        value: "states a clear refund window"   # graded: costs a call, can flake

Strengths. The deterministic assertions are the reason it sits at the top. contains and is-json do not flake, cost nothing, and run in milliseconds. It is language-agnostic: a Python shop, a Go shop, and a TypeScript shop all wire it the same way, because it is a CLI that speaks exit codes. Config lives in version control next to the code it guards, so a bad gate change shows up in the same diff.

Limits. The YAML sprawls once you pass a few dozen cases, and there is no type safety net until you run it. The model-graded assertions carry the same judge nondeterminism as everything else here, so if you lean on llm-rubric to block merges you have reintroduced the flake you came to avoid. A Python-first team also takes on a Node dependency it may not otherwise want in the CI image.

Best for. Gating prompt and output regressions in any stack, as long as you keep the blocking assertions deterministic and treat the graded ones as advisory.

2. DeepEval

Repo: github.com/confident-ai/deepeval. License: Apache-2.0. Install: pip install deepeval.

What it is. A Python eval framework built to feel like pytest. As of mid-2026 it carries 20-plus metrics, including G-Eval (a rubric metric you define in plain language), answer relevancy, faithfulness, hallucination, contextual precision and recall, plus safety metrics like bias and toxicity. Most of them are LLM-judged.

How it gates CI. You write a normal-looking test file and run deepeval test run test_file.py. Under the hood it wraps pytest, so you inherit pytest's exit codes and its JUnit XML reporter for free. If your CI already understands pytest, it already understands DeepEval. That is the shortest path to a green check for a Python team in this whole list. In my run the pytest wiring took under an hour to stand up. The flake, when it came, came entirely from the judge-based metrics drifting near their thresholds, not from the harness.

# test_support_bot.py  ->  run with: deepeval test run test_support_bot.py
import pytest
from deepeval import assert_test
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

# The judge is an LLM, so the score is not deterministic.
# A 0.7 gate that scores 0.71 today can score 0.68 tomorrow on the same input.
CASES = [
    ("What is the refund window?", "You can request a refund within 30 days."),
    ("What is the refund window?", ""),   # empty output: the edge case that bites
]

@pytest.mark.parametrize("query,answer", CASES)
def test_answer_relevancy(query, answer):
    metric = AnswerRelevancyMetric(threshold=0.7)
    case = LLMTestCase(input=query, actual_output=answer)
    # Empty actual_output does not raise; it scores near 0 and fails the gate,
    # which is correct. The flaky judge near the threshold is the real risk.
    assert_test(case, [metric])

Strengths. The pytest wrapping is the cleanest on-ramp here for a Python team. Per-metric thresholds are explicit and live in code. The catalog is the widest in this list, so you rarely hand-roll a metric, and G-Eval lets you encode a house rule ("must cite a ticket id") without writing a scorer from scratch.

Limits. Most headline metrics are judge calls, so the flakiness story from my outage applies directly: keep thresholds loose or your queue pays for it. It nudges you toward Confident AI, the hosted product from the same team, once you want dashboards and shared datasets. And a wide catalog is a wide surface to keep pinned, because a judge-model upgrade can shift scores under a fixed threshold with no code change on your side.

Best for. Python teams that already gate on pytest and want deterministic wiring, as long as you pick a few metrics and treat the judge-based ones with suspicion near the threshold.

3. Future AGI

Repo: github.com/future-agi/future-agi. License: Apache-2.0. Install: pip install ai-evaluation.

What it is. An open-source eval SDK (fi.evals). As of mid-2026 its README lists 50-plus evaluation metrics plus guardrail scanners, behind one evaluate() call, with Python and TypeScript clients. It is one piece of a larger open-source platform, but for CI gating only the eval SDK matters, so that is all I put in front of the merge queue.

How it gates CI. There is no dedicated test runner. You construct an Evaluator, call evaluate() over your inputs, and write your own assert on the returned scores, the same shape as RAGAS below. One wrinkle worth knowing for a gate: the documented quickstart authenticates with an API key and runs against the hosted service, so the naive setup puts a network call in your blocking path. The SDK also supports local metric execution, which is the mode you want for a merge gate, because a deterministic local metric does not flake the way a judge does and does not add a per-run token bill. Turn the hybrid judge on and you inherit the same nondeterminism as everyone else in this list. The actual time sink was the harness code I had to write around evaluate() myself, because nothing here hands you a runner.

# pip install ai-evaluation   (module: fi.evals)
from fi.evals import Evaluator
# documented quickstart authenticates with API keys (hosted execution)
evaluator = Evaluator(fi_api_key="...", fi_secret_key="...")
result = evaluator.evaluate(eval_templates="...", inputs={...})   # signature abbreviated
# no test runner: read the score off `result` and assert it in your own gate

Strengths. When you run its metrics in local mode, a deterministic score is what a gate wants: same input, same score, no per-PR token bill. One evaluate() surface for both local and judge scoring keeps the harness small. The TypeScript client means a Node CI job can gate without shelling out to Python, which is rare in this list.

Limits. It is younger than DeepEval and Promptfoo and it shows. There is no first-class pytest plugin or JUnit reporter, so you write more of the harness yourself. The documented quickstart is hosted (API key), so you have to configure local execution yourself to keep a network call out of the blocking path. The community, examples, and CI recipes are thinner, which matters at 2am when the gate breaks and you are hunting for the one forum answer that does not exist yet. On raw metric breadth and CI-native ergonomics it does not beat DeepEval, and it is not the most mature option on this list. If you want a tool that gates straight out of the box, this is not the shortest path today.

Best for. Teams that already have a CI harness and want fast local metrics to call from it, in Python or TypeScript, without paying a judge-token bill on every PR.

4. RAGAS

Repo: github.com/explodinggradients/ragas. License: Apache-2.0. Install: pip install ragas.

What it is. A library focused on retrieval-augmented generation. As of mid-2026 it offers around a dozen RAG-specific metrics: faithfulness, answer relevancy, context precision, context recall, answer correctness, and a few newer ones. The metrics are well-researched and map cleanly onto the stages of a RAG pipeline.

How it gates CI. There is no runner. You build a dataset, call evaluate(), and get back a scores object. Turning that into a gate is on you: read the metric, compare to a threshold, raise.

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy
# dataset is a HF Dataset with question / answer / contexts columns
scores = evaluate(dataset, metrics=[faithfulness, answer_relevancy])
assert scores["faithfulness"] >= 0.9   # you write the gate yourself

In my run the scores moved enough between two identical runs that I could not keep it in the blocking path, so it lived in a nightly job instead. Why it sits below a younger tool here: nearly every RAGAS metric is a judge call. This ranking is about the merge queue, and judge calls are the exact thing the merge queue punishes. This is not a knock on RAGAS as software. For the narrow job of a hard merge gate, the judge dependency is what drops it below a younger tool.

Strengths. If your problem is specifically RAG quality, these are among the most thought-out metrics available, and the decomposition (retrieval versus generation) tells you where the regression lives, not just that one happened. For diagnosing a bad retrieval step, that split is worth a lot.

Limits. RAG-only by design. Judge-based, so nondeterministic and token-costly per run. Scores drift when the judge model or its version changes, which turns a green history red with no code change. You own all of the pass/fail plumbing.

Best for. Measuring RAG retrieval and generation quality, ideally in a nightly or pre-merge advisory job rather than a hard blocking gate.

5. Arize Phoenix

Repo: github.com/Arize-ai/phoenix. License: Elastic License 2.0 as of mid-2026 (source-available, not OSI-approved; confirm in the repo, because this kind of license has changed before). Install: pip install arize-phoenix.

What it is. Primarily an observability tool. It ingests OpenTelemetry traces of your LLM app and gives you a local UI to inspect them. It also ships phoenix.evals, a library with a handful of prebuilt LLM evaluators (hallucination, QA correctness, relevance, toxicity) and a run_evals harness.

How it gates CI. You can call run_evals in a script, get a dataframe of labels or scores back, and assert on the aggregate.

from phoenix.evals import run_evals, HallucinationEvaluator, OpenAIModel
# returns a dataframe of labels/scores; assert on the aggregate  (API abbreviated)
evals_df = run_evals(dataframe=df, evaluators=[HallucinationEvaluator(OpenAIModel())])
assert (evals_df["label"] == "factual").mean() >= 0.95

It works, but gating is not the design center. The product is built around the trace UI, where you sit and inspect runs. A blocking merge gate is not what it optimizes for. In my run I got more out of pointing it at captured traces than at a merge gate. As a blocker it was awkward, and I kept reaching for the UI to understand a failure instead of reading an exit code.

Strengths. If you also want tracing, this is the one tool here that does eval and observability under a single install, so your CI check and your production debugging speak the same vocabulary. The classification-style evaluators (a label, not a free-form score) are more gate-shaped than raw judge numbers.

Limits. The license is the first thing to run past legal, because source-available is not the same as open source and some orgs treat that line as a hard stop. Gating is a bolt-on, so you build the pass/fail yourself. The evaluators still lean on a judge, with the usual nondeterminism and token cost.

Best for. Teams that want tracing and eval together and will run evals mostly over captured traces, with CI gating as a secondary use.

6. MLflow LLM evaluate

Repo: github.com/mlflow/mlflow. License: Apache-2.0. Install: pip install mlflow.

What it is. mlflow.evaluate() is the LLM-eval entry point inside MLflow, the experiment-tracking platform. As of mid-2026 it offers a dozen-plus built-in metrics, split between heuristic ones (toxicity, reading-grade, exact match, ROUGE, token count, latency) and genai ones you build with make_genai_metric that call a judge.

How it gates CI. You call mlflow.evaluate() on a model or a static dataset, it returns a results object, and you read a metric off it and assert. The heuristic metrics are deterministic, which is the good news for gating. In my run the exact-match and ROUGE metrics held steady as a gate. The genai metrics behaved like every other judge here, and the run-and-experiment ceremony was more setup than a single CI check wanted.

import mlflow
# heuristic metrics are deterministic; genai metrics call a judge
results = mlflow.evaluate(data=eval_df, model_type="question-answering")
assert results.metrics["exact_match"] >= 0.9   # key names vary by version

Strengths. If you already live in MLflow, your eval numbers land next to your training runs and artifacts, with lineage, which is genuinely useful for audits and post-incident review. The heuristic metrics do not flake, so a gate built on exact match or ROUGE holds steady.

Limits. It is built for experiment tracking, not gating, so reducing a run to one clean pass/fail feels like fighting the grain. It expects a run and an experiment context, which is a lot of ceremony for a CI check. The genai metrics reintroduce judge nondeterminism, and MLflow is a heavier dependency to pull into a lean CI image than a single-purpose eval library.

Best for. Teams already standardized on MLflow that want eval logged alongside experiments, using the heuristic metrics for any actual blocking gate.

When not to gate CI on any of these

Picking the right tool does not mean you should gate at all. The best framework in the world is the wrong call in some situations, and I have watched teams (mine included) reach for a merge gate when the real problem was upstream. Gating is not free, and sometimes it is the wrong move no matter which tool you pick:

You have no golden dataset. If you cannot say what the right answer looks like, an eval gate just encodes a guess and fails at random. Build the dataset first, gate second.
Your output is open-ended. Marketing copy, brainstorms, and open chat have no single correct answer for a judge to hit. Gate the structure (valid JSON, required fields present), not the quality.
The judge bill beats the value. Forty PRs a day times a multi-metric judge run is a line item. If nobody will defend that spend, move the eval to nightly.
No one owns the drift. Judge scores move when models update. If no human owns re-baselining, the gate rots into a check everyone force-merges past, which is worse than no gate at all.
Latency breaks your SLA. If the eval adds minutes and your team merges constantly, you have traded correctness theater for a queue backup.

In those cases the answer is usually an async nightly eval, an online eval on a canary, or a deterministic structural check in CI with the quality eval running out of band where it cannot page anyone.

What I'd check first

Run the gate 20 times on one unchanged input. If the pass/fail flips even once, treat it as an advisory signal and make it non-blocking.
Time a single run and multiply by your daily PR count. If the minutes or the judge-token bill blow your queue budget, move it to nightly before you tune a single threshold.
Delete your flakiest metric and see whether one deterministic assertion (regex, JSON-schema, contains) catches the same regression. It usually does, and it never pages you at 1am.

The golden set stopped catching regressions the day traffic changed

Ethan Walker — Mon, 13 Jul 2026 05:02:20 +0000

TL;DR. Our overall eval pass rate read 0.88 through a model change and looked stable. Sliced by request language, German had fallen to 0.60 while English held near 0.90. The aggregate hid that because German was a rounding error inside the golden set even though it had grown into almost a quarter of real traffic. A bigger golden set does not fix this. Slicing every run by the production distribution, and refreshing the set from real traffic, does.

The dashboard stayed green while users complained

We shipped a prompt change and a model bump on the same afternoon. The eval ran against the golden set. Overall pass rate moved from 0.90 to 0.88. A two-point move sits at our run-to-run noise floor, so we shipped and moved on.

Four days later support forwarded a cluster of complaints, all of them German. Truncated sentences. Wrong register, formal where it should have been plain. English words leaking into German answers. The eval had flagged none of it. The number was still 0.88, green as ever.

Here is the shape of the set that produced that green. The golden set held 400 cases. We built it eighteen months earlier when the product was English-only, so roughly 370 of those cases were English and about 30 were German, added later as an afterthought. Meanwhile German requests in production had climbed from a rounding error to nearly a quarter of traffic after a market launch that quarter. So the eval was 7% German while production was closer to 22% German. A hard regression on German could move the aggregate by about two points and still be a live fire for a large and growing group of real users.

The golden set encoded last year's traffic, not this quarter's, and the single number it produced averaged over a distribution we no longer had.

Why does an average hide a slice?

An aggregate pass rate is a weighted average, and the weights are whatever mix of cases you happened to freeze into the set. If the set is 90% English and English holds at 0.90, the aggregate sits near 0.90 no matter what the other languages do. German at 0.60 carrying a 7% weight pulls the overall down by roughly 0.02. You will read 0.02 as noise. On most runs you would be right, and that is exactly what makes it dangerous.

Two things had to be true at the same time for this to bite, and both were true for us.

The slice regressed. The change helped English and hurt German at once. That is more common than people expect. A prompt edited and spot-checked against English examples can shift tokenization, instruction-following, and register in another language that nobody re-read before the merge. One model swap can lift your largest slice and quietly drop a smaller one on the same commit.

The slice grew. German had gone from a sliver of real traffic to almost a quarter of it, but the eval set never tracked the change. The group that mattered most in production was represented by the fewest cases in the test. The faster a slice grows in the wild, the more badly a stale set under-weights it.

Put those two facts together and the aggregate becomes an average over the wrong distribution. It gives a confident answer to a question we had stopped asking a year earlier. The same trap sits behind any slice key, not just language: a new input-length bucket, a big tenant you just onboarded, an intent that spiked after a UI change. Whichever slice grew fastest since you froze the set is the one the aggregate is now lying to you about.

The check: passrate_by_slice

The measurement fix is small and boring, which is the point. Tag every eval case with the slice keys you care about (language, input-length bucket, tenant, intent), compute pass rate per slice, take the delta against the overall on every run, and sort by that delta so the worst slice lands at the top of the output where you cannot scroll past it.

Here is the whole thing. Standard library, no dependencies, runs on plain Python 3.

import collections

def passrate_by_slice(rows, slice_key, pass_key="passed"):
    total, passed = collections.Counter(), collections.Counter()
    for r in rows:
        s = r.get(slice_key, "unknown")
        total[s] += 1
        passed[s] += 1 if r.get(pass_key) else 0
    overall = sum(passed.values()) / max(1, sum(total.values()))
    rows_out = []
    for s in total:
        pr = passed[s] / total[s]
        rows_out.append((s, total[s], round(pr, 3), round(pr - overall, 3)))
    return round(overall, 3), sorted(rows_out, key=lambda x: x[3])

# one eval run, sliced by request language. the model regressed on 'de' only.
run = ([{"lang": "en", "passed": True}] * 90 + [{"lang": "en", "passed": False}] * 10 +
       [{"lang": "de", "passed": True}] * 60 + [{"lang": "de", "passed": False}] * 40)

overall, table = passrate_by_slice(run, "lang")
print("overall pass rate:", overall)
for s, n, pr, delta in table:
    print(f"  {s:6} n={n:4}  pass={pr:.3f}  delta_vs_overall={delta:+.3f}")

Run it and you get:

overall pass rate: 0.75
  de     n= 100  pass=0.600  delta_vs_overall=-0.150
  en     n= 100  pass=0.900  delta_vs_overall=+0.150

The overall 0.75 is the number that would have shipped and passed the gate. The de row is the one that matters: pass 0.600, a delta of minus 0.150 against the aggregate, sitting first because the sort pushes the worst slice to the top. English reads fine at 0.900. Same eval, same run, two very different stories, and only one of them was visible before the slice existed.

This toy is exaggerated on purpose. A 0.15 slice gap is loud, and real ones rarely are. In production the per-slice deltas are small and they jitter a little between runs from sampling alone, so a single run in isolation will not tell you much. The signal is a slice delta that drifts in one direction across several runs while the aggregate holds flat. Track each slice run over run and alert on the movement, not on the absolute gap in any one snapshot. A slice that slid from minus 0.01 to minus 0.06 over three runs is worth a look even though 0.06 by itself looks like nothing.

Refresh the set from traffic, not from memory

Slicing tells you where a regression is hiding. It does not close the second gap, which is that a frozen set drifts away from production every week it sits still. The set is fixed at the moment you built it. The traffic it is supposed to represent keeps shifting. The distance between the two only grows, and every point of drift is another slice the aggregate is quietly mis-weighting.

So refresh on a cadence. Every sprint, sample recent production traffic stratified by the same slice keys, label it, and fold a fresh batch into the set. Keep a frozen core of regression cases you never want to break again, the specific failures you have already paid for once. Add current cases that reflect the mix you actually serve today. Retire cases for intents you have dropped. The set should track the live distribution, not embalm an old one.

Reweighting buys you most of the protection before you label a single new case. Score the aggregate against the current production mix instead of the historical set mix. If German is 22% of traffic this month, weight German at 22% of the number. A static set that is 7% German is quietly asserting that German is 7% of your risk, and it is wrong by a factor of three. The reweight is one dictionary of production shares, refreshed whenever the mix moves.

None of this is heavy. The slice function is twenty lines. The refresh is a weekly job that pulls a stratified sample plus a short human pass to accept or reject cases. The reweight is a lookup table. Against that you are weighing a week of a broken language behind a green dashboard, which is what the old setup actually cost us. The corpus you evaluate against has to move at the speed your traffic moves, or the number it gives you ages out from under you.

What I'd check first

The slice mix versus reality. Compare your set's language, length, tenant, and intent breakdown to last week's real traffic, and treat a wide divergence as the tell.
The per-slice deltas across runs. Diff the slice output over your last two or three runs; a slice falling while the overall holds flat is the regression.
The age of the cases. If most of the set predates your last launch, you are grading last year's product and the green number has already expired.

Your LLM-as-judge disagrees with itself between runs

Ethan Walker — Wed, 08 Jul 2026 19:51:30 +0000

Same outputs, same judge, two runs, two scores. The gate flickered red then green on a branch with zero code changes, and that flapping cost me more trust than any real regression.

The flap

I had a faithfulness gate on merge: judge scores every case, the mean has to clear 0.80. One Tuesday it failed at 0.79. I re-ran the identical job, no code change, no prompt change, and it passed at 0.82. Ran it a third time: 0.80 exactly. Nothing in the repo had moved. The judge was disagreeing with itself.

A gate that returns a different verdict on the same inputs is worse than no gate. People stop believing the red, they re-run until it goes green, and now the check is a slot machine you pull until it pays out. The regression it was supposed to catch could sail through on the lucky pull. So before I trusted that gate again I had to make the judge reproducible enough to stand on.

Where the jitter comes from

Four sources, in the order they bit me.

Sampling temperature. A judge call is a generation. If temperature is above zero the model samples, and a borderline case lands on 4/5 one time and 3/5 the next. This is the biggest lever and the easiest to miss because most SDK defaults are not zero.

Model version drift. "gpt-4o" or "claude-latest" is a moving alias. The provider ships a new snapshot, your scores shift a few points overnight, and you blame your prompt. Pin the dated snapshot, not the floating name.

Prompt ambiguity. If your rubric says "rate helpfulness 1 to 5" without anchoring what a 3 versus a 4 means, the model resolves the ambiguity differently each call. Vague rubrics convert directly into variance.

Tie-breaking. When the judge is genuinely on the fence between two scores, tiny sampling noise decides, and that decision is exactly where your threshold tends to sit.

Make it reproducible enough to gate

You will not get bit-identical determinism from a hosted model. That is fine. The goal is not zero noise, it is noise small enough that a threshold crossing means a real change and not a coin flip. Five things got me there.

Temperature 0, and a seed where the provider supports one. This alone collapsed most of my flap. Seeds help further on providers that honor them, but do not assume a seed gives you exact reproducibility across a model update.

Pin the exact judge model and prompt version in the cache key. Same discipline as any eval cache: the score is only reusable if the input, the judge snapshot, and the rubric version all match. Bump the version string whenever you touch the rubric.

Average over k judged samples, or take majority vote. One call is a sample from a distribution. k calls and a mean (or a vote for pass/fail rubrics) shrink the variance of your estimate by roughly sqrt(k). I run k=5.

Quantize the score. If you gate on a continuous 0 to 1, every hundredth flaps. Round to a coarse grid (0.0, 0.25, 0.5, 0.75, 1.0) per case so sub-grid noise stops moving the aggregate.

Version the judge prompt as code. The rubric lives in the repo, gets a version string, and changes go through review. A judge prompt edited in a UI and not tracked is a silent score change you cannot bisect.

The gate that respects the noise band

The real fix is conceptual: stop treating one judged score as ground truth. Judge k times, keep the mean and the spread, and only fail when the mean is below the threshold by more than the noise you actually measured. If the mean sits inside the noise band around the threshold, that is not a regression, it is jitter, and failing on it is how you get a flapping gate.

import statistics
from typing import Callable

def stable_judge_score(
    judge: Callable[[str, str], float],
    output: str,
    reference: str,
    k: int = 5,
    quantize_to: float = 0.25,
) -> tuple[float, float]:
    """Run the judge k times at temperature 0. Return (mean, stdev),
    each raw score snapped to a coarse grid to kill sub-grid jitter."""
    scores = []
    for _ in range(k):
        raw = judge(output, reference)          # judge must be called at temperature 0
        snapped = round(raw / quantize_to) * quantize_to
        scores.append(snapped)
    mean = statistics.fmean(scores)
    stdev = statistics.pstdev(scores) if k > 1 else 0.0
    return mean, stdev


def gate(mean: float, stdev: float, threshold: float = 0.80) -> bool:
    """Fail only when the mean is below threshold by more than the
    observed noise. Inside the noise band counts as pass, not a flap."""
    return mean >= (threshold - stdev)


if __name__ == "__main__":
    # toy judge: deterministic here, real one hits an LLM at temperature 0
    def fake_judge(out: str, ref: str) -> float:
        return 0.79 if "borderline" in out else 0.9

    mean, stdev = stable_judge_score(fake_judge, "a borderline answer", "ref", k=5)
    passed = gate(mean, stdev, threshold=0.80)
    print(f"mean={mean:.3f} stdev={stdev:.3f} pass={passed}")
    raise SystemExit(0 if passed else 1)   # this exit code is what CI reads

The raise SystemExit is the load-bearing line. That is what makes the branch protection rule refuse a real regression. Everything above it exists so that exit code means something. On my suite, moving from one raw judge call to k=5 with quantization took the run-to-run swing on that faithfulness metric from about 0.03 down to under 0.01, which was finally tight enough that a red meant a real drop and people stopped re-running to dodge it.

One caution on k: more samples cost more judge calls and more wall-clock, so I only spend the k on the cases near the threshold, and run the obviously-passing and obviously-failing cases once. The noise only matters where the decision is close.

What I'd check first

Log temperature on the judge call. If it is not zero, nothing else you do about jitter matters until it is.
Diff the judge model string between the run that passed and the run that failed. A floating alias silently swapped a snapshot on you more often than you would think.
Measure the run-to-run stdev of your gated metric before you trust the gate. If the swing is wider than the margin your threshold sits on, you are gating on noise and the red is meaningless.