DEV Community: Ethan Walker

Your eval criteria are code. Version them like code.

Ethan Walker — Wed, 10 Jun 2026 16:54:37 +0000

A judge prompt is an implementation. The criterion it encodes is a contract, and ours rotted for three months before anyone noticed.

TL;DR: We version our prompts and our eval datasets in git. We never versioned the meaning of our eval criteria, the human-readable definition of what "complete" or "helpful" is supposed to mean. So when three people tweaked the "completeness" judge prompt over a quarter, the criterion drifted, kappa fell from about 0.70 to 0.55, and no diff explained why. We started treating each criterion as a versioned contract (a short definition, an owner, a date) kept separate from the judge prompt that implements it. Here is the shape that worked.

The silent rot
Our "completeness" criterion was a judge prompt, versioned in git like everything else. Over about three months, three different engineers edited it, each to fix a specific false positive they had hit. Every edit was reasonable on its own. The cumulative effect was that the criterion now meant something none of them had agreed to: stricter on multi-part answers, looser on follow-ups. Agreement with human labels slid from roughly 0.70 to 0.55. The prompt history was all there in git, but a diff of prompt text does not read as a definition. Nobody could answer the simple question: what is this criterion supposed to mean now.

Implementation versus contract
The judge prompt is the implementation: how we operationalize a criterion for a specific model. The contract is the human-readable definition: what the criterion means, who owns it, when it was last agreed. We had the implementation under version control and the contract nowhere. So the implementation could drift away from the (unwritten) contract one reasonable edit at a time, and nothing flagged it.

The contract shape that worked

criterion_id: completeness_v3
summary: answer addresses every sub-question in the user's message; a partial answer fails
owner: the eval lead for this surface
last_agreed: a dated review, re-confirmed when the prompt changes
judge_prompt_ref: git sha of the implementation
canonical_examples: two frozen passes, two frozen fails

from dataclasses import dataclass
from datetime import date

@dataclass
class Criterion:
    id: str
    summary: str            # the human-readable contract, not the judge prompt
    owner: str
    last_agreed: date
    judge_prompt_ref: str   # git sha of the implementation
    canonical_examples: list  # 2 pass, 2 fail, frozen as the meaning regression test

What changed in practice
Editing the judge prompt now forces you to touch the contract: bump last_agreed, update the summary if the meaning moved, and re-run the canonical examples. The canonical examples are the regression test for the criterion's meaning. If a prompt edit flips a frozen example, you did not fix a false positive, you redefined the criterion, and that goes through review as a contract change. Cheap edits that quietly shift meaning stopped being cheap.

The open question
The owner field is the part I am least sure how to scale. With thirty criteria and six engineers, half of them have no obvious owner, and an unowned criterion is exactly the one that rots. Assigning ownership without it turning into box-ticking is the unsolved part for us. If you have made criterion ownership stick on a real team, I want to hear how.

FAQ
Isn't the judge prompt enough documentation? No. A prompt is tuned for the model, not written for a human to read as a definition. The two drift apart, which is the whole problem.
How many criteria is too many? When you cannot name the owner of each from memory.
Which judge model? Genericize: a frontier model from a different family than the system under test. The cross-family part matters more than the exact name.

Datadog dashboards for prompt regression: the panels we actually keep

Ethan Walker — Mon, 08 Jun 2026 18:18:16 +0000

We wired our LLM eval suite into Datadog over about four months. Most of the panels we built got deleted. These are the five that stayed, and the metrics that feed them.

TL;DR: We run an LLM-as-judge eval suite on every PR that touches a prompt, and we ship the results to Datadog as custom metrics. The dashboard started with fourteen panels. We kept five. The one that catches the most real regressions is per-criterion pass-rate split out by judge criterion, not the single rolled-up pass-rate number, because an aggregate of 91 percent hid the fact that one criterion had dropped from 0.95 to 0.62. Below are the metrics we emit, the Python that submits them, the monitor config we alert on, and the panels we tried and dropped.

Some context on the setup so the rest makes sense. We are a Series-C dev-tool startup. We have a handful of prompts in production that do real work (classification, extraction, a summarization step in an agent loop). Each one has an eval set of tagged examples, somewhere between 80 and 400 per prompt. The judge is a separate model call that scores each output against a rubric. We run the suite in GitHub Actions. The eval job emits metrics to Datadog at the end of every run. Backend service health was already in Datadog, so putting eval data next to it meant one place to look during an incident instead of two.

1. Emit per-criterion pass-rate, not just the rolled-up number

This is the one that earns its place. Our judge scores each output against multiple criteria. For the extraction prompt it is four: correct fields, no hallucinated fields, format valid, no refusal. Early on we only emitted one number, prompt_eval.pass_rate, the fraction of examples that passed every criterion. That number is fine for a smoke test and useless for debugging.

The problem showed up on a prompt change that looked clean. Overall pass-rate went from 0.93 to 0.91. Two points. Nobody would block a PR on two points. But underneath, the "no hallucinated fields" criterion had dropped from 0.96 to 0.71, and "format valid" had gone up enough to mask it in the average. We were trading correctness for formatting and the rolled-up number said everything was basically fine.

So now every criterion gets its own metric, tagged. The metric name stays prompt_eval.pass_rate and the criterion rides as a tag. That keeps the metric count sane and lets you graph all criteria on one panel.

# eval_metrics.py
# Submits eval results to Datadog after a run completes.
from datadog import initialize, api
import os, time

initialize(api_key=os.environ["DD_API_KEY"], app_key=os.environ["DD_APP_KEY"])

def submit_eval_metrics(prompt_name, git_sha, results):
    now = time.time()
    base_tags = [f"prompt:{prompt_name}", f"git_sha:{git_sha[:12]}", "env:ci"]
    series = []
    for criterion, rate in results["per_criterion"].items():
        series.append({"metric": "prompt_eval.pass_rate", "points": [(now, rate)],
                       "type": "gauge", "tags": base_tags + [f"criterion:{criterion}"]})
    series.append({"metric": "prompt_eval.pass_rate", "points": [(now, results["overall_pass_rate"])],
                   "type": "gauge", "tags": base_tags + ["criterion:overall"]})
    series.append({"metric": "prompt_eval.judge_kappa", "points": [(now, results["judge_kappa"])],
                   "type": "gauge", "tags": base_tags})
    series.append({"metric": "prompt_eval.token_cost", "points": [(now, results["token_cost_usd"])],
                   "type": "gauge", "tags": base_tags})
    series.append({"metric": "prompt_eval.p95_latency_ms", "points": [(now, results["p95_latency_ms"])],
                   "type": "gauge", "tags": base_tags})
    api.Metric.send(series)

Two things I got wrong the first time. I submitted the criterion in the metric name (prompt_eval.pass_rate.no_hallucinated_fields) instead of as a tag. That generated a new custom metric per criterion per prompt, the cardinality climbed, and you cannot graph them together without listing each one. Tags fix both. The other thing: I tagged with the full 40-character git SHA, which is a high-cardinality tag value and not useful at that length. Truncating to 12 is enough to find the commit and stops the tag from exploding.

2. Track the judge against humans, or you are graphing noise

My standing opinion, and I will say it plainly: LLM-as-judge is the only scalable eval, but most teams use it wrong because they never validate the judge itself. A pass-rate panel that looks beautiful is worthless if the judge agreeing with itself is all you are measuring. We learned this the slow way on a hallucination-detection judge that ran around a 30 percent false-positive rate for weeks. The dashboard was green. Customers were not.

So prompt_eval.judge_kappa is a first-class metric now. We keep a small human-labeled holdout per prompt (200 examples, labeled by two of us, disagreements resolved by a third). Every eval run scores that holdout too and computes Cohen's kappa between the judge and the human labels. That number goes to Datadog next to the pass-rate.

The panel for it is a single timeseries with a marker line at 0.6. When kappa drifts under the line, the pass-rate numbers above it stop meaning anything and we know to re-look at the judge prompt before trusting any regression signal. In our setup kappa sits around 0.66 to 0.72 on a good prompt. When we rewrote a judge rubric badly once, it fell to 0.41 in a single run, and that drop is what told us the rubric change was the problem, not the model.

from sklearn.metrics import cohen_kappa_score

def compute_judge_kappa(human_labels, judge_labels):
    # labels: 1 = pass, 0 = fail, aligned by example id.
    if len(human_labels) != len(judge_labels):
        raise ValueError("label lists must align by example id")
    return round(cohen_kappa_score(human_labels, judge_labels), 3)

The holdout does not need to be big. It needs to be labeled by an actual person and refreshed when the prompt's job changes. We re-label maybe once a month, or whenever a prompt's scope moves.

3. Wire the monitors before you trust the dashboard

A dashboard nobody is staring at does not catch anything at 2am. The panels are for debugging once you already know something moved. The monitors are what tell you something moved. We run two kinds. The first is an absolute floor on per-criterion pass-rate. The second is a change-based monitor on the overall pass-rate, so a slow week-over-week slide gets caught even when no single run trips the floor.

Here is the per-criterion floor as a Terraform datadog_monitor resource, so it lives in version control instead of someone's browser tab.

resource "datadog_monitor" "extraction_no_hallucinated_fields" {
  name  = "[prompt-eval] extraction: no_hallucinated_fields below floor"
  type  = "metric alert"
  query = "min(last_3): min:prompt_eval.pass_rate{prompt:extraction,criterion:no_hallucinated_fields,env:ci} < 0.85"
  monitor_thresholds { critical = 0.85
    warning  = 0.90 }
  notify_no_data    = true
  no_data_timeframe = 60
  message = "no_hallucinated_fields for extraction fell below 0.85 on the last 3 runs. Check the most recent prompt change. @slack-eval-alerts"
  tags = ["team:ai", "prompt:extraction"]
}

A note on min(last_3). We do not alert on a single run. Eval sets have sampling noise, and one unlucky run can dip a criterion below the floor and recover on the next. Requiring three consecutive runs under the line cut our false pages down a lot. The CI check itself goes red on the first run, so the PR is already blocked. The page is for the slow drift, the red check is for the obvious break. notify_no_data: true matters more than it looks. The most common failure was not a regression. It was the eval job silently not running and the dashboard quietly going flat.

4. The five panels we kept, and the nine we dropped

The test we landed on: if a panel has not changed what someone did in the last month, it goes.

Panel	Metric	Keep or drop
Per-criterion pass-rate (one line per criterion)	prompt_eval.pass_rate by criterion	Kept. The single most-used panel.
Judge kappa vs human (marker at 0.6)	prompt_eval.judge_kappa	Kept. Tells you whether to trust everything else.
Token cost per run	prompt_eval.token_cost	Kept. A rewrite that doubles cost shows here before the bill does.
Pass-rate by git SHA (table, last 20)	prompt_eval.pass_rate by git_sha	Kept. The "which commit moved this" lookup.
p95 eval latency	prompt_eval.p95_latency_ms	Kept, barely.
Single big pass-rate number	overall pass-rate	Dropped. A green 0.91 gave false confidence.
Per-example score heatmap	per-example gauge	Dropped. Too dense, never drove a fix.
Cost cumulative sum for the month	summed cost	Dropped. A billing question, not an eval one.

The pattern in what we dropped: anything that was a different view of a number we already had a better panel for, and anything too dense to read in the ten seconds you actually look at a dashboard mid-incident. We started by copying a generic service dashboard layout, and that was a mistake. Service dashboards assume a continuous stream of requests. Eval runs are discrete events on PRs.

5. Tag everything by prompt and SHA so the board answers "which change"

The whole point during a regression is to answer one question fast: which prompt change moved this metric. Every metric we send carries prompt, git_sha (truncated), and env. The pass-rate also carries criterion. With those tags, the "which commit" table is a straight group-by on git_sha. When a criterion drops, you read the table, find the SHA, and you are looking at the diff in under a minute. We also post a Datadog event at the start of each eval run as an overlay, so a drop on the graph lines up visibly with a commit.

FAQ

Do you really need a human-labeled holdout for kappa? You need it once per prompt and refresh it occasionally. 200 examples labeled by two people is an afternoon. Without it you are trusting the judge with no check.

Why Datadog instead of the eval tool's own dashboard? We already lived in Datadog for service health. If your team does not, this is probably not a reason to adopt it. The metrics matter more than the surface they render on.

What thresholds should I start with? Do not copy mine. Run the suite on main for a week, watch where each criterion sits, set the floor a little below the normal range.

Does this replace running Promptfoo or your eval framework locally? No. The framework still runs the evals and is where you read per-example detail. Datadog is the rollup and the alerting layer on top.

Why gauge and not count or rate? A pass-rate is a snapshot value at a point in time, so gauge fits. Using the wrong type was one of my early mistakes.

What I am still chewing on

The kappa holdout goes stale when a prompt's job drifts, and I do not have a clean signal for when it has gone stale short of re-labeling. The min(last_3) window trades detection speed for fewer false pages, and I am not sure three is the right number per eval set. And the harder one: this catches regressions in the prompts I already have eval sets for. The judge can only score what the rubric asks about. The class of bug where everything passes and the customer is still wrong lives in the gap between the criteria, and I do not have a panel for the thing I forgot to measure.

If you have wired per-criterion eval alerting and found a better window than three runs, or a way to tell when a judge holdout has gone stale without re-labeling it, I want to hear it.

Switching our LLM-as-judge from 5-class to binary in CI: the patterns we kept

Ethan Walker — Wed, 03 Jun 2026 17:24:59 +0000

A few months back our LLM-as-judge ran on a 1-to-5 helpfulness scale. The CI gate stayed green because we were averaging that score. Spot-checking against humans put Cohen's kappa at 0.47. The rubric was the problem, not the tooling. Same labellers re-rating on per-criterion binary got to 0.78. The CI pipeline had to learn the new shape. This post is the engineering work that came after the methodology decision.

Not a war story. Pattern share.

What changed in our Promptfoo config

# Before: single 5-class assertion
assertions:
  - type: llm-rubric
    rubric: "Score 1-5 on helpfulness"

# After: 4 binary assertions per criterion
assertions:
  - type: llm-rubric
    rubric: "Is the answer accurate? (yes/no)"
  - type: llm-rubric
    rubric: "Is the answer grounded in the context? (yes/no)"
  - type: llm-rubric
    rubric: "Does the answer follow the required format? (yes/no)"
  - type: llm-rubric
    rubric: "Does the answer address the question asked? (yes/no)"

The first thing that breaks: your existing pass-threshold logic. The old gate was "if avg-score is below 3.5, fail." The new gate has 4 separate signals.

The threshold question

We tried three threshold patterns:

Conjunction: fail if ANY criterion drops below 90% pass rate. Strict. Caught 30% more regressions but also tripped on noise.
Weighted sum: assign weights (accuracy 0.4, groundedness 0.3, format 0.2, question-answered 0.1), fail if weighted score below threshold. Easier to tune.
Per-criterion thresholds: each criterion has its own pass-rate threshold. Catches criterion-specific regressions. Most code to maintain.

We landed on option 2 for the daily CI gate and option 3 for the weekly deep check. Option 1 we dropped after a week of false positives.

What got harder

(a) The dashboards. The old Datadog panel was one line. The new one is 4 lines plus a weighted-score line. Operators have to learn the new layout.

(b) The judge prompt itself. Each binary criterion needs its own prompt. We started with copy-paste-and-tweak; that was a mistake. The criteria need to be debated upfront and the prompts written carefully. Otherwise rater drift sneaks back in at the prompt level.

(c) Calibration set labelling cost. 4x the labels per trace. We compensated by reducing the calibration set from 200 traces to 100 traces. Still got stable kappa.

What got easier

(a) Debugging regressions. When accuracy kappa drops while groundedness holds, the prompt change broke generation, not retrieval. The single-number score was averaging away the signal.

(b) Per-criterion alerting. Format compliance kappa cratering at 3am means the JSON parser broke. Set up a dedicated alert. Page on it.

(c) The human spot-check loop. Reviewing per-criterion is faster than re-reading the full 5-class rubric. Our weekly calibration job dropped from 90 minutes to 50.

What I would tell a friend who is mid-switch

The CI plumbing is the straightforward part. The harder work goes into the judge prompts themselves. Each binary criterion deserves the same care as a feature prompt: write it deliberately, version it in git, calibrate it against humans, and watch the per-criterion kappa over time.

Default to 3 or 4 criteria. We tried 6 and the labelling cost killed us. 2 hides too much. 4 was the sweet spot in our data; your traces may need different.

Discussion

Anyone else done this switch? What criteria did you settle on, and how did the threshold tuning go?

Promptfoo is a CI gate, not an eval framework. Treating it like one cost us $4,200

Ethan Walker — Tue, 26 May 2026 18:12:09 +0000

Last Monday I logged into our billing dashboard and saw a $4,200 LangSmith spike from the weekend. Our auto-eval pipeline had been running overnight against a fresh prompt change. The Promptfoo regression suite passed 91% of its 300 questions. The release went out Monday at 9am.

By Tuesday evening, our on-call channel had 14 customer escalations about wrong refund amounts.

That is when I stopped treating Promptfoo as an eval framework.

The category error

I had built what looked like a real evaluation pipeline. 300 frozen test cases. Pass-fail thresholds. CI gate that blocked merges on any drop below 85%. A monthly review of the test set. The bookkeeping was tight.

It still missed the bugs that hit production.

The reason is a category error. Promptfoo is a regression test runner. It tells you "your prompt change did not break the cases you had already thought to test." That is useful. It is not eval. Eval requires a judge that has been validated against humans on your task. Promptfoo runs whatever judge you point it at. It does not validate the judge. We had been running an unvalidated judge against a frozen test set and calling the green result "eval."

Our judge was a GPT-4 call with this prompt:

Score the agent's response 1-5 against the expected answer.
Question: {q}
Agent response: {a}
Expected: {e}
Score (1-5):

When I hand-labeled 200 production traces over a weekend and compared them against the judge's scores, Cohen's kappa was 0.47. For a 5-class scoring problem, that is barely above chance. The judge was passing exactly the failures we most wanted to catch.

I had been measuring nothing.

The fix is two pieces

The fix took 8 weeks. Most teams I talk to have piece 1 and are missing piece 2.

Piece 1: Promptfoo stays as the CI gate

We did not throw away Promptfoo. We bounded its scope.

# .promptfoo.yaml (excerpt)
prompts: [refund_agent_v3.txt]
providers: [openai:gpt-4]
tests: !file ./tests.yaml
defaultTest:
  assert:
    - type: model-graded-fact
      value: "Matches expected refund amount and reason"
    - type: latency
      threshold: 3000

That tells you when a prompt change broke a known case. Nothing more.

Piece 2: A separate judge-validation pipeline against production traces

The piece that did not exist before is a CI step that pulls a sample of last week's production traces, asks human labelers to score them, and compares humans against the judge.

# weekly_judge_validation.py (runs every Monday 9am)
from datadog import statsd
from sklearn.metrics import cohen_kappa_score
import scipy.stats

def run():
    traces = pull_traces(days=7, n=50)
    judge_scores = [run_judge(t) for t in traces]
    human_scores = await_human_labels(traces, timeout="48h")

    kappa = cohen_kappa_score(judge_scores, human_scores)
    statsd.gauge("eval.judge.kappa", kappa)

    if kappa < 0.55:
        pagerduty.trigger(
            "judge-drift",
            details=f"kappa={kappa:.2f}, threshold=0.55"
        )

The wiring inside our GitHub Actions:

# .github/workflows/judge-validation.yml
name: Judge validation (weekly)
on:
  schedule:
    - cron: '0 9 * * 1'  # every Monday 9am UTC
  workflow_dispatch:

jobs:
  validate-judge:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.12' }
      - run: pip install -r eval/requirements.txt
      - run: python -m eval.weekly_judge_validation
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          DATADOG_API_KEY: ${{ secrets.DATADOG_API_KEY }}
          PAGERDUTY_KEY: ${{ secrets.PAGERDUTY_KEY }}

When we wired this up 8 weeks ago, kappa was 0.47. Today it is 0.68.

What we changed in the judge

The fix is structural. Three changes:

Score criteria separately. Three things instead of one 1-5 score: refund amount, denial reason, customer-facing tone. Kappa per criterion runs 0.65 to 0.74.
Force the judge to cite. The judge has to quote the expected answer portion that justifies its score.
Score against a rubric, not vibes. A 4-page rubric per criterion.

Those three changes moved kappa from 0.47 to 0.68 in 6 weeks.

Position bias and verbosity bias

Position bias: shuffled answer order, scored again, self-agreement was 71%. 29% of judgments flip based on order.

Verbosity bias: padded responses with 50 benign tokens. Padded responses scored 0.4 points higher on average.

Mitigations: randomize answer order and average. Truncate to max length before judging.

The lesson

Promptfoo is a CI gate, not an eval framework. The actual eval is the judge-validation pipeline that lives next to it.

If you only have Promptfoo, you are flying on uncalibrated faith. The judge will confidently pass exactly the failures you most want to catch, because the judge and the failures share the same training distribution.

Most teams I talk to are missing piece 2. They have Promptfoo (or DeepEval, or a custom harness). They have CI thresholds. They have a frozen test set. They do not have a judge-validation step against production traces. So they are running an unvalidated function and calling its output "eval."

Total cost of the fix: about 20 engineer-hours and $180 per month in API calls. The $4,200 weekend was the bigger number.

Three things I am still working on

The first is calibration set size. I use 200 traces per week. I suspect 100 with tighter stratification gives the same CI, but I have not run the variance experiment yet.

The second is whether cross-judge agreement can stand in as a noisy proxy for human labels. If three LLM judges agree, is that enough to skip the human pass? My hunch is yes for the obvious cases and no for the edge cases where you most need the eval, which is the worst possible failure mode.

The third, and the one I find hardest, is putting a dollar value on lost user trust when production breaks on cases the judge passed. The $4,200 was visible on the invoice. The trust hit was not. I do not know how to frame that for budget conversations with non-engineering leadership.

If you have solved any of these, I would like to compare notes.