LLM judge rubrics in 2026: promptfoo vs DeepEval vs my builder

Use promptfoo if your evals already live in YAML and CI. Use DeepEval if you're in pytest and want G-Eval's generated reasoning steps. Write and validate the rubric before either tool sees it. On my 40-reply test, fixing the rubric moved agreement with my hand labels more than switching frameworks did. Explicit score anchors cut verdict flips more than any tool setting I touched.

Quick disclosure: the LLM Judge Rubric Builder I link to below is one I built. I'd tried four other approaches, mostly spreadsheets and hand-written YAML. None of them warned me when my weights summed to 1.1 or when a pass/fail criterion somehow had five score anchors. It's free and runs client-side. No signup, and your rubric never gets uploaded anywhere. If you have a better one, tell me.

The task: 40 refund replies, graded on October 2

On Friday, October 2, I sat down to answer a question that had bugged me for weeks. Does the support assistant I help maintain actually follow our refund policy, or does it just sound confident?

The input was 40 replies the bot had generated to real refund requests (anonymized). Each one was paired with the customer's message and our refund policy, a 312-word doc covering a 30-day window, prorated annual plans, and no refunds on add-on credits once they're spent.

The rubric I wanted:

• policy_accuracy, weight 0.5, Likert 1-5. Does the reply state the policy correctly for this customer's situation?
• next_step, weight 0.3, Likert 1-5. Does the customer know exactly what happens next and when?
• tone, weight 0.2, pass/fail. No blame, no groveling.

Pass threshold: a weighted score of 0.7 or higher after normalizing each criterion to 0-1.

Here's one reply from the set:

"Hi Dana, you're within the 30-day window, so I've started a full refund of $49.00. You'll see it in 5-7 business days. The add-on credits you used (112 of 200) aren't refundable."

My expected output for that one: policy 5, next step 4, tone pass. That works out to 0.5 + 0.225 + 0.2 = 0.925, a clear pass.

Before touching any tool, I hand-labeled all 40 myself. 29 passed, 11 failed. Every number below is agreement with those labels. One person, one dataset, one afternoon, so treat this as an anecdote rather than a benchmark.

Promptfoo's llm-rubric: 120 judge calls and one quiet typo

I started with promptfoo because I already had it in CI for a different project. Each criterion became its own llm-rubric assertion with a weight, and the grader returns a pass flag, a 0-1 score, and a reason string. Three assertions times 40 replies is 120 judge calls.

Setup took about 25 minutes. The full run took 3 minutes 12 seconds and cost $0.94 on the grader model I picked.

First-run agreement with my labels: 34 of 40. Not bad.

Then I found the typo. I'd written weight: 0.3 on the tone assertion instead of 0.2, so my weights summed to 1.1. Promptfoo didn't complain at all. As far as I can tell from the output, it computes a weighted average over whatever weights you give it, which is a reasonable design choice. It also meant my tone criterion had quietly been worth more than I intended, and I'd have shipped that config if I hadn't been staring at one weird score for 20 minutes.

Honestly this annoyed me for a day. Not at promptfoo, at myself. So I wrote a tiny checker that runs before any eval framework touches the rubric:

import json
import math

RUBRIC = json.loads("""
{
  "name": "refund-support-replies",
  "pass_threshold": 0.7,
  "criteria": [
    {"id": "policy_accuracy", "weight": 0.5, "scale": "likert_1_5"},
    {"id": "next_step", "weight": 0.3, "scale": "likert_1_5"},
    {"id": "tone", "weight": 0.2, "scale": "pass_fail"}
  ]
}
""")

SCALES = {
    "likert_1_5": (1, 5),
    "pass_fail": (0, 1),
    "percentage": (0, 100),
}


def validate(rubric):
    problems = []
    total = sum(c["weight"] for c in rubric["criteria"])
    if not math.isclose(total, 1.0, abs_tol=1e-6):
        problems.append(f"weights sum to {total:.4f}, expected 1.0")
    for c in rubric["criteria"]:
        if c["scale"] not in SCALES:
            problems.append(f"{c['id']}: unknown scale {c['scale']!r}")
    return problems


def normalize(scale, raw):
    lo, hi = SCALES[scale]
    if not lo <= raw <= hi:
        raise ValueError(f"score {raw} outside {scale} range {lo}-{hi}")
    return (raw - lo) / (hi - lo)


def weighted_score(rubric, raw_scores):
    return sum(
        c["weight"] * normalize(c["scale"], raw_scores[c["id"]])
        for c in rubric["criteria"]
    )


if __name__ == "__main__":
    problems = validate(RUBRIC)
    if problems:
        raise SystemExit("\n".join(problems))
    judge_output = {"policy_accuracy": 4, "next_step": 3, "tone": 1}
    score = weighted_score(RUBRIC, judge_output)
    verdict = "PASS" if score >= RUBRIC["pass_threshold"] else "FAIL"
    print(f"{score:.3f} {verdict}")

Run it and you get 0.725 PASS. Change the tone weight to 0.3 and it exits with weights sum to 1.1000, expected 1.0. It's 50 lines and it would have saved me an evening.

The other promptfoo wrinkle: llm-rubric thinks in 0-1 scores, and I wanted Likert anchors. I ended up writing the five anchor descriptions into the rubric text and telling the grader to report (level - 1) / 4. It worked. It also felt like I was fighting the tool a little.

DeepEval's G-Eval: better reasons, more drift

DeepEval was next. Each criterion became a GEval metric with a criteria string and a threshold. If you don't pass explicit evaluation_steps, G-Eval generates them from your criteria first and then scores against those steps, landing on a 0-1 score with a reason.

Setup took closer to 40 minutes, mostly because I had to decide how to combine three independent metric scores into one weighted verdict. DeepEval scores each metric on its own terms, so the weighting was on me. I reused weighted_score from the script above. The run took 4 minutes 48 seconds and cost $1.31, a bit more than promptfoo, which I'd guess comes from the extra step-generation calls.

First-run agreement: 31 of 40. Worse than promptfoo.

The reason strings were easily the best of the bunch, though. When G-Eval failed a reply on policy_accuracy, it usually named the exact policy clause the reply got wrong. Promptfoo's reasons were shorter and sometimes just restated the score.

The drift surprised me. I re-ran each setup three times. With generated steps, DeepEval flipped the pass/fail verdict on 7 of the 40 replies at least once. I'm fairly sure that's because the steps got regenerated between runs and came out slightly different each time. Once I pinned evaluation_steps by hand, flips dropped to 3 and agreement rose to 35 of 40.

Promptfoo flipped 4 across its three reruns, for comparison. So the default DeepEval setup was the noisiest, and the pinned one was the steadiest. That's a config choice more than a tool verdict.

My rubric builder: the step before either of them

The third "tool" belongs to a different category, and I want to be upfront about that. My builder doesn't call a model or run evals. It's the authoring step: you define criteria, weights, a scale per criterion (Likert 1-5, pass/fail, or percentage), score anchors, and it spits out rubric YAML or JSON plus a judge-prompt scaffold you can paste into whatever runner you use.

The validation is the part I built it for. It checks that weights sum to 1.0, and it checks that each criterion's anchors match its declared scale. When I rebuilt the refund rubric in it, it flagged two things in about 10 minutes. The 1.1 weight sum was the first. The second I hadn't noticed at all: my tone criterion was declared pass/fail but I'd copied five Likert anchors into its description from another criterion. Both promptfoo and DeepEval had happily graded with that contradiction for days.

Then I did the experiment I actually cared about. I took the cleaned-up anchors and judge prompt from the builder and pasted them into both frameworks, changing nothing else.

• Promptfoo went from 34 to 37 of 40.
• DeepEval (with pinned steps) went from 35 to 36 of 40.

I don't fully know why promptfoo gained more. My best guess is that its llm-rubric prompt leaned harder on my rubric text, so better text helped it more, while G-Eval's own step structure was already compensating for some of my sloppiness. I could be wrong about that.

Things the builder doesn't do, so nobody's disappointed: it doesn't run anything, it doesn't keep history, and its YAML export isn't promptfoo's config schema. You'll reshape it by hand into assert blocks. That reshaping took me about 6 minutes.

Scores, and the one I'd actually use

Criterion	promptfoo llm-rubric	DeepEval G-Eval	Rubric builder
Setup to first result	25 min	40 min	10 min (authoring only)
Agreement, first run (of 40)	34	31	n/a
Agreement after cleaned anchors	37	36	n/a
Verdict flips across 3 reruns	4	7, or 3 with pinned steps	n/a
Cost per 40-reply run	$0.94	$1.31	$0
Warns when weights don't sum to 1.0	No, averages what you give it	No, you combine scores	Yes
Catches scale/anchor mismatch	No	No	Yes
Reason text quality	Short, usable	Longest, most specific	n/a
Fits into CI	CLI and YAML config	pytest	Export only

My actual setup now, ten days later: I write the rubric first and validate it, then run it in promptfoo in CI. If a criterion is failing a lot and I need to understand why, I run that slice through DeepEval with pinned steps, because the reasons are better for debugging.

If I could only pick one runner, it'd be promptfoo. Cheaper, faster, fewer flips with default settings, and it was already in my pipeline. If you live in pytest and don't have a YAML-driven eval setup, I'd pick DeepEval and pin your steps on day one. Don't skip that part.

The bigger lesson was humbling. I spent the first few days comparing frameworks when the rubric was the weakest link. A 0.3 that should've been a 0.2, and five anchors on a pass/fail criterion, cost me more agreement than any framework difference did. If you want the validation step without writing your own checker, the LLM Judge Rubric Builder does the weight and scale checks in the browser and gives you a judge prompt to start from. Or copy the Python above. It catches the weight bug, which was the one that actually bit me.

One caveat I keep coming back to: 40 replies is small, and I labeled them myself. If your criteria are fuzzier than "did it state the refund policy correctly," expect more drift than I saw, whichever runner you pick.

Written with AI assistance and human review. Try the tool at aidevhub.io/llm-judge-rubric-builder.