LLM Evals in 2026: A Practitioner's Field Guide

Last week I watched a team ship a support agent that passed 94% of their eval suite and then hallucinated a refund policy on day two of production. Their eval was measuring the wrong thing, well. This guide is what I wish someone had handed me the first time I built an LLM eval harness that actually caught regressions before customers did.

What "LLM evals" actually means in 2026

An LLM eval is a repeatable measurement of a model or agent's output quality on a fixed dataset, using deterministic checks, model-graded judges, or human review, run in CI or on a schedule so you can detect regressions before shipping. In 2026 the term covers four distinct things people conflate: offline benchmarks (fixed dataset, run pre-merge), online evals (production traces sampled and scored), guardrails (runtime checks that block bad outputs), and judge calibration (measuring whether your LLM-as-judge agrees with humans). If your team says "we have evals" and cannot tell you which of the four they mean, they do not have evals. They have a vibes-based test set.

The mental model I use with clients: evals are unit tests for probabilistic systems. They will not be green/red. They will be distributions. Your job is to make the distribution shift visible.

The eval framework landscape, honestly compared

I have used all four of these on real projects in the last 18 months. Here is how I actually pick between them.

Framework	Best for	Self-host	LLM-judge built in	Trace-linked	Where it hurts
Braintrust	Teams that want a polished UI and fast iteration on prompts	No (SaaS)	Yes	Yes	Cost at scale, vendor lock on eval definitions
LangSmith	Teams already deep in LangChain/LangGraph	Limited (enterprise)	Yes	Yes, tightly	Framework gravity: hard to leave once your traces live there
Promptfoo	Config-as-code, CI-first, model comparison	Yes	Yes	Weak (bring your own tracing)	UI is basic, no real prod monitoring
Ragas	RAG-specific metrics (faithfulness, context precision)	Yes	Yes	No	Only useful if your problem is retrieval, judge metrics are noisy without calibration

My default stack for a new client engagement: Promptfoo in CI + a lightweight custom dashboard on Postgres for online evals. I add Ragas only when the RAG faithfulness question is central. I recommend Braintrust when a non-engineering PM needs to run experiments without opening a PR. LangSmith is fine if you are already on LangChain and not planning to leave. I do not use it as a starting point in 2026 because the migration cost later is real.

The single biggest mistake I see: teams pick the framework before they have written 20 real eval cases by hand. The framework is 10% of the problem. The dataset is 90%.

The 12 failure modes I check for on every LLM system

This is the table I keep in my head, and now on paper. Each row is a failure I have shipped or watched someone else ship in production. Each has a specific check.

#	Failure mode	What it looks like	How I catch it
1	Prompt drift	Small prompt edit tanks 3 unrelated cases	Snapshot eval on every prompt PR, diff scores
2	Hallucinated citations	Agent invents URL, doc ID, or policy	Regex check against known-good ID list + faithfulness judge
3	Tool-call malformation	JSON args missing required field	Schema validation on every tool call, fail loud
4	Silent tool failure	Tool returns empty, agent proceeds as if it worked	Assert non-empty tool return, log to trace
5	Context poisoning	Bad retrieval chunk flips the answer	Retrieval eval separate from generation eval
6	Judge bias	LLM judge always prefers longer answers	Calibrate judge against 50 human labels quarterly
7	Refusal creep	Model version upgrade adds unwanted refusals	"Should answer" set: track refusal rate over time
8	Loop / infinite planning	Agent replans same step forever	Max-steps guardrail + step-count metric
9	Cost blowup	One query burns 400k tokens	Per-trace token budget alert
10	Latency regression	p95 doubles after model swap	Latency in the eval report, not just quality
11	PII leakage	Model echoes user data into logs or downstream calls	Regex + entity detector on outputs
12	Eval set rot	Test cases become trivial after months of prompt tuning	Rotate 10% of eval set per quarter with real prod failures

If you build a scorecard around these 12, you will catch roughly 80% of what actually breaks LLM systems in production. The remaining 20% is domain-specific, and that is where the interesting engineering happens.

The workflow I use to build an eval suite from scratch

Every engagement follows the same rough sequence. It takes about a week for a scoped agent, three weeks for something ambitious.

Step 1: Collect 50 real inputs, not synthetic ones. If the system is not live yet, sit with the people who will use it and ask them to type the messages they would actually send. Synthetic eval inputs generated by GPT are the reason so many production systems fail on real traffic. The distribution is wrong.

Step 2: Hand-label expected behavior for each input. Not expected output, expected behavior. "Should refuse", "should call search_docs with query about pricing", "should ask a clarifying question". Behavior labels survive prompt changes. Exact-output labels do not.

Step 3: Write deterministic checks first. Anything you can catch with a regex, a JSON schema, or a substring match, catch that way. Deterministic checks are free, fast, and never disagree with themselves. I usually get 30-40% of my checks deterministic.

Step 4: Add LLM-as-judge for the rest, then calibrate. Write your judge prompt. Run it on 50 cases. Have a human (usually me and the domain expert) label the same 50. Compute agreement. If your judge disagrees with humans more than 15% of the time, the judge prompt is broken, not the model. I wrote about this in more depth in my post on Hamel's method vs a lean in-house setup.

Step 5: Wire it into CI and a nightly cron. CI blocks PRs that regress the eval score by more than 2 points on any critical metric. The nightly job runs against a sample of yesterday's production traces so you catch drift the fixed set will not.

Step 6: Publish the numbers where the team sees them. A Slack post every morning with the score, a small table of what regressed, and a link to the diff. If the number lives in a dashboard nobody opens, you do not have evals. You have decoration.

Real numbers from a recent build

For a content generation agent I shipped this year, the eval investment looked like this. I am sharing the actual numbers because "evals are worth it" without numbers is a slogan.

• Time to write initial 60-case eval suite: 9 hours
• Time to calibrate the LLM judge to 91% human agreement: 4 hours
• Model + judge cost per full eval run: $0.42
• CI overhead added per PR: ~90 seconds
• Regressions caught before merge in first 8 weeks: 11
• Estimated cost of those 11 regressions reaching prod (rework + reputation): I do not know, but the two that involved wrong pricing claims would have been bad.

The eval suite paid for itself the first time it caught a prompt tweak that broke tool calling on 4 out of 60 cases. That took 90 seconds in CI. Debugging it in production would have taken hours and would have shipped wrong outputs to real users first.

Where LLM-as-judge quietly lies to you

This is the section I wish someone had written for me two years ago. LLM judges have systematic biases and if you do not measure them, your eval score is a lie you are telling yourself.

Position bias. In pairwise comparison ("is A or B better?"), most judges prefer whichever answer comes first. Fix: run every pair twice, swapped, and only count agreement.

Length bias. Judges prefer longer answers even when shorter is correct. Fix: include "concise" as an explicit criterion in the rubric, or normalize by length.

Self-preference. GPT-4 class judges prefer GPT-4 class outputs. Claude prefers Claude. If you use the same model family to generate and judge, you are grading your own homework. Fix: use a different family for the judge, or triangulate with two judges.

Rubric ambiguity. "Rate helpfulness 1-5" produces noise. "Score 1 if the answer cites the correct policy section by ID, 0 otherwise" produces signal. Push rubrics toward binary or ternary decisions with explicit criteria whenever you can.

I recalibrate judges every time I change the underlying model version. When Anthropic or OpenAI ships a new snapshot, judge behavior shifts. Skipping calibration and trusting the score is how teams end up with green dashboards and angry users.

What I would do if I were starting today

If you are a founder or head of engineering standing up an LLM system in 2026, this is the shortest path I know:

1. Do not pick a framework yet. Write 50 real eval cases in a spreadsheet, by hand.
2. Ship Promptfoo in CI on week one. It costs nothing, it works, and the config format is portable if you outgrow it.
3. Track the 12 failure modes above as a checklist, not a philosophy. Tick them off one by one.
4. Calibrate your judge quarterly against fresh human labels. Put it on the calendar.
5. Rotate 10% of the eval set every quarter with real production failures. Otherwise the set rots and your scores lie.
6. Do not spend more than 20% of your engineering budget on the eval framework itself. The dataset and the judge rubric are the leverage points. The tooling is a commodity.

The teams I see get this right treat evals the way good backend engineers treat observability: a first-class part of the system, built as the system is built, not bolted on after the first outage.

Closing

Evals in 2026 are not a checkbox. They are the difference between an LLM system you can operate with confidence and one you are constantly patching in response to angry users. If you are somewhere in that gap and want a second set of eyes on your eval strategy, or want help standing one up from scratch, I take a small number of these engagements each quarter: lazar-milicevic.com/#contact. More posts on production LLM work live on the blog if you want to keep reading.

Last updated: July 2026. Written by Lazar Milicevic, Senior Technical Engineer, 10+ years in AI automation, cloud architecture and B2B SaaS. Founder of BizFlowAI.