Evals: The Skill 56% of Jobs Want
Here is a strange fact about AI engineering in 2026. In the 390 real job postings behind Part 1 of this series, evaluation appeared in 56 percent of listings. That puts evals ahead of PyTorch at 33 percent, ahead of RAG at 26 percent, and far ahead of LangChain at 10 percent. Only LLMs themselves at 63 percent and Python at 59 percent rank higher. Yet evals are the part almost nobody teaches. Tutorials show you how to build a RAG pipeline or an agent. Almost none show you how to prove it works, or how to notice when it quietly gets worse.
This part fixes that. You will learn what an eval is, the four ways to grade AI output, why LLM judges are useful and where they lie to you, and which tools to reach for in 2026. Then I run a real eval on two versions of the same tiny RAG system, and show you every number. The headline result is not what I expected: the grounded prompt lifted answer accuracy from token F1 0.64 to 0.79, the judge wrongly flagged two correct answers as unfaithful, and the buggiest part of the whole setup turned out to be my own abstention grader. If that sounds small, good. Evals are a boring superpower, and boring is the point.
Part 5 of the AI Engineer Series. Part 1 mapped the roadmap from 390 real job postings, Part 2 covered the Python that matters for AI work, Part 3 opened the black box of tokens and temperature, and Part 4 built RAG from zero. This part adds the missing piece: measurement. Part 6 builds your first agent with LangGraph, and evals are what will keep that agent honest.
The four days that proved evals matter
On 25 April 2025, OpenAI shipped a GPT-4o update meant to make ChatGPT more helpful. Within days, users were posting screenshots of the model agreeing with obviously bad ideas, validating doubts, fuelling anger, and urging impulsive actions. OpenAI started rolling the update back on 28 April and finished the rollback for free users by 29 April. Four days from launch to full retreat, for the most used AI product on earth.
The postmortem is the interesting part. OpenAI wrote on 2 May 2025 that their offline evals "generally looked good" and A/B tests were positive. The problem was not that they ran no evals. It was that they ran the wrong evals. In their own words: "We also didn't have specific deployment evaluations tracking sycophancy." A new reward signal based on thumbs-up and thumbs-down feedback had weakened the signal that kept sycophancy in check, and no eval was watching that exact behaviour. Some expert testers had said the model "felt" slightly off, but the metrics were green, so it shipped.
That is the whole case for evals in one story. Evals do not prevent every failure. They decide which failures you can see before your users do. OpenAI had a large eval suite and still missed a behaviour change that millions of people noticed in a weekend, because the suite measured what they thought to measure, not what changed. Every eval suite has this blind spot. The job is to shrink it on purpose, by writing evals from real failures instead of from imagination.
What an eval actually is
An eval has three parts, and only three. A task: one input with defined success criteria. A grader: the logic that scores the output. And a score: the aggregate over many tasks. Anthropic's January 2026 guide on agent evals puts it the same way: an eval is a test for an AI system where you give it an input, apply grading logic to the output, and measure success. Everything else is packaging.
The part most teams skip is choosing the tasks. The standard advice from people who run evals for a living is to start from real failures, not from a brainstorm. Anthropic recommends 20 to 50 simple tasks drawn from real failures as a great start. Hamel Husain's method is even more concrete: sample 50 to 100 real traces, note the first error in each, cluster them into failure modes, then write evals for the highest-impact modes. He calls error analysis the most important activity in evals, and he is right, because an eval written before you have looked at real failures usually tests the easy cases and misses the ones that cost you users.
Three rules follow from this. First, grade the outcome, not the path. Anthropic's guide tells a story about an agent that solved a flight-booking task by finding a loophole in the policy. It failed the eval as written, but it came up with a better solution for the user. If your grader demands one exact path, it punishes good agents for being creative. Check the final state of the world, not the transcript. Second, start early and do not wait for the perfect suite. A 20-case golden set you run on every change beats a 500-case suite you never finish building. Third, an eval suite is a regression suite. Capability evals ask whether the system can do something yet. Regression evals should sit near 100 percent, and when one drops, something you shipped broke something that used to work.
The four ways to grade AI output
Every grader falls into one of four families. Each one fails differently, which is why serious setups combine them.
Deterministic graders. Exact match, token F1, ROUGE, BLEU, regex checks, unit tests, "does the code run". They are free, instant, and perfectly repeatable. They are also blind. Token F1 punishes a correct answer phrased differently and rewards a fluent answer that invents facts, as long as it shares words with the reference. Use them for anything with one right answer: dates, numbers, IDs, code that must execute. Do not use them to judge open-ended text.
Retrieval graders. For RAG, grade the retriever separately from the generator, or you will fix the wrong half. Ragas, the open-source RAG eval framework, splits this into four metrics: faithfulness and answer relevancy diagnose the generator, while context precision and context recall diagnose the retriever. Context precision asks whether the retrieved chunks are relevant, with relevant chunks ranked first. Context recall asks whether retrieval fetched everything the reference answer needs. The split matters because the fix is different: bad retrieval means better chunking or hybrid search, bad faithfulness means a stricter prompt or a better model. I covered Ragas hands-on in a full guide, including the thresholds I use as a starting point: faithfulness above 0.80 and context recall above 0.80, calibrated from your own baseline, not from a blog post.
LLM-as-judge graders. A second model grades the first model's output against a rubric. This is the 2026 standard for anything subjective, and it is also the most misunderstood tool in the field. The idea comes from the MT-Bench paper by Zheng and colleagues (2023), which found that GPT-4 as a judge matched human preferences with over 80 percent agreement, about the same level as agreement between two humans. That number gets quoted everywhere. The fine print gets quoted nowhere: on clear cases the agreement was 85 percent, but when ties were included it fell to 66 percent, and human-to-human agreement fell the same way. A judge is a measuring instrument with error bars, not an oracle.
Human graders. The gold standard, and the slowest and most expensive. Humans are what you calibrate everything else against. The working pattern is: humans grade a sample, the judge grades everything, and you check the judge against the humans on the sample. If the judge agrees with humans at 80 percent or better on your rubric, you can trust it for day-to-day regression checks. Re-check whenever the judge model changes, because scores drift when the judge changes.
LLM judges: useful, biased, and noisy
Three biases from the MT-Bench paper show up in every judge deployment. Position bias: a judge prefers whichever answer it reads first. Verbosity bias: it prefers longer answers, even when the extra words add nothing. Self-enhancement bias: it prefers answers written in its own style, which means a judge from the same model family as the generator grades its own family generously. All three are fixable in part: swap answer order and average, penalise length in the rubric, use a judge from a different model family than the generator. None of them is fixable completely, which is why judge scores need error bars.
The Ragas paper gives a sense of how good a well-built judge pipeline can be. On the WikiEval set, Ragas metrics agreed with human annotators at 0.95 for faithfulness, 0.78 for answer relevance, and 0.70 for context relevance, against direct GPT scoring baselines of 0.72, 0.52, and 0.63. Faithfulness is the easiest thing to judge automatically, because it is a check against a fixed context. Relevance is harder, because it needs taste. That matches my own experience: use the judge for groundedness checks, and use humans for anything that needs judgement about quality.
My opinion, after running the experiment below: treat a single judge run the way you would treat a single unit test run on a flaky test. It tells you something, but not enough to ship on. Re-run the judge, or run two judges and take the majority, before you call a 3-point difference a real regression. The experiment below shows why.
The experiment: one sentence, two systems, real numbers
Enough theory. I built a miniature RAG system and evaluated it twice, changing exactly one thing between runs: the prompt. Everything else is fixed, so any difference in the scores is the prompt's doing.
The setup. The corpus is 8 short passages drawn from numbers already published in this series: chunking recall from the chunking tests, the Mayo Clinic accuracy lift, the asyncio benchmark from Part 2, the RRF formula from the hybrid search guide, the cost savings from the LLM bill article, and token counts from Part 3. The test set is 12 questions: 10 answerable, each tied to one gold passage, plus 2 unanswerable questions whose answers are nowhere in the corpus. The retriever is plain TF-IDF, top 3, built with numpy in about 30 lines. The generator and the judge are both Gemini Flash-Lite.
The two systems. System A, the naive version, gets the retrieved context and the instruction "Answer the question." System B, the grounded version, gets the same context plus one sentence: "Answer using ONLY the context below. Do not use outside knowledge. If the answer is not in the context, reply with exactly: I don't know". Same retriever, same model, same questions. Only the prompt differs.
The metrics. Retrieval recall@1 and recall@3 measure whether the gold passage was retrieved. Token F1 against a reference answer measures answer correctness deterministically. Faithfulness is graded by the LLM judge: every claim in the answer must be supported by the retrieved context. Abstention is measured on the 2 unanswerable questions. I also re-ran the judge on 8 answers to check whether it agrees with itself.
The results
Retrieval first. Recall@3 was 1.00 and recall@1 was 1.00 across the 10 answerable questions, identical for both systems because they share the retriever. That is the point of measuring retrieval separately: if recall were low, no prompt change could fix it, and I would be tuning the generator to cover for a retrieval problem. On this tiny corpus the retriever did its job, so the differences below belong to the prompt.
On the 10 answerable questions, System A averaged token F1 of 0.643 and System B averaged 0.791. The grounded prompt produced answers closer to the references, mostly by being terse and copying the exact numbers instead of rephrasing them. The judge told a different story: it marked 10 of 10 System A answers faithful and only 8 of 10 System B answers faithful. Two metrics, two winners. One of them is wrong, and manual inspection says it is the judge. System B's two "unfaithful" answers were "ticket KB-2024-7831" and "Context precision plus context recall". Both are exactly correct, and both scored well on token F1. The judge penalised short answers that state the fact without showing their work. That is a 2-in-20 judge error rate on this run, both false negatives, and it is the most useful finding in the whole experiment: a judge that disagrees with your deterministic metric is not automatically right. Read the answers it flags before you act on the score.
The unanswerable questions produced the strangest result, and it was my bug, not the model's. My abstention grader counted only the exact phrase "I don't know", so System A scored 0 of 2 and System B scored 2 of 2. But read what System A actually said when asked which embedding model scored highest on the MTEB leaderboard in 2026, a question nothing in the corpus can answer:
Based on the context provided, there is no mention of the MTEB leaderboard or embedding models in 2026. Therefore, the question cannot be answered using the given text.
And System B, same question, same context:
I don't know
System A refused correctly. It just refused in its own words instead of my magic phrase, and my grader marked it wrong. The same happened on the second unanswerable question, about LogicStack's paying customer count: System A wrote "there is no mention of LogicStack's Shopify-Xero service or the number of paying customers it has. Therefore, I cannot answer the question." A correct refusal, scored as a failure by a brittle grader. The prompt did change behaviour: System B always used the exact phrase, which is what you want when downstream code parses the refusal. But the 0-of-2 score for System A was my grader's error, and I only caught it because I read the raw answers. Nobody reads raw answers at 500 cases, which is exactly why graders that check meaning beat graders that check exact strings.
On judge stability, the news is better than I expected. I re-ran the same faithfulness judge on the first 4 questions, 8 answers, and it agreed with itself 8/8 times. Stable, but thin evidence: 8 samples, all initially judged faithful. Stability is also not accuracy. The same judge that agreed with itself 8 out of 8 also made the two false-negative errors above. A stable judge can be stably wrong, so re-running is necessary but not sufficient. You still need a human to spot-check a sample.
Finally, the cost. The whole experiment, 24 generations plus 32 judge calls, used 13,217 tokens and cost about $0.0015 at Gemini 2.5 Flash-Lite list pricing ($0.10 per million input tokens, $0.40 per million output tokens). Median generation latency was about 1.1 seconds for both systems. System A's mean latency looks worse at 3.68 seconds against 1.19 for System B, but that mean is pulled up by a single 31-second answer on an unanswerable question. One outlier moved the mean by seconds, which is another small lesson: report medians for latency, or one weird call will rewrite your conclusions. A 500-case judge run at these rates costs roughly $0.09. Evals are not expensive. Building the golden set is the expensive part, and that cost is mostly your time, not tokens.
What the numbers mean
Five lessons, in order of how much they will save you.
1. Unanswerable questions are the highest-value test cases you own, and your grader for them is probably broken. Answerable questions mostly confirm that your system works. Unanswerable questions are the only ones that catch the failure users actually complain about: confident invention. Put at least 10 to 20 percent unanswerable cases in every golden set. And grade refusal by meaning, not by exact string. My grader counted only "I don't know" and scored two correct refusals as failures. If your system never says "I don't know" in testing, check whether it refuses in other words before you conclude it never refuses.
2. Measure retrieval and generation separately. Ragas splits them for a reason. A single end-to-end score hides which half broke. When my recall@3 came back at 1.00, I knew the generator owned the remaining errors, and the prompt fix was the right lever. Without that split, I would have been guessing.
3. A one-sentence prompt change is a measurable change. "Answer only from the context, say I don't know" lifted token F1 from 0.643 to 0.791 and made every refusal use the exact phrase, which downstream code can parse reliably. Prompt changes are the fastest iteration loop you have, and evals are what make the loop honest. Without a score, a prompt tweak is a vibe. With a score, it is an experiment.
4. Your judge needs its own eval. Judge self-agreement of 8/8 out of 8 on re-run looks reassuring until you notice the same judge wrongly flagged two correct answers as unfaithful. Stability is not accuracy. Calibrate the judge the same way you calibrate anything else: grade a sample by hand, measure agreement, fix the rubric, and re-run. A judge that agrees with humans 80 percent of the time is a fine regression detector. A judge you never checked is a random number generator with good branding.
5. Small test sets lie by omission. Twelve questions is a demo, not a verdict. My recall@3 of 1.00 could be luck: with 10 answerable questions, one miss moves the score by 10 points. Anthropic's 20 to 50 task starting point exists because below that, the confidence intervals are wider than the differences you are trying to detect. Build the small set, use it daily, and grow it from every real failure your users report.
The tooling landscape in 2026
You do not need a platform to start. You need a script, a golden set, and a habit. But when the script outgrows itself, here is what the field looks like now.
Ragas is the open-source RAG eval framework, Apache 2.0, free to run locally. You pay only for the judge and embedding API calls. It is the default choice when your system is RAG-shaped, and the four metrics above are its vocabulary. promptfoo is an open-source CLI for YAML-driven prompt and model evals, with red-teaming built in. Its strength is comparing many prompts and models in one run, which is exactly the experiment I ran by hand above. DeepEval is a pytest-style Python framework, open source, with a hosted cloud called Confident AI on top: free tier at $0, Starter at $200 a month. It fits teams that already think in tests and want evals inside their CI pipeline.
LangSmith is LangChain's tracing and eval platform. The Developer tier is $0 per seat with 5,000 traces a month, and Plus is $39 per seat. If you already trace with LangSmith, its evals are the shortest path from "we saw a bad trace" to "we have a test for it". Braintrust is experiment-first: Starter is $0 a month with 10,000 free scores, Pro is $249 a month. It shines when you run the same eval across many model and prompt variants and want to compare them side by side.
My take: start with a 30-line script like the one behind this article, not a platform. Platforms solve the problems you get at 500 test cases and three teammates. A script solves the problem you have today, which is that you have no evals at all. When the script hurts, you will know exactly which platform feature you need, because you will have felt the absence of it.
Build your first eval set this week
Here is the smallest version that works, in five steps.
Step 1: collect 20 real failures. Read your logs, your support inbox, or your own testing notes. Copy the exact inputs that produced bad outputs. Do not write synthetic questions yet. Real failures carry the distribution your users actually hit.
Step 2: write the reference answers. For each input, write what a good answer looks like, in one or two sentences, and note which document it comes from. Add 2 to 4 unanswerable questions, things your system should refuse or deflect. This is the slowest step and the most valuable one.
Step 3: pick two or three metrics, not ten. For a RAG system: context recall for retrieval, faithfulness for the generator, and abstention rate on unanswerable cases. For an agent: task success rate plus one cost metric, like tokens per task. Three metrics you look at beat ten metrics you ignore.
Step 4: set thresholds from your baseline, not from a blog post. Run the eval on your current system. That score is your baseline. A sensible starting bar for RAG is faithfulness above 0.80 and context recall above 0.80, but your corpus decides what is realistic. The threshold that matters is "better than last week", not a magic number.
Step 5: run it on every change. Prompt tweak, model swap, chunk size change, new tool: run the eval first, then decide. The moment evals become a gate instead of a chore, they start paying for themselves, because every regression gets caught by a test instead of by a user.
What evals cannot do
Three honest limits, so you do not over-trust the thing I just spent an article praising.
First, benchmarks saturate. MMLU, the general knowledge benchmark, had top models above 90 percent by 2025, and HumanEval pass rates sit in the same range. When every model scores above 90, the benchmark no longer separates good from great. It still separates broken from working, which is useful, but it cannot tell you which model to pick. Your own evals on your own tasks are the only benchmark that stays relevant, because your tasks are the ones your users care about.
Second, Goodhart's law applies. Once a metric becomes the target, it stops being a good measure. Teams that optimise for faithfulness alone get answers that quote the context verbatim and never synthesise. Teams that optimise for brevity get answers that omit the caveats. Keep at least one metric that nobody is allowed to optimise directly, usually a periodic human review, and watch for the system gaming the score.
Third, evals measure what you thought to measure. OpenAI's sycophancy incident is the proof. Their suite was large and their metrics were green, and the model still shipped broken, because nobody had written an eval for the exact behaviour that changed. The defence is not a bigger suite. It is a habit: every time a user reports a failure your evals missed, add that failure to the golden set before you fix the bug. The suite grows in the direction of reality instead of the direction of your imagination.
What is Next in This Series
Part 6 builds your first AI agent with LangGraph: a stateful agent that plans, calls tools, checks its own work, and recovers from errors. The evals from this part are what will keep that agent honest, because an agent that loops is an agent that can loop wrong, and you will want a test that catches it. Until then, agents that run code or call tools belong in a sandbox, not on your laptop with full permissions. After that, Part 7 connects agents to real tools with MCP, and Part 8 ships the whole thing to production with LLMOps.
If you take one thing from this part, take the habit, not the tools. Write down 20 real failures. Grade them automatically. Run the grading on every change. That habit is the skill 56 percent of job postings are asking for, and it is the difference between an AI engineer who ships demos and one who ships systems people trust.
Originally published on LogicStack. Run the code examples right in your browser with our free interactive Python and SQL playgrounds — no setup, no sign-up.
Top comments (0)