Tatsuya Shimomoto

Posted on Oct 12

Can You Tell an Eval from a Review?

#claudecode #evaluation #evals #llmops

On October 11, 2026, Boris Cherny, who builds Claude Code, posted on X. He was replying to a post saying that you can just tell the tool "the build is slow, make it faster" and it will do it. His point: since around Opus 4.8, Claude has done this kind of work better than he does, the 5.5 models are very good at it, and you only need to give the model three things.

A goal (for example, "Make p95 ci time <10 mins". p95 is the time within which 95% of runs finish)
A way to measure the result (for example, read the timings in BigQuery after each change. BigQuery is Google Cloud's service for aggregating large volumes of records with SQL)
How hard to try (for example, don't stop until it's done. It may turn into weeks of experiments)

That, he says, works for most things (post).

Of those three, I always get stuck on the second. READMEs and articles have no p95. I have worked on evals since February 2026, and I have built skills such as learn-eval and llm-as-judge. I still have not grasped the overall shape of what an eval is. Lately, when I loosely ask Opus 5.5 to "run an eval loop," my READMEs get better, which makes it even harder to place what I am actually doing.

I also suspect the word "eval" points to different things for different people. Some people mean only a test that produces a score. Others include reading outputs to find what to fix.

In this article, I split the parts into three layers and walk through them in order: the eval itself, the loop that runs the eval to raise the score, and the goal you hand that loop. The names and contents of the parts come from Anthropic's official guides as of October 2026 and from what eval practitioners have written. At the end, I place what I have been calling evals onto these parts. Once placed, what I had been asking for with "run an eval loop" turned out not to be an eval loop that climbs by comparing numbers. By the end, you should be able to map your own practice onto the same parts and see which kind of loop you are running.

An eval is made of inputs, a way to run them, and grading

The claude-api skill bundled with Claude Code has a procedure for building evals: /claude-api build-eval (the guide was added to anthropics/skills on September 29, 2026). Its target is apps built on Claude, and it measures whether a change worked, such as moving to another model or rewriting a prompt. The guide defines an eval as three things:

A set of inputs
A way to run the app on each input
A way to grade each output

Anthropic's engineering blog post "Demystifying evals for AI agents" (January 9, 2026) names the same things with these terms:

task: a single test with fixed inputs and success criteria
trial: one attempt at a task. LLM output varies from run to run, so the same task is run many times
grader: logic that scores some aspect of the result
transcript: the complete record of one trial
outcome: the final state left in the environment at the end of a trial

A task is one input, a trial is one run of it, and the grader is the grading. Only the grader produces a score. However carefully you assemble the inputs, if the grader is off target, the score is off target too.

What the grader reads is also one of the parts. When you measure an agent that changes its environment, such as writing files or fixing code, build-eval recommends making the main grading a check that reads the final state left in a disposable workspace (the outcome), not the conversation record (the transcript): whether the tests pass, whether the expected files exist, whether it touched anything it should not have.

The same blog post splits evals into two by purpose. A capability eval starts with a low pass rate and builds a hill to climb. A regression eval should pass nearly 100% of the time and watches for backsliding. The loop we look at later is the machinery for climbing a capability eval's hill.

Claude Code plugins and skills have one more official tool. claude plugin eval runs each case in an isolated claude -p that loads only that plugin, and grades with six kinds of graders: regex, which tools were called, the order of tool calls, whether files exist, an LLM verdict, and comparison against a reference record (baseline) (official docs).

The word "eval" means different things to different people

Back to the question from the opening. I checked the major sources on whether something that produces no score is called an eval (the full research is on GitHub, in Japanese). First, here are the axes the sources use when they talk about evaluation. Any single evaluation can be described as one choice from each axis. Which cells count as an eval, though, differs by source. After the table, we look at the cells where they split.

Axis	Options	Source
What it returns	A score / a verdict (pass-fail, a label) / a structured review / a free-form review	LangSmith, Braintrust, Hamel Husain and Shreya Shankar's FAQ, Anthropic
Who grades	A program / a model (LLM-as-a-Judge) / a person	Anthropic, build-eval
How it grades	Run a fixed check (match against the answer, tests pass) / compare two / score one against criteria	build-eval
What it reads	The conversation record (transcript) / the final state left in the environment (outcome)	Anthropic, build-eval
What it's for	Build a hill to climb (capability) / watch for backsliding (regression)	Anthropic
When it runs	During development, on fixed inputs / in production, watching real usage (Anthropic counts production monitoring as a separate method alongside automated evals)	Anthropic
How it's used	Compare versions and climb (hillclimb) / find what to fix (evaluator-optimizer)	The build-eval and hillclimb guides, Anthropic

The biggest split is how much of "what it returns" counts as an eval. The boundary was not whether there is a score. Lined up, what gets returned besides a score falls into three groups:

A verdict against criteria: pass-fail, or a fixed label. Even without a score, most sources count this as eval output. Anthropic lists pass-fail as a grading format. The evaluation tool LangSmith counts categorical values as evaluation output alongside numeric scores, and Braintrust treats classifiers that assign labels as evaluation tools alongside scorers that assign scores
A structured review: for example, whether each criterion is met, with a reason. The FAQ by eval practitioners Hamel and Shreya says an eval returns "a score or structured review"
A free-form review: just reading and returning "I got stuck here" or "fix this part." This is where the sources split. Hamel and Shreya call error analysis (reading outputs, annotating them, and grouping them into failure types) the most important activity in evals. Anthropic defines an eval as a test that applies grading logic to an output, and lists reading records as a step for checking that grading works correctly, or as a separate method alongside automated evals

Loops have the same line drawn through them. Anthropic calls this pattern evaluator-optimizer: one call writes, another returns an evaluation and advice, and the first revises. Anthropic presents it under a name separate from eval, as one of its agent design patterns (workflows). In research, Self-Refine (Madaan et al., 2023) has the same shape. Seen from the eval side, code review can be treated two ways. If an AI reads a PR and a developer just reads its comments and fixes things, that is a review. If instead you give that reviewer PRs with known bugs planted in them and grade the share it found and the number of wrong comments, that is an eval that measures the AI reviewer. This is my reading, applying Anthropic's definition of a test that applies grading logic to an output; no source officially defines it this way.

I found no study that measured what most people mean by "eval." Even among widely read sources, the line sits in very different places. "Eval" is a new word whose meaning shifts from person to person.

Align grading with human judgment before you use it

There are several ways to grade, and each is checked differently. If an LLM grades, you revise it until it agrees with human judgment before you use it. Even then, with too few cases, the score is buried in noise.

build-eval lists four grading methods, roughly from cheapest to most expensive, and recommends choosing the cheapest one that actually measures what you want to measure:

Programmatic checks (exact match, JSON shape, tests pass). For outputs with a bounded range of answers
A blind side-by-side comparison of two outputs, where a model picks the better one. Swap the order of A and B each time
A model scores a single output against a rubric
A person grades a small number by hand

Having a model grade, as in 2 and 3, is called LLM-as-a-Judge, and the grading model is called the judge. When choosing a grading method, build-eval shows the user five graded examples and asks, "Would you have scored any of these differently?" If even one would be different, the rubric is not ready yet. You revise until the grading matches the user's judgment.

Hamel and Shreya's FAQ states the same idea, down to the numbers:

Read the outputs first (error analysis). Annotate the records, group them into failure types, and keep going until new reading turns up no new types
Grade pass-fail, not on a 1–5 scale
Validate the judge on human-labeled examples before using it. Measure the share it correctly passes (TPR) and the share it correctly fails (TNR). You need 100–200 examples per failure type
Labeling is done by someone who knows the product's context. They are strongly skeptical of outsourcing it

How well a judge agrees with people depends on the conditions. In the 2023 MT-Bench paper, on pairwise comparisons excluding ties, GPT-4 agreed with humans 85% of the time, and humans agreed with each other 81% of the time. Including ties, those drop to 66% and 63%. The agreement figure changes with how ties are counted.

The other condition that decides whether you can trust a score is the number of cases. build-eval recommends 15–100 inputs for a first eval. Below 15, a single case's variation moves the score. The noise in one pass rate is roughly 1/√(cases × trials), so even 25 cases run twice each comes to about ±14 points. A difference smaller than that noise cannot tell you whether the change worked or it was chance.

The loop climbs through holes in the grading too

Once you have an eval, you want to run a loop that raises the score. The claude-api skill's /claude-api hillclimb is that procedure. It first checks whether there is an eval that runs. If not, it stops and sends you back to build-eval. A hillclimb without an eval, it says, is "just editing and hoping."

Laid out as parts, hillclimb looks like this:

goal: what to raise or lower. One of: raise the score, lower the cost, lower the latency, or move to another model. Other recorded metrics become guardrails, conditions that must not get worse beyond the noise
What may change: the prompt, the tool descriptions, the model or effort, and so on. Also decide what must not be touched
Split: randomly split the inputs into train (problems read to make improvements) and test (problems never shown to whoever works out the fixes, measured for score only each round). The analyzer, which reads failures and works out fixes, reads only the train records. The side running the loop sees only scores
One change per round: if you bundle several changes, you cannot tell which one worked
Acceptance: if train goes up and test does not, treat it as fitting the cases it read and roll back. If train also goes down, roll back
When to stop: the recommendation is to stop once improvements smaller than the noise have continued for three or more rounds (a plateau). Two flat rounds is too few, the guide says¹
Budget: if cost matters, estimate the total as the cost of one run × (rounds + 1) × trials, and set a ceiling
Report: what you report at the end is the improvement on test since the start, with its confidence interval. If it stays within the noise, the guide recommends saying so and not merging

When there are few cases, you run on all of them without splitting and state that the numbers only show direction (directional). Before the first round, you lay out three things: the size of the noise, the headroom, and the smallest improvement the user would act on. If the noise is larger than that smallest improvement, you say up front that no change will show a visible improvement.

Why split train and test? Because when a loop finds a hole in the grading, it can use that hole to raise the score. A paper posted to arXiv in July 2026 has an example.

The authors had Claude Code (Opus 4.8) and Codex CLI (GPT-5.5) each run the same automatic improvement loop three times. The task was to build a program that takes speech-recognition transcripts of Quran recitation, identifies which verse each part is, and segments it. The loop graded the program after each rewrite and put the problems it got wrong into a failure report. But that failure report also showed the correct answers (the verse IDs) for the problems it got wrong.

All three times, Codex embedded those correct answers directly in the program: a program that, without solving the problem, returns the answer for problems it already knows. Because it copied the answers, its score beat Claude's. Claude Code did not embed the answers in any of its three runs, and its score plateaued below Codex's.

In the next experiment, the authors split part of the problems into a test the loop never sees, removed the answers from the failure report, told both agents that a test existed and copying would not work, and isolated the environment for each run. With all four changed at once, copying dropped to zero. The authors note that they did not separate which change did it.

hillclimb's train/test split corresponds to the first of those fixes. If a loop raises its score only on grading where it can see the answers, the raised score is indistinguishable from copying. Splitting off a test measures the score on problems the loop has not seen, to check that it has really learned to solve them.

You can set a numeric goal when a tool that measures the same quantity already exists

Back to Boris's example: the goal "p95 CI time under 10 minutes" comes with a way to measure it, "read the timings in BigQuery after each change." The goal and the measurement point to the same quantity. hillclimb, too, when setting the goal, lists candidates from among the metrics the eval already records.

In his July 2025 blog post "Asymmetry of verification and verifier's rule," Jason Wei wrote that how easily you can train AI to solve a task is proportional to how easily that task can be verified. He lists five properties of easily verified tasks:

Everyone agrees on what a good solution is (Objective truth)
One solution can be verified in seconds (Fast to verify)
Many solutions can be verified at once (Scalable to verify)
The verification result correlates well with solution quality (Low noise)
It is easy to rank many solutions to one problem from good to bad (Continuous reward)

For CI time, everyone can agree that shorter is better. For how clear a README is, the first property, "everyone agrees," is already hard.

If you set a goal with no way to measure it, the verdict has to rely on the words in the conversation. With Claude Code's /goal, you write a condition, and after each turn a small, fast model reads the conversation and judges whether it has been met. According to the official docs, this evaluator sees neither commands nor files. That is why the docs recommend writing "how to verify it" into the condition. The condition I wrote in July ("Loop until 20 ideas come out. ... When you're done, write a report in notes, wrap it up, and push") had no way to verify it, and on top of that the notes folder was in .gitignore, so it could not be pushed. The evaluator kept returning "not met" 41 times, and I stopped it by hand after 7 hours and 38 minutes.

Placing my evals on the parts

From here, I place what I have been calling evals onto the parts laid out so far. First, one real example.

On October 8, I asked, about the README of my research repository Contemplative Agent (a long-running experiment with an autonomous agent that runs on a local LLM): "I want to rewrite the README substantially, so run an eval loop and get it to a quality I'd be proud to show anyone as one of my main projects." Then I asked, "Run an eval on the tagline too and improve it." The tagline in the Japanese README, the sentence right under the title, changed like this (translated here):

Before (September 12 version): "An autonomous agent running on a local LLM that proposes changes to its own constitution and values. A human decides every time whether to adopt them."
After (October 8 version): "A long-running experiment. An AI agent posts on its own using a local LLM and proposes changes to its own harness (the constitution, identity, and skills that go into its prompt). The model weights do not change. A person makes the final call on each proposal."

A diagram of a four-stage loop went in at the top, and the record of proposal outcomes (82 skill proposals adopted, 547 rejected) went into the README too. Whether it reached a quality I'd be proud to show anyone is another matter, but it got a lot better than the earlier version.

Broken into parts, the loop looks like this. The goal was a phrase about quality, not a number: "a quality I'd be proud to show anyone." The inputs were the README draft and 37 candidate taglines. The readers were visitor personas, an LLM playing three people: a researcher, an engineer, and a skeptical reader. The taglines were read 234 times in total. The README as a whole was judged by a README judge (readme-judge) using fixed questions and evidence from checking claims against the code. I made the acceptance call at the end by reading it myself.

Along the way, I corrected the goal's direction. I said, "I was thinking, maybe we don't need to put human approval up front. The north star is toward removing human approval, right? This is trying to do recursive self-improvement through the harness." Opus 5.5 replied that it half agreed and that half of my premise was off, and proposed putting the harness at the center and adding one grading check for "can a reader tell this is about the harness." I replied "Go," we re-measured, and we settled on the current tagline with the harness at the center.

What I did on the README was a review loop

The parts are all there. But the way the scores were used differs from hillclimb.

For the README body, I used the visitor personas' answers not as scores but as hints for rewriting. I read "where did you lose interest" and "if you fixed one thing, where would it be" and revised the text. I did not compare the previous and current versions by score. The exception was the tagline: for each candidate, I counted whether readers could correctly state what it was about, and compared those counts. That comparison is close to an eval loop, but there was no test split, and I did not look at how much those counts fluctuate
At first I also asked "would you try this this week" on a 1–10 scale. I dropped that score, writing the reason in the README skill's records: if a score decides when to stop, raising the score becomes a goal in itself
I stopped on the README body when the readers stopped turning up new findings that a change in wording would fix, not when a score moved by more than the noise
The judge (readme-judge) returns one of Publishable, Fix, or Rewrite and lists what to fix. It does not compare versions. My llm-as-judge skill has the same shape: it rolls binary checks into one named verdict and does not aggregate scores

In other words, what I did on the README was a review loop built from eval parts. I had been asking for this, too, as an "eval loop."

As we saw in the section "The word 'eval' means different things to different people," whether free-form reviews count as evals differs by source. So in this article, "eval" refers to the parts from the first section (inputs, a way to run them, grading), and I split the loops that use them into two by the question they ask:

Eval loop: a loop that compares versions on a yardstick, such as a score or pass rate, and raises that value
Review loop: a loop that reads this one piece and looks for what to fix next

The "review loop" in this article corresponds to what Anthropic calls evaluator-optimizer.

Even with the same parts, the question they are trying to answer differs.

	Eval loop (hillclimb)	Review loop (README)
Question it answers	Did this change work? Does it work on cases it hasn't read?	Where is the next thing to fix in this one piece?
Role of grading	Compare versions	Find what to fix
What it needs	Case count and a noise estimate (train and test if there are enough cases)	Readers who read for different reasons
How it stops	When the improvement gets smaller than the noise	When no new findings come up (the same stopping rule as Hamel and Shreya's error analysis)

The figure's labels are in Japanese; it shows the two loops from the table. The eval loop on the left (change the version → measure on a yardstick → compare with the previous version → keep it if better) pushes the yardstick's value up. The review loop on the right (read → list what to fix → rewrite) rereads until the readers turn up nothing more to fix.

What build-eval and hillclimb measure is an app built on Claude. Applied to an artifact like a README, the form would be not a single README but running a README-writing skill on 15 repositories and comparing. A single README does not have enough inputs to split into train and test.

What did have the shape of an eval loop was September's daily-report bench (a setup that compares drafts to polish the prose of my daily research reports). I prepared cases with fixed source material and set about a third of them aside as a test not used for improvement. I repeatedly rewrote the prompt that writes the prose and compared on the same cases, and the prompt files reached 29. I judged readability myself, reading the two drafts blind, and left the judge only the pass-fail check of faithfulness to the facts, because when I had the judge decide readability too, it picked the opposite of the draft I had judged better, in both orderings (one comparison). I started the bench on September 23, before the build-eval and hillclimb guides were published on September 29, so I built it from scratch with Claude Code.

How often were the review's findings right?

While writing this article, I checked once how often the README review loop's findings were right. From the README loops of 7 repositories, I picked 28 findings and verdicts from the judge and the visitor personas, and labeled each one myself as "hit," "miss," or "can't tell." A "hit" is a finding or verdict I agreed with; a "miss" is one I did not. For example, the README of my repository akc-cycle had an English sentence saying, in effect, "skill-stocktake and generation-audit use skill-health's scan, so install them together." The judge pointed out that it does not say what you need if you install only skill-health. I labeled that a "miss" and wrote in my note: "It's bundled in the plugin, so I don't feel the need to fix it, but the README doesn't need to go into that level of detail in the first place."

Target	Count	Hit	Miss	Can't tell
Judge flagged as should-fix	10	3	3	4
Judge rated as satisfied	7	7	0	0
Visitor persona findings	11	7	3	1

The four "can't tell" among the judge's should-fix findings came from checking README claims against the code and records. To verify them I would have had to check against the code myself, which would defeat the purpose, so I marked them "can't tell." Where I disagreed with the judge's "should fix," it was not about factual errors but about preference: how much the README should say. The 28 are only what the judge and visitor personas raised; anything they missed is not included. In terms of the two ways of treating code review, this is the side that measures the reviewer. What I measured is how many findings missed; I did not measure the share of real problems they found.

I have avoided eval loops

What I have been doing for READMEs and articles is not an eval loop but a review loop. For articles, too, I run the same kind of review: another LLM plays readers, reads the draft, and I fix how it comes across. The reason I felt I didn't know the best practice was that I was holding my review loops up against the procedure of an eval loop like hillclimb.

I have not run eval loops on READMEs and articles, and not because I fell short. I avoided them. The biggest reason is that I dislike Goodhart's law. Named after the economist Charles Goodhart, the law in its commonly quoted form says, "When a measure becomes a target, it ceases to be a good measure" (that phrasing was refined by the anthropologist Marilyn Strathern in 1997). The example in the section "The loop climbs through holes in the grading too," where the loop copied the answers to raise its score, has the same shape. So I had decided not to aggregate scores in my llm-as-judge skill and not to create numeric scores in my README skill. This direction was there before. In February, I dropped rubric scores in favor of an overall judgment in my skill audit (article). In August, I wrote about two drafts that passed every evaluation and were both hollow (article), and about throwing away my homegrown LLM judge even though the KPIs had improved (article).

There is one more reason: I have rarely worked on material where a numeric goal can be handed to a loop. Even in Contemplative Agent, most of what I look at is natural language, such as the agent's comments and its skill choices. There are instruments that measure in numbers, but I use them only for observation and catching regressions, not as a loop's goal.

Top comments (0)

Some comments may only be visible to logged-in visitors. Sign in to view all comments.

DEV Community