You changed one line of your agent's prompt. What else broke?

You tweak one line of your agent's prompt to fix a complaint. The fix works. You ship it.

Two days later, a different user reports something that used to work just fine.

Nobody touched that feature. You changed the prompt, and the agent's behavior shifted everywhere at once. That's why AI agents need evals, and why the evals most teams write miss the bugs that matter.

TL;DR

• An eval is a task, a grader, and a number of trials. Not a vibe check.
• Grade what the agent did, not just what it said.
• Run every task more than once. "Works 90% of the time" fails a lot.

What an eval actually is

An eval gives the agent a task, runs it, and scores the result automatically. Here's one for a scheduling agent:

task: "Book 30 minutes with Priya next Tuesday afternoon"
expect:
  - create_event called once
  - attendee is [e-mail]
  - start time is Tuesday, between 12:00 and 17:00
  - no existing meeting was moved or deleted
trials: 5
pass: all 5 trials

Run your suite after every prompt, model, or tool change, and you'll know what broke before your users do.

Grade all three layers

Most eval suites only check the final reply. But an agent can say "Booked!" while calling the wrong tool, or call the right tool with the wrong date. Check all three: the outcome, the trajectory (tool calls and their arguments), and the state that changed.

Use the cheapest grader that works

Start with plain code. It's fast, free, and never disagrees with itself:

def grade(trace, calendar):
    events = [c for c in trace.tool_calls if c.name == "create_event"]
    assert len(events) == 1
    assert "[e-mail]" in events[0].args["attendees"]
    assert calendar.unchanged_except(events[0].result_id)

Use an LLM judge only for things code can't check, like tone or whether an answer is actually helpful. Then have a human review a sample of the judge's scores now and then, so you know the judge is grading the way you would.

Run it more than once

Agents are non-deterministic. If a task passes 90% of the time, the chance it passes five times in a row is only 59%. For anything user-facing, "it worked when I tried it" isn't good enough. Measure how often it works every time.

Where your first 20 evals come from

Don't invent test cases. Mine them:

• Real failures: every bug report and bad transcript becomes an eval.
• Messy inputs: vague dates, missing details, typos, two requests in one message.
• Happy paths: a few normal requests that must keep working, so a fix can't quietly break real users.

Twenty real cases beat two hundred made-up ones.

Keep two suites

Regression evals check things that already work. They should pass almost 100% of the time, and a failure blocks the release.

Capability evals check things the agent can't do reliably yet. A low score is expected. Watching it climb tells you whether your changes are actually helping.

The one thing to remember

An eval suite is the agent's memory of every mistake it has made. Without one, every prompt change is a guess.

How do you decide when an agent change is safe to ship? Share in the comments.

Comments

[howcani howcani]

The two-trial rule, the three layers, and "cheapest grader first" are right. The one thing I would add is a check on the suite rather than on the agent — because a suite that passes and a suite that is blind look exactly the same in the report.

Treat the prompt change as a mutation and ask whether your eval kills it. The catch is where the mutants come from. I built a small version of your scheduling task: grader asserts create_event called once, attendee is [e-mail], start is Tuesday 12:00–17:00, and nothing existing was moved or deleted. Then a second grader version representing a plausible-looking diff that had swapped that last clause for "exactly one event was created", which reads like a state check and is not one:

mutant injected	real grader	weakened grader
attendee typo'd	killed	killed
wrong day (Mon 14:00)	killed	killed
create_event called twice	killed	killed
an existing meeting moved	killed	survives

Score on the first three: 3/3 and 3/3 — the two graders are indistinguishable. Only the fourth mutant, drawn from the change you are actually afraid of instead of from the shape of the trace, separates them (4/4 vs 3/4). Both grader versions also pass "break it first, watch it go red, restore": red-first tells you the grader runs, not what it pins.

So the number I would want next to pass@5 is: I broke this on purpose in N ways and my suite noticed M. Yours visibly catches the day and the attendee; the useful question about your own suite is which of your mutants it would quietly miss. (For what it is worth, 0.9^5 = 0.590 — I checked the chart.)