Your agent eval tests whether it succeeds. It should test whether it recovers.

Most agent evaluations I've read measure one thing: given a task, did the agent complete it. That's the happy path. It's necessary and it's not enough, because in production the interesting question is almost never "does it work when everything goes right." It's "what does it do when the API returns a 500, the tool gives a plausible-but-wrong answer, or the user contradicts themselves halfway through." Those are the moments that generate incidents, and they are exactly the moments most eval suites don't cover.

I want to argue that recovery is a separate, testable axis, and that if you only report task success you are grading on the easy half.

Why the happy path over-reports

A single success rate collapses two very different agents into the same number. Agent A completes the task by getting everything right the first time. Agent B completes the same task after a tool errors, it notices, retries with corrected arguments, and recovers. Same score. In production these are not the same agent: A will fall over the first time reality deviates from the eval, and B won't.

There's a subtler version. Agent B "recovers" by silently ignoring the tool error and hallucinating a result that happens to be right for the eval case. Same score again, but now B is actively dangerous, because it treats failures as things to paper over. You cannot tell these three apart from a success rate. You have to instrument the failure itself.

(This is also why I've come around on reliability-across-trials metrics. Benchmarks like tau-bench (arxiv.org/abs/2406.12045) report a pass^k style number, the probability an agent succeeds on the same task across k independent runs, not just once. That's a recovery-adjacent idea: it punishes the agent that gets lucky once and can't do it twice.)

The cases I'd actually test

Not exhaustive, and the exact set depends on your tools, but this is the skeleton I now add to every agent eval:

1. Tool returns an error (the honest failure). Inject a 500 or a timeout on a tool the task needs. The question isn't whether the agent finishes. It's whether it retries sensibly, escalates, or gives up cleanly, versus fabricating a result. Grade the behavior, not just the outcome.

2. Tool returns wrong-but-plausible data (the quiet failure). Have a lookup return stale or subtly incorrect data. Does the agent take it at face value? Some tasks are unrecoverable here (the agent has no way to know), and that's fine to record. What you're looking for is whether it cross-checks when it could have.

3. Ambiguous or contradictory user input. The user says "cancel my order" then two turns later "actually keep the blue one." Does the agent track the contradiction or act on the stale instruction? (This one catches a lot of context-management bugs.)

4. Missing precondition mid-task. The agent needs an auth token or a field that isn't there. Does it ask, or does it invent a value to keep going? Inventing-to-keep-going is the failure mode I see most and the one a happy-path eval never surfaces.

5. Recovery cost, not just recovery. When the agent does recover, how many extra steps did it take? An agent that recovers in 3 extra tool calls is fine. One that thrashes for 20 is going to blow your latency and cost budgets even though it "succeeded."

Instrumenting it

The mechanical part is a fault-injection wrapper around your tools plus a grader that scores the trajectory, not just the final answer. This is pseudocode, not a runnable snippet: the trajectory methods stand in for whatever your framework actually exposes.

def flaky_tool(real_tool, fault):
    def wrapped(*args, **kwargs):
        if fault == "error":
            raise ToolError("503 upstream unavailable")
        if fault == "stale":
            return real_tool(*args, **kwargs, _inject="stale")
        return real_tool(*args, **kwargs)
    return wrapped

def score_recovery(trajectory, fault):
    steps = trajectory.tool_calls
    # did it notice the failure at all?
    acknowledged = any(s.reasoning_mentions_error for s in steps)
    # did it fabricate through the failure?
    fabricated = trajectory.final_answer_asserts_unverifiable_fact()
    recovered  = trajectory.task_completed and not fabricated
    extra_steps = len(steps) - trajectory.baseline_step_count
    return {"acknowledged": acknowledged, "fabricated": fabricated,
            "recovered": recovered, "extra_steps": extra_steps}

The fabricated check is the one worth investing in. It's the difference between an agent that recovered and an agent that got away with it. In practice I approximate it with a check on whether the final answer asserts something no successful tool call could have grounded, which is imperfect but catches the obvious cases.

Where I'd push back on this

The strongest objection: this is a lot of eval surface for teams that can barely maintain a happy-path suite, and a half-built recovery suite can give false confidence worse than none. That's fair. If you only have budget for one recovery case, make it case 4 (missing precondition), because "invents a value to keep going" is the failure that most reliably becomes a production incident.

Second objection I'll concede partly: some of this overlaps with guardrails and monitoring, and you might argue recovery belongs in prod observability, not offline eval. I think you need both, but if your prod tracing already catches fabrication-through-failure, then yes, you can lean on that and keep the offline suite thin. The thing I won't concede is reporting a bare success rate and calling the agent evaluated. A bare success rate only tells you the agent works when nothing goes wrong, and "works when nothing goes wrong" is not the same as evaluated.

What I'm still unsure about: how to weight recovery against first-try success in a single headline metric, or whether they should stay two numbers. Right now I keep them separate and I'm not convinced that's right. If you've found a defensible way to combine them, I'd like to see the formula.