A passing agent run is not a reliable evaluation

A single successful run is not strong evidence that an AI agent is reliable.

If the agent took forty steps, made three unsafe tool calls, needed an unrecorded human correction, and happened to land on the right final answer, the outcome hides the part you actually need to improve.

For agent systems, evaluation has to cover both the result and the path that produced it.

Start with a frozen task packet

A reproducible evaluation begins by pinning the case:

• the exact source snapshot
• the instruction and acceptance criteria
• the tools and permissions available
• the model, prompt, runtime and policy versions
• the expected output and permitted side effects

Without that packet, two runs may look comparable while operating on different evidence or under different authority.

Score more than the final answer

I find it useful to separate at least four dimensions.

1. Outcome

Did the run produce the expected artifact? Did deterministic checks pass? Was the answer complete and factually grounded?

2. Trajectory

Which sources did the agent read? Which tools did it call? How many retries or dead ends occurred? Did it stay inside its declared scope?

3. Intervention

Did a person change the prompt, repair a file, approve a different action or quietly supply missing context? Human help is not a failure, but hidden help makes the score misleading.

4. Recovery

What happened when a tool failed, context was stale or an action only partially completed? A dependable agent should fail closed, surface the problem and resume from evidence rather than improvising around it.

A composite score can be useful for ranking, but it should not erase these individual signals. The diagnosis matters more than the leaderboard number.

Verify effects, not only text

For tool-using agents, the final message is only one output.

An evaluation should also inspect durable effects: files written, records changed, messages prepared, external actions attempted and approvals consumed. A plausible summary cannot prove that the underlying action was correct or idempotent.

This is especially important for retries. If the first callback succeeds but the acknowledgement is lost, a naive retry may duplicate the side effect. The evaluation needs to test the recovery path, not only the happy path.

Keep an exact run receipt

Each run should return enough evidence to compare it later:

• task-packet identifier and source hashes
• model, prompt, tool and policy versions
• tool-call sequence and permission decisions
• human interventions and approvals
• deterministic test results
• produced artifact hashes
• cost, latency and retry counts
• final status and recovery state

That receipt makes regressions explainable. When a model or prompt changes, you can rerun the same cases and see whether quality improved by taking a safer path—or merely by spending more tokens.

The practical loop

A useful evaluation loop is simple:

1. Freeze a representative case.
2. Run it inside explicit tool and permission boundaries.
3. Record the complete trajectory.
4. Verify the artifact and every durable effect.
5. Compare the result with previous versions.
6. Promote a change only when the evidence supports it.

The goal is not to make agents deterministic. It is to make the evaluation repeatable enough that change becomes measurable.

That is the difference between “it worked in a demo” and evidence you can use to operate a real system.

I wrote a longer practical guide with a concrete evaluation packet, failure cases and promotion criteria on ChaseOS:

Reproducible model evaluations for AI agents

You can also explore ChaseOS, a local-first Windows workspace for governed agent work.