What a Passing Eval Still Doesn’t Tell You

Before giving an agent access to a production system, ask what its passing eval actually authorizes. Opening a pull request, merging it, and running a database migration can require very different evidence even when they use the same model and underlying system.

An eval establishes something specific about a system under the conditions that were tested. Problems arise when the result is treated as evidence for behavior outside those conditions. An agent may pass an evaluation and then be given authority to take actions whose relevant failure modes were never part of the test.

Consider a coding agent asked to migrate records into a new database table. In its evaluation environment, the migration produces the expected rows and the checks pass. Now suppose the deployed workflow loses its connection immediately after the database commits but before the agent receives confirmation. The agent observes a failed tool call even though the database has already changed.

If the workflow retries by inserting the records again, it may duplicate data. If it assumes the operation succeeded and continues, the same behavior could skip necessary work when a similar connection failure happens before the commit. Successful uninterrupted runs do not tell us how the system behaves when the result of an external operation is uncertain.

This is a familiar distributed-systems problem. Adding a model to the workflow introduces another decision-maker, but it does not remove the need to establish what happened in the external system. A better prompt cannot supply information that the system itself does not have.

The original evaluation is still useful because it showed that the migration produced the expected state in the tested scenario. However, it did not establish that the workflow could safely recover when execution was interrupted around the commit boundary, so it should not be used as evidence for granting the agent that level of authority.

Start with the action you are about to permit

“Can we trust this agent?” is too underspecified to be a useful evaluation question. The more useful question is whether this configuration should be allowed to perform this operation with these permissions.

For an agent limited to proposing a patch, repository tests and human review may provide the relevant gate. Giving the same agent production write access changes which failures need to be tested because the consequences of getting them wrong are different.

In the migration example, deliberately interrupt the workflow around the commit boundary in a disposable environment and observe what happens during recovery. The system should be able to determine whether the write occurred, retry safely where appropriate, or represent the result as uncertain until it can resolve the external state. A normal successful run does not exercise any of those behaviors.

Check what produced the verdict

A tool returning “success” is only useful if that response corresponds to the state you care about. Inspect the resulting data and verify that the checks ran against the intended version and actually exercised the assertions you expected.

The same applies to an agent’s final summary. A statement such as “migration complete” should correspond to an independently inspectable result rather than serving as the evidence itself.

Also consider who controls the evidence used by the gate. If the same process proposing a consequential change can modify the record used to show that the change passed, the approval mechanism depends on evidence the acting process can rewrite. For higher-consequence actions, approval should depend on evidence outside that process.

Look behind the aggregate

Aggregate scores are useful for comparing systems, but they can hide structure in the underlying results. A system may perform well overall while consistently failing on one workflow that matters, and a single successful run provides little information about consistency.

Representative tasks should be run repeatedly, with individual outcomes retained and comparisons made under equivalent task versions and execution conditions. Repetition helps characterize variability within the scenarios being tested, but it does not compensate for missing scenarios.

Running the uninterrupted migration a hundred times may give good evidence about repeatability under normal execution, but it still gives no evidence about recovery after a lost acknowledgement. More runs only help if they exercise behavior relevant to the claim you are trying to support.

Make the authority boundary real

The deployed identity should have only the permissions needed for the work the agent is expected to perform, and the evaluation should include attempts to perform actions outside that boundary.

An instruction telling an agent not to modify backups provides limited assurance if the same credentials allow it to delete both the live data and the recovery copy. The permission boundary is part of the system being evaluated, not just an implementation detail around the model.

The controls should be proportional to the consequences of the action. A disposable prototype and a production migration do not need the same release gate, but in either case the evidence should match the authority being granted.

Before expanding an agent’s authority, make sure you understand the claim its evaluation supports and the consequential behavior that remains untested, and include that gap in the release decision. A passing eval is useful only to the extent that its evidence matches what the system is actually being allowed to do.