One Passing Agent Run Is Not a Release Signal

I can make almost any agent change look good with one carefully chosen prompt.

Run it once. Watch the right tool fire. Read a polished answer. Record the demo.

Ship it?

Not yet.

A single trace is excellent evidence for one execution: what ran, what failed, how long it took, and whether declared invariants held. A release decision asks a different question:

Does this change behave acceptably across the representative cases we care about?

That shift—from run to set—is where agent testing starts to resemble engineering instead of demonstration.

Give each evidence layer one job

In agent-inspect@6.17.6, I use three local, deterministic layers:

Layer	Question
Suite	Did each named case satisfy its expected checks?
Cohort	What moved between baseline and candidate groups?
Gate	Should CI accept the recorded evidence?

representative traces
        |
        v
      suite --------> named case results
        |
        v
baseline vs candidate -> cohort deltas
        |
        v
       gate --------> CI exit code + evidence

These commands read persisted traces. They do not rerun the agent, invoke a model, or decide whether prose is semantically good.

Start with named cases, not random traffic

A compact suite makes the review set explicit:

{
  "name": "refund-release",
  "traces": "./.agent-inspect",
  "cases": [
    {
      "id": "eligible-order",
      "runId": "refund-eligible",
      "requireTools": ["lookup_order", "refund_order"]
    },
    {
      "id": "unknown-order",
      "runId": "refund-unknown",
      "forbidTools": ["refund_order"]
    },
    {
      "id": "approval-required",
      "runId": "refund-approval",
      "requireTools": ["request_approval"]
    }
  ]
}

Validate the configuration and run it:

npx agent-inspect suite validate \
  --config agent-inspect.suite.json

npx agent-inspect suite run \
  --config agent-inspect.suite.json \
  --markdown

The suite distinguishes a violated expectation from missing evidence. If refund-unknown does not exist, the case is skipped with a diagnostic; it is not silently treated as a pass. An all-skipped suite cannot become a green release signal.

That distinction sounds small until CI says “passed” for a case that never ran.

The set is part of the test

The most important suite question is not “Did it pass?” It is “Why are these the representative cases?”

For a refund agent, I would begin with:

1. an ordinary eligible order;
2. an unknown order that must not trigger a refund;
3. an amount requiring approval;
4. a missing-data path;
5. a retryable dependency failure; and
6. a non-retryable tool failure.

Version the suite beside the application. When product behavior intentionally changes, update the evidence set and expectations in the same review.

A useful review matrix keeps scenario coverage and evidence type visible:

Scenario	Structural check	Outcome check	Semantic check
Eligible order	lookup before refund	refund record exists	explanation is accurate
Unknown order	refund tool forbidden	no mutation occurred	refusal is useful
Approval required	approval tool present	request queued	message explains next step

No single column can substitute for the other two. A structurally correct run can still answer badly; a fluent answer can still hide an unauthorized mutation.

This is also where “prove the suite can say no” becomes useful. Keep at least one known-bad trace or mutation that must fail. A check that has never rejected evidence and a check that silently stopped running can produce the same green icon.

Compare behavior as a cohort

For a prompt, model, tool-schema, or orchestration change, label captured runs:

import { inspectRun } from "agent-inspect";

await inspectRun("refund-agent", runScenario, {
  traceDir: ".agent-inspect",
  metadata: {
    cohort: "candidate",
    scenario: "eligible-order",
    model: "approved-model",
  },
});

Then compare baseline and candidate traces:

npx agent-inspect cohort \
  --dir .agent-inspect \
  --baseline before \
  --candidate candidate \
  --cohort-key cohort \
  --group-by model \
  --metric errorRate,duration,toolChoice,observationFailure

A cohort report preserves the run count and the actual aggregate values. Imagine this synthetic result:

before / approved-model (10 runs)
  error rate: 0%
  average duration: 820ms
  dominant tool: lookup_order

candidate / approved-model (10 runs)
  error rate: 10%
  average duration: 970ms
  dominant tool: refund_order

That is a reason to investigate, not a statistically universal conclusion. The CLI can describe the recorded groups; it cannot make a small, biased fixture set representative.

A positive delta is not automatically bad

Suppose the candidate makes one extra tool call. That might be waste—or a newly required authorization check.

Suppose duration improves by 20%. That might be a better path—or the candidate skipped retrieval.

Suppose the dominant tool changed. That might be a regression—or the exact migration you intended.

Structural metrics describe what changed. Product assertions and semantic evaluation decide whether the change is acceptable. Keep those judgments separate so the report does not pretend to know your domain.

Turn credible evidence into a fail-closed gate

Once cases and thresholds reflect actual release policy, add a gate:

npx agent-inspect gate \
  --suite agent-inspect.suite.json \
  --format github \
  --output ./agent-gate-artifacts \
  --evidence-on fail \
  --evidence-profile share

Or gate a directory directly:

npx agent-inspect gate \
  --dir .agent-inspect \
  --max-error-rate 5 \
  --forbid-tool delete_account

Gate exit codes distinguish a policy failure from invalid configuration, unreadable traces, and unsupported output formats. Evidence generation does not suppress the original gate result.

That behavior matters because agent tooling is especially vulnerable to accidental fail-open configurations. Version 6.17.5 strengthened empty-check handling, rule-execution evidence, observed-outcome requirements, and requiredOrder semantics; 6.17.6 preserves those behaviors while hardening packaging and ingest boundaries.

What a green gate actually means

A passing gate means the encoded checks passed for the inspected evidence set.

It does not prove:

• the cases represent production traffic;
• the final answers are correct;
• the model will repeat the behavior;
• the thresholds are good; or
• the agent is safe in production.

That is not a weakness unique to agent tests. It is the normal boundary of evidence-based release decisions. The remedy is not one giant magical score; it is several checks with clear jobs.

My smallest credible release loop

capture named cases
      -> run deterministic suite
      -> compare baseline/candidate cohorts
      -> inspect semantic quality separately
      -> make CI fail closed
      -> preserve the evidence

The pinned suites, cohorts, and gates documentation labels these surfaces Beta, and the cohort recipe includes a deliberately regressing fixture.

One good run can prove that a path exists. It cannot prove that a release is ready.

What is the smallest set of agent cases you would require before approving a change?