Right Answer, Wrong Path: Two Ways to Catch AI Agent Regressions

Disclosure: I'm the author of AgentSnap, one of the two tools compared here. I've tried to describe both tools by what they did when I ran them, and the maintainer of the other tool, Raju Dandigam, reviewed the agent-inspect-specific facts and suggested clarifications that I've incorporated. The conclusions are mine.

Output-only tests have a blind spot. An agent can take the wrong path and still hand back the right answer, and a test that only looks at the final text will pass it.

agent-inspect's broken-agent-debugging starter fixture is built around exactly that case, so I used it to compare two different approaches to catching it:

• agent-inspect (npm · GitHub; TypeScript): in the workflow I tested, you declare what must be true of a run, and it checks a single trace against those invariants. (agent-inspect also has diff and baseline-comparison facilities, which I did not exercise here.)
• AgentSnap (Python, PyPI): you record a golden run, and it diffs later runs against it.

Both fail the regression run. They fail for different reasons, and neither directly detects the fixture's headline defect.

The fixture

The fixture has two runs that return the same final answer:

Per policy P-204: refunds are available within 30 days of purchase.

• Good run: plan → retrieve_policy → rank → generate → the observed outcome passes.
• Regression run: the model generates before retrieving, retrieve_policy is called twice, a forbidden search_docs tool is called (and errors), and the observed outcome fails.

The fixture is agent-inspect's broken-agent-debugging starter. It's keyless and deterministic.

The two runs. Plan and rank steps are omitted from the good run for clarity.

What I ran

One caveat: AgentSnap has no native adapter for agent-inspect traces. I rebuilt the event sequence by hand, so treat the AgentSnap side as "AgentSnap given an equivalent trace," not as an official integration.

Both tools passed the good run and failed the regression run:

• agent-inspect failed with three named rules: outcome.status, tool.usage, circuit.same-tool-repetition.
• AgentSnap failed with failed_checks: ["structural", "model_tools", "model_tool_args"].

The matching verdicts look like agreement. They mostly aren't. The two tools were reacting to different parts of the trace.

The core difference

A note on scope: this compares the specific workflows and versions I tested, not everything either tool can do. agent-inspect also has diff and baseline-comparison facilities that this experiment didn't use, so the invariant-versus-golden distinction below describes the approaches as exercised here.

As tested, agent-inspect evaluates declarative invariants against one trace. You write down what should be true. It needs no history, works on the very first run, and fails with a named rule id. It stays right even if every run you've ever recorded was wrong.

As tested, AgentSnap diffs a new run against a recorded golden. You record what happened. It needs no specification, catches things nobody thought to assert, and fails with a label for the layer that changed. It can't be wrong about what changed, but it has no opinion about what should have happened.

What each tool compared a run against in this experiment: a written contract, or a recorded golden run.

Neither approach, as exercised here, is a superset of the other:

• Invariants catch what is wrong without needing a baseline. A day-one bug fails immediately.
• Diffs catch what changed without needing a spec. An unanticipated regression fails for free.

The practical consequence: AgentSnap's AgentAsserter records the golden on first use, so it can't fail the first run. A bug present on day one gets silently canonized as correct. Conversely, a regression in a dimension nobody wrote a rule for sails past agent-inspect and trips AgentSnap's diff automatically.

The data models differ to match. agent-inspect emits an execution tree (RUN / LOGIC / TOOL / LLM / OUTCOME), each step with a start row and a completion row carrying duration and status. AgentSnap emits a flat list with two event types, tool_call and llm_call, each a single point with no status field.

Where the concepts line up (and where they don't)

A note on the first result above: AgentSnap's model_tools failure fired because the model requested search_docs in tool_requests, and that requested sequence differed from the golden's. The executed search_docs never appeared in the trace, because it raised. This compares model requests across runs. It does not check requested tools against executed tools within one run, so a mismatch between the two would surface only if the requested or executed sequence also differed from the golden.

A note on rigor: the version-specific behavior in the table comes from the runs and source I inspected. Several rows describe constructed scenarios (inserting an intermediate tool, five baselined retries, P-204 vs P-205) that I reasoned through from how each check works, not separate runs. Treat those as "by construction."

What both taxonomies missed

Cross-kind ordering, including the fixture's own headline defect. Neither tool could express "the LLM step must come after the tool step," and I verified this on both sides:

• agent-inspect: a contract with requiredOrder: ["retrieve_policy", "generate_answer"] failed identically on the good and regression runs, both with Required tool generate_answer did not appear. The rule never evaluates either way, because generate_answer is emitted as kind: LLM and tool rules match only kind: TOOL.
• AgentSnap: taking the good trace and swapping only those two events, so the LLM call comes first, returned structural_diff = None and passed: True. The structural diff projects through tool calls only and discards LLM position entirely.

Why swapping the LLM step changes nothing: both checks look only at tool steps.

So "generate before retrieve", the regression the fixture is built around, was caught by agent-inspect only incidentally, through the other three violations. A trace with that defect and nothing else wrong would have escaped both tools.

Data flow and grounding. Neither tool can express "the P-204 in the answer came from retrieve_policy's result." agent-inspect has a result-dependency field, but it's scoped to retry oracles rather than being a general provenance relation. AgentSnap compares response strings and never links them to any tool result. An agent that retrieves correctly and then states a different policy number can pass ordering and tool-sequence checks: the tool sequence is perfect and the output is one token off. An explicitly implemented outcome assertion could catch the wrong number, but someone has to write it.

Two issues I noticed in agent-inspect (at 6.29.4)

I shared these with the maintainer while testing. Neither is about the taxonomy question.

1. circuit.same-tool-repetition counted raw events, not logical calls. The regression run reported expected: 3, actual: 4 for two retrieve_policy calls. Synthetic probes confirmed the unit was exactly 2 × calls (2 calls reported 4, 3 calls reported 6). The cause matched a note in the project's KNOWN-ISSUES.md: rules reading raw context.events see both the start and completion rows rather than using the logicalEvents projection. The 2× counting was measured; the cause was inferred from that note, since the analyzer ships bundled.
2. The requiredOrder LLM exclusion was real but undocumented, and the error misreported it. Putting an LLM step in tools.requiredOrder produced a "did not appear" error that never mentioned the step exists as kind: LLM. The rule that this is deliberate lived only in a source comment in the fixture's prove-same-output-wrong-path.mjs, while the contract reference's own worked example used requiredOrder: ["retrieve", "generate"].

Maintainer updates (after my tests)

Everything above reflects the versions I tested. I haven't retested the following. Raju shared these releases afterward:

• 6.29.5: logical-event circuit counting, selected-run scoping, and extension-rule accounting.
• 6.29.6: clarification of TOOL-only requiredOrder.
• 6.31.0: additive, experimental typed TOOL/LLM ordering through steps.orderRelations. See the contract reference for the explicit opt-in.

Two qualifications from the maintainer are worth repeating. Upgrading alone does not add the missing relation to the original gate: you have to request "TOOL retrieve_policy before LLM generate_answer" explicitly. And an ordering assertion does not prove that an answer is semantically grounded in the retrieved result.

What's still open

Neither tool settles every question this fixture raises, and the maintainer updates above don't either. Three gaps remain in both styles of testing:

• Grounding. An ordering assertion can say the model answered after retrieval. It can't say the answer actually used what was retrieved. An agent that retrieves correctly and then states a different policy number can pass a tool-order check and a tool-sequence diff. Catching it takes an explicitly implemented outcome assertion.
• Whose intent is being tested. AgentSnap diffs the tools the model requested across runs, and separately diffs the tools that executed. It doesn't check requested against executed within a single run. In this fixture agent-inspect recorded what executed, though some adapters also record action intent. Neither tells you whether the request itself was the right one.
• Where the baseline or the rules come from. Invariants are only as good as the person who wrote them. A golden run is only as good as the day it was recorded. In both cases the check starts from someone's judgment, and that judgment is the part that isn't tested.

I don't think these point to a winner. They point to questions worth asking of any regression gate for agents: what does it compare a run against, which relationships between steps can it express, and what would a wrong-but-plausible run look like to it?

---

Thanks to Raju Dandigam, agent-inspect's maintainer, for the fixture, for the release references, and for reviewing the agent-inspect-specific facts. Any errors that remain are mine.

Comments

[alex]

Output-only tests give false confidence, an agent that skips a verification step or calls the wrong tool can still return the right answer. I'm curious how you handle legitimate path variation, where the agent reaches the same result by a different but valid route.