Regression-testing an agent whose output is never the same twice

You change one line of a prompt and have no idea what you broke, because diff is useless on free text. The fix is to stop comparing text: freeze a set of real traces, assert on behaviour — which tools got called, which constraints held, what the agent refused to do — and run that suite on every prompt or model change.

Concretely, three files and a command. A cases.jsonl of frozen inputs, an assertions.ts that checks behaviour rather than strings, and a runner you invoke after each change. If you work in Claude Code or Cursor, this is a good candidate for a slash command, so the check costs you one line instead of a context switch.

How it works

The unit under test is not the final message. It's the trace: the ordered record of what the agent did to produce that message — tool calls and their arguments, retrieved documents, the number of turns, tokens spent, and the final output. A trace is structured data, and structured data can be asserted on.

Four kinds of assertion cover most real failures:

• Sequence — the agent called search before answer, and never called delete_record at all. Cheap, deterministic, catches the scariest class of regression.
• Constraint — the answer cites at least one retrieved document; the JSON parses against the schema; the reply stays under the token budget.
• Refusal — for inputs that should be declined, the agent declines. Prompt edits aimed at making an agent more helpful loosen refusals as a side effect, every time.
• Judged — a second model scores the output against written criteria ("does it answer the question actually asked?"). Use it last and sparingly; it's the only assertion that can itself be wrong.

{ "id": "refund-past-window",
      "input": "I want a refund, bought 400 days ago",
      "assert": {
        "tools_called":     ["lookup_order", "check_policy"],
        "tools_forbidden":  ["issue_refund"],
        "must_mention":     ["policy", "30 days"],
        "max_output_tokens": 300
      } }

Run the suite twice per change: once before, once after. What you report is not pass/fail but movement — three cases that used to call check_policy no longer do. That's the sentence you want in front of you before a deploy, and it's the one a screenshot of a happy-path chat will never give you.

Keep the case set small and adversarial. Twenty cases you chose because each one broke something once beats two hundred generated to look thorough — the suite has to run in under a minute or you will stop running it.

Where it pays, and where it doesn't

Worth doing

Prompts change weekly, models get swapped, the agent can take actions with consequences, or more than one person edits the prompt. Anywhere a silent behaviour change costs more than the hour this takes to set up.

Not worth it

A single-turn wrapper with no tools and no side effects — assert the JSON schema and move on. Also skip it while you're still changing the agent's job description weekly: you'd be freezing traces of a thing that doesn't exist yet.

The honest downsides. Behavioural assertions encode your current idea of correct, so a deliberate improvement shows up as a failure and you rewrite cases — budget for that. Judged assertions add cost and their own flakiness, and a judge model that gets upgraded under you is itself an untested change. And a frozen trace set drifts away from real traffic; refresh it from production logs every so often, or it slowly starts testing last quarter's product.

The alternative is not "no testing" — it's testing by user report, on a Tuesday, in public.

I package this workflow — eval suites, trace review, regression checks, red-teaming — as slash commands and sub-agents for Claude Code and Cursor. It's my own product, $29 one-time: AI Agent QA & Eval Toolkit. The approach above works fine hand-rolled, too.