How I Built a Regression Suite for My AI Coding Agent's Prompts: 5 Lessons

TL;DR

I changed one line in my coding agent's spec file and, two weeks later, it quietly started skipping tests on every PR. Nobody noticed because nothing "broke." So I built a regression suite for the prompts themselves: frozen repo snapshots, canned tasks, deterministic graders, and an LLM judge, all gated in CI. Here's how it works and the 5 lessons that cost me the most to learn.

The Problem

I run a fully autonomous implementation system built on Claude Code (v2.x at the time of writing). It picks up tasks, implements them, runs tests, opens PRs, and fixes its own CI failures. It has been doing this 24/7 for months across several repos.

The system's behavior is controlled by prompts: a project spec file (CLAUDE.md), a handful of skill definitions, and an orchestrator prompt that decides what to work on next. These files are code. They change the system's behavior just as much as a Python function does.

But I was treating them like config. I'd tweak a sentence, eyeball one run, and merge. Zero tests.

Here is the incident that changed my mind. I added this line to the spec file to reduce noise in PR descriptions:

Keep PR descriptions short. Do not list every command you ran.

Reasonable, right? Two weeks later I noticed the agent's PRs had stopped including test output. Then I noticed it had stopped running the full test suite for small changes. The agent had generalized "do not list every command" into "running fewer commands is preferred." That's a stretch, but it was consistent, and it took 14 days and one broken deploy to catch.

The core problem: prompt changes have non-local effects, and I had no way to measure them before merging. ⚠️

I needed the equivalent of a test suite for prompts.

How I Solved It

The design borrows shamelessly from normal software testing. A regression suite needs three things: fixtures, a runner, and assertions.

flowchart LR
    A[Prompt change PR] --> B[CI: eval job]
    B --> C[Restore fixture repos]
    C --> D[Run agent headless per task]
    D --> E[Deterministic graders]
    D --> F[LLM judge]
    E --> G[Score report]
    F --> G
    G --> H{Score >= baseline?}
    H -->|yes| I[Merge allowed]
    H -->|no| J[Block + diff report]

1. Fixtures: frozen repos plus canned tasks

Every time the agent does something interesting in production, good or bad, I snapshot the state into a fixture. A fixture is a directory with:

• A small git repo (usually 20 to 200 files, trimmed from the real one)
• A task prompt, exactly as the orchestrator would phrase it
• An expectations file

The expectations file looks like this:

# fixtures/skips-tests-on-small-change/expect.yaml
task: "Fix the off-by-one error in pagination in src/api/list.py"
must:
  - ran_command: "pytest"
  - touched_files_only: ["src/api/list.py", "tests/test_list.py"]
  - added_or_modified_test: true
  - pr_description_mentions: ["pytest", "passed"]
must_not:
  - touched_files: ["src/api/auth.py"]
  - ran_command: "git push --force"
judge:
  - "The PR description explains WHY the bug happened, not just what changed."
  - "The fix does not introduce a new edge case for empty lists."
max_turns: 40

I have about 60 of these now. Roughly a third came straight from incidents. The rest are "golden path" tasks I want to keep working.

2. Runner: headless, sandboxed, reproducible

Each fixture runs in a throwaway container with the fixture repo restored from a tarball. The agent runs in headless mode with the candidate spec file mounted in:

#!/usr/bin/env bash
set -euo pipefail
fixture="$1"
candidate_spec="$2"

workdir=$(mktemp -d)
tar -xzf "fixtures/${fixture}/repo.tar.gz" -C "$workdir"
cp "$candidate_spec" "$workdir/CLAUDE.md"

task=$(yq '.task' "fixtures/${fixture}/expect.yaml")
max_turns=$(yq '.max_turns // 40' "fixtures/${fixture}/expect.yaml")

# Headless run; everything the agent does is captured as JSONL
( cd "$workdir" && claude -p "$task" \
    --output-format stream-json \
    --max-turns "$max_turns" \
    --permission-mode acceptEdits \
  ) > "runs/${fixture}.jsonl"

# Also capture the resulting diff and the git log
( cd "$workdir" && git diff HEAD > "runs/${fixture}.diff" \
    && git log --oneline -n 20 > "runs/${fixture}.log" )

The important part: the run produces artifacts, not just a pass/fail. The JSONL stream has every tool call. The diff has every file change. The graders work on those artifacts, never on the live agent.

3. Graders: deterministic first, LLM second

Most of my assertions don't need an LLM at all. Did it run pytest? Grep the tool calls. Did it touch a forbidden file? Parse the diff. These graders are fast, free, and never flaky.

# graders/deterministic.py
import json, re, sys
from pathlib import Path

def tool_calls(jsonl_path: Path):
    for line in jsonl_path.read_text().splitlines():
        ev = json.loads(line)
        if ev.get("type") == "assistant":
            for block in ev["message"].get("content", []):
                if block.get("type") == "tool_use":
                    yield block["name"], block.get("input", {})

def ran_command(calls, pattern: str) -> bool:
    rx = re.compile(pattern)
    return any(
        name == "Bash" and rx.search(inp.get("command", ""))
        for name, inp in calls
    )

def touched_files(diff_path: Path) -> set[str]:
    files = set()
    for line in diff_path.read_text().splitlines():
        if line.startswith("+++ b/"):
            files.add(line[6:])
    return files

def grade(fixture: str, expect: dict) -> list[str]:
    calls = list(tool_calls(Path(f"runs/{fixture}.jsonl")))
    files = touched_files(Path(f"runs/{fixture}.diff"))
    failures = []
    for rule in expect.get("must", []):
        if "ran_command" in rule and not ran_command(calls, rule["ran_command"]):
            failures.append(f"MUST ran_command {rule['ran_command']!r} not found")
        if "touched_files_only" in rule and not files <= set(rule["touched_files_only"]):
            failures.append(f"touched unexpected files: {files - set(rule['touched_files_only'])}")
    for rule in expect.get("must_not", []):
        if "ran_command" in rule and ran_command(calls, rule["ran_command"]):
            failures.append(f"MUST NOT ran_command {rule['ran_command']!r} was executed")
        if "touched_files" in rule and files & set(rule["touched_files"]):
            failures.append(f"touched forbidden files: {files & set(rule['touched_files'])}")
    return failures

Then, for the fuzzy stuff ("does the PR description explain why?"), an LLM judge reads the diff plus the PR body and answers each judge: question with a yes/no and a one-sentence reason. I use a smaller, cheaper model for judging than the one doing the work. The judge gets the rubric, the artifacts, and nothing else. It never sees the candidate spec file, so it can't be biased by it.

# graders/judge.py (trimmed)
RUBRIC = """You are grading an AI coding agent's output.
Answer each question with exactly YES or NO on its own line,
followed by one sentence of justification.
Questions:
{questions}

--- DIFF ---
{diff}
--- PR DESCRIPTION ---
{pr_body}
"""

4. Scoring and the CI gate

Each fixture yields a score: deterministic failures are hard fails, judge questions are weighted 1 point each. The suite score is the mean across fixtures. CI compares it against the baseline from main.

A prompt PR is blocked if:

• Any fixture that passed on main now hard-fails (regression)
• The suite score drops more than 3 points

The CI job posts a diff report as a PR comment: which fixtures flipped, and the judge's one-sentence reasons for any changed answer. That comment is honestly the most useful artifact in the whole system. It turns "I think this wording is better" into "this wording made fixture #23 stop writing tests, here's the evidence." 💡

Running the full suite costs me about $4 and 25 minutes. I run a 15-fixture smoke subset on every push and the full 60 on merge to main and nightly.

Lessons Learned

1. Fixtures from incidents are worth 10x fixtures you invent

Every time the agent does something dumb in production, I freeze it into a fixture the same day. Those fixtures catch real regressions. The "golden path" fixtures I wrote from imagination almost never fail. If you only do one thing from this post, do this.

2. Assert on behavior, not on output text

My first version asserted on strings in the final diff. It was flaky within a week, because a correct fix can be written ten different ways. Asserting on what the agent did (which tools it called, which files it touched, whether it ran tests) is stable across runs. Save the fuzzy judgments for the LLM judge.

3. Non-determinism is a feature you have to budget for

The same fixture with the same prompt will not produce identical runs. I run each fixture 3 times and take the median score. That tripled the cost and cut false alarms to nearly zero. A single-run eval suite will train you to ignore it, which is worse than having none.

4. The judge needs to be dumber than the worker

When I used the same large model as both worker and judge, the judge was too forgiving. It would rationalize the worker's choices. Switching to a smaller model with a tight rubric made the judge stricter and 5x cheaper. Counterintuitive, but it held up across months of runs.

5. Prompt diffs need code review, and now they get it

Once the suite existed, prompt changes started going through actual review. Reviewers ask "which fixture covers this?" the same way they'd ask "where's the test?" That cultural shift mattered more than any single caught regression. The spec file went from a scratchpad to a maintained artifact with a changelog.

What's Next

Two things I'm working on:

• Mutation testing for prompts. Automatically delete each rule from the spec file one at a time and check that at least one fixture fails. Rules that nothing depends on are dead weight (and there are more of them than I'd like to admit).
• Fixture aging. Fixtures capture a moment in the codebase. Some are now testing behavior on code that no longer looks like production. I want an automated "is this fixture still representative?" check.

I'll write both up once they've survived a month of real use.

Wrap-up

If your agent's behavior lives in prompt files, those files are code, and code without tests will drift. You don't need anything fancy to start. One fixture from your last incident, one grep-based grader, and a CI job that runs it. Build from there.

If this was useful, follow me here on Dev.to 🚀. I write about running autonomous coding agents in production, including the parts that break. And if you've built something similar, drop a comment. I'd love to compare notes on graders. ✅

Comments

[Mythex]

The "do not list every command" → "run fewer commands" drift is a perfect example of a non-local prompt effect, and freezing incidents into fixtures the same day is the part I'd steal first.

One grader gap worth checking: ran_command: pytest still passes if the agent ran the tests and then edited code again afterwards. Tests green, then a "small cleanup" edit, then the PR. Since your JSONL keeps the order of tool calls, asserting that the last test run comes after the last file edit closes that cheaply.

On mutation testing: a lighter first step is to tag each rule in the spec with the fixture IDs that cover it, as a comment next to the rule. Rules with no tag are your deletion candidates, and reviewers can ask "which fixture?" right in the diff.