How I Built an Eval Suite to Catch My AI Agent's Silent Regressions

TL;DR

My autonomous coding agent got quietly worse for about two weeks and nothing told me. No errors, no crashes — just slightly sloppier output that I didn't notice until I went digging. I built a small eval harness that runs the agent against a fixed set of "golden" tasks every time I touch its config, scores the output, and flags it if quality drops. Here's how it works, what I got wrong the first two times, and why "it still runs" is a terrible definition of "it still works."

The Problem

I run an agent that handles real engineering work autonomously — refactors, bug fixes, small features, PR descriptions, the whole loop from "here's a task" to "here's a merged change." It's been running for months, and over that time I've tweaked its system prompt, adjusted its tool permissions, swapped in a different model tier for cheaper tasks, and added new instructions as I learned what broke.

Every one of those changes felt safe in isolation. I'd make the edit, run the agent on whatever task was in front of me, see it work, and move on.

The problem is "I saw it work once" is not the same as "it still works as well as it did last week." Language model behavior doesn't fail loudly. It doesn't throw a stack trace when a prompt edit makes it 15% more likely to skip an edge case, or when a new instruction makes it verbose in a way that buries the actually important parts of a PR description. It just... drifts.

I found out the hard way. I went back through two weeks of the agent's PRs to write up some metrics and noticed a pattern: commit messages had gotten noticeably more generic ("update logic" instead of naming what changed and why), and it had started leaving half-finished-looking placeholder comments in spots where it used to just write the real implementation. Nothing had crashed. Nothing had errored. I'd shipped a regression through pure vibes-based QA and only caught it because I happened to be looking.

I traced it back to a system prompt edit from about two weeks earlier. I'd added a paragraph asking the agent to "be more concise in commit messages" to fix a different, smaller complaint — messages that ran too long. The model overcorrected in a direction I didn't predict: conciseness turned into vagueness, and the same instruction that trimmed one verbose message also quietly trimmed the specificity out of every message after it. A single line, added for a reasonable-sounding reason, degraded output quality for two weeks before I noticed.

That's the moment I stopped trusting "looks fine when I glance at it" as a quality bar.

How I Solved It

The fix is unglamorous: a small, boring eval suite. Not an LLM-judges-everything framework, not a fancy dashboard — a golden set of tasks with known-good answers, run automatically, scored, and diffed against the last known baseline.

Step 1 — Build a golden set, not a vibes set

I picked about 20 tasks that represent the agent's actual job: a couple of refactors, a bug fix with a subtle edge case, a task that requires reading three files before touching any of them, a task that should explicitly not touch a certain file, and a couple of "should ask before doing this" scenarios. The key constraint: every task needs an objective way to check the output, not just "did it seem reasonable."

# golden_tasks.py
GOLDEN_TASKS = [
    {
        "id": "refactor-01",
        "prompt": "Extract the duplicated validation logic in user_service.py into a shared helper.",
        "checks": [
            "helper_function_exists",
            "no_duplicated_validation_blocks",
            "existing_tests_still_pass",
        ],
    },
    {
        "id": "boundary-01",
        "prompt": "Fix the off-by-one in paginate(). Do not touch the caching layer.",
        "checks": [
            "pagination_bug_fixed",
            "cache_py_unchanged",
        ],
    },
    # ...
]

Each check is a small deterministic function — AST inspection, a test run, a diff assertion — not another LLM call grading vibes. I learned this one the hard way (more below).

Step 2 — Run it after every meaningful change

Any time I touch the system prompt, tool permissions, or the model tier, the harness runs the full golden set in isolated sandboxes and records a pass/fail plus a few soft signals: diff size, number of files touched outside the expected set, whether it asked for confirmation when it should have.

def run_eval(agent_config, tasks=GOLDEN_TASKS):
    results = []
    for task in tasks:
        sandbox = spin_up_sandbox(task["id"])
        output = run_agent(agent_config, task["prompt"], cwd=sandbox)
        score = {
            "task_id": task["id"],
            "checks": {c: run_check(c, sandbox, output) for c in task["checks"]},
            "diff_size": diff_line_count(sandbox),
            "files_touched": files_changed(sandbox),
        }
        results.append(score)
    return results

Step 3 — Diff against baseline, not against a fixed bar

A pass/fail count alone hides drift. What actually caught my two-week regression was diffing the current run against the previous run: same tasks passing, but diff sizes creeping up 30%, or a "should ask for confirmation" task quietly stopping asking. I store the last N runs and flag any task whose soft metrics move more than a threshold, even if the hard pass/fail didn't change.

flowchart LR
    A[Config change] --> B[Run golden set in sandboxes]
    B --> C[Score: pass/fail + soft signals]
    C --> D{Diff vs last baseline}
    D -->|Within threshold| E[Promote as new baseline]
    D -->|Drift detected| F[Block + show which task regressed]

Step 4 — Make failures actionable, not just alarming

The first version of this just printed "REGRESSION DETECTED" and I ignored it twice because I had no idea what to do with that. Now every failure links back to the specific task, the specific check that failed, and a diff between this run's output and the last passing run's output for that same task. If I can't tell what changed in under 30 seconds, the eval isn't useful — it just becomes another warning I learn to ignore.

Lessons Learned

1. "It still runs" is not a quality bar. Agents rarely fail loudly. They degrade in ways that look like normal variance until you have a baseline to compare against. If you don't have a golden set, you're flying blind on every prompt tweak.

2. Don't use an LLM to grade the LLM, at least not alone. My first version used a second model call to score "is this a good PR description?" It was inconsistent between runs on identical input — sometimes a 7, sometimes a 9, no code changed. Deterministic checks (does the test suite pass, is the forbidden file untouched, does a specific function exist) are boring but they don't have their own variance stacking on top of the thing you're trying to measure. Save the LLM grading for genuinely subjective stuff, and even then, average multiple samples.

3. Soft signals catch what hard pass/fail misses. My regression didn't fail any check — everything still technically passed. It was the diff size and "touched more files than expected" metrics that would have caught it two weeks earlier if I'd been tracking them against a baseline instead of a fixed threshold.

4. Keep the golden set small and adversarial, not big and average. I was tempted to throw 200 historical tasks at this. Twenty sharp, deliberately tricky tasks — the ones designed to catch a specific failure mode — told me more than 200 tasks that were all variations of "write a simple function." Pick tasks for what they'd catch, not for coverage numbers.

5. Run it before you trust a change, not after you've already shipped a week of work on top of it. The whole point evaporates if the eval is a nice-to-have you run occasionally. It has to be the gate between "I edited the prompt" and "I let the agent loose on real work" — otherwise you're back to vibes-based QA with extra steps.

What's Next

Right now the golden set is hand-curated, which means it only catches regressions I thought to test for. The next iteration is auto-generating new golden tasks from real failures — any time the agent does something wrong in production that I have to manually fix, that failure becomes a new permanent entry in the set, so the same mistake can't silently sneak back in unnoticed.

I'm also looking at running the eval on a schedule even when nothing's changed, since model providers update models under the hood and that's a config change I don't control. And I want to track the soft-signal baselines over a longer window than "last run" — a slow one-percent-a-week creep would still slip past a single-run diff, and that's exactly the kind of drift this whole project exists to catch. A rolling seven-day baseline instead of a single previous run would probably have caught my regression a week earlier than I actually caught it.

One thing I'm deliberately not doing yet: full statistical significance testing on pass rates. With only twenty tasks, a single flaky failure looks like a trend when it's actually noise. I'd rather grow the golden set to a size where that starts to matter than bolt stats onto a sample size that can't support it.

Wrap-up

If you're running an autonomous agent on anything that matters, "I checked it once and it looked fine" is not a regression testing strategy. A golden set with deterministic checks and baseline diffing costs an afternoon to build and will catch the two weeks of silent drift that vibes-based QA never will.

If this was useful, follow me here for more build logs from running an agent on real engineering work — and if you've built your own eval setup for an AI agent, I'd genuinely like to hear how you're scoring it. Drop it in the comments.