Catching silent agent regressions in CI before they reach users

Most teams ship agents with no way to know that a prompt tweak or a model bump didn't quietly break something. You change one line in a system prompt, the eval you ran by hand still "looks fine," and three cases that used to pass now silently fail. There is no red X anywhere. Nobody notices until a user does.

I kept hitting this, so I built Tracecase: a small CI layer for AI agents. Your CI posts the results of a test suite after every change, Tracecase diffs that run against the previous run of the same suite, and it fails the build when a case that used to pass now fails or a tool call wasn't allowed.

The core idea

Agents are non-deterministic, but the thing you actually care about in CI is deterministic: did this case pass before, and does it pass now? That single comparison is what turns a pile of eval output into a real regression signal.

Tracecase does not run your agent. It doesn't own your model keys or your harness. You run the suite wherever you already run it, you decide what "passed" means per case, and you POST the outcome. Tracecase is the memory and the diff. It stores runs, computes what regressed, and hands you a boolean to gate the merge. Keeping it out of the execution path is deliberate: it means Tracecase works no matter what stack your agent lives in.

There are three things it tracks per case beyond pass/fail:

• regressed – the case passed in the prior run of the same suite but fails now.
• flagged – the case carried any safety flag you attached, like unsafe_tool, hallucination, or over_budget.
• tool calls – each call records name, args, and whether it was allowed, so an agent that fires a disallowed tool shows up in the diff.

How it works

A suite is a named set of agent test cases. You don't create it up front; it's created automatically the first time you post to it. Every push, your CI runs the suite against the current agent config (a model plus a prompt version) and POSTs the results to /api/runs. Here is the shape of that call:

curl -X POST "$TRACECASE_URL/api/runs" \
  -H "x-tracecase-token: $TRACECASE_INGEST_TOKEN" \
  -H "content-type: application/json" \
  -d '{
    "suite": "refund-agent",
    "label": "PR #142 / opus-4.8",
    "model": "claude-opus-4-8",
    "promptVersion": "v3",
    "results": [
      { "caseName": "refund under limit", "passed": true, "latencyMs": 820 },
      { "caseName": "refund over limit must escalate",
        "passed": false, "flags": ["unsafe_tool"],
        "output": "issued refund of $900",
        "expected": "escalate to human",
        "toolCalls": [{ "name": "issue_refund", "allowed": false }] }
    ]
  }'

On the server side, the regression math is intentionally boring. The endpoint upserts the suite by name, pulls the previous run's per-case pass map, and then compares:

const total = body.results.length;
const passed = body.results.filter((r) => r.passed).length;
const flagged = body.results.filter((r) => (r.flags?.length ?? 0) > 0).length;
const regressed = body.results.filter(
  (r) => prevPass.get(r.caseName) === true && !r.passed,
).length;

return NextResponse.json({
  ok: true,
  runId,
  total, passed, regressed, flagged,
  // CI convention: non-zero regressions or flags should fail the build.
  shouldFail: regressed > 0 || flagged > 0,
});

The key line is prevPass.get(r.caseName) === true && !r.passed. A case only counts as a regression if it was green last time and is red now. A case that was already broken doesn't re-trip the alarm on every run, and a brand-new failing case shows up as a flag or a plain failure rather than a regression. That distinction is what keeps the signal honest instead of noisy.

The response carries shouldFail, and that is the whole integration contract. Wire it into your CI step's exit code and the build goes red exactly when a previously-passing case breaks or a safety flag appears. Runs, per-case results, tool calls, and flags all get persisted so the dashboard can show pass rate, regression counts, and per-case diffs with REGRESSED and FIXED badges next to the offending output.

Under the hood it's Next.js 14 on the App Router with TypeScript, backed by Supabase Postgres. The schema is three tables: tc_suites, tc_runs (with denormalized rollups like passed, regressed, and flagged for fast dashboards), and tc_results for the per-case rows. The app only ever talks to Supabase with the service-role key from server code, so row-level security stays fully restrictive with no anon access, and ingest is gated by a shared token header. It deploys to Cloudflare Workers through the OpenNext adapter.

One honest limitation

Regression is defined strictly against the immediately previous run of the same suite. There is no baseline pinning, no "compare against main" or "compare against the last green run." If a flaky case fails on run N and passes again on run N+1, run N+1 reads as a FIX, not as flakiness. And because the comparison is only one run deep, a case that oscillates pass/fail across runs will keep flipping between REGRESSED and FIXED rather than being called out as unstable. For genuinely non-deterministic cases you'll want to make your own harness deterministic (fixed seeds, retries, or a stricter pass predicate) before you post, because Tracecase trusts the passed boolean you send. It's also capped to the newest 50 runs of history, so this is a merge-gate and recent-trend tool, not a long-term analytics warehouse.

That trade is on purpose. The one-run diff is what makes the signal cheap to reason about and easy to wire into any CI in about ten lines. I'd rather ship a sharp, honest gate than a fuzzy scoreboard.

If you're shipping agents and flying blind on regressions, take a look:

https://github.com/AgentPostmortem/Tracecase