ke yi

Posted on Oct 28 Originally published at fp8.co

AI Agent Regression Testing: Production Traces to CI

#aiengineering #testing

AI Agent Regression Testing: How to Turn Production Traces into CI Gates

TL;DR: Trace-based regression testing turns a failing production agent trace into a CI test case. The trace's recorded tool and model outputs become fixtures, so the test replays offline and deterministically on every pull request. Tracely is an MIT-licensed open-source project that packages this loop: OTLP ingest, graders, failure clustering, promotion of a trace to a case, and a GitHub Action gate. Langfuse, LangSmith and Promptfoo cover parts of the same loop. None of them removes the main engineering cost, which is routing every tool and model call through a seam you can record and replay.

Key Takeaways

An agent regression test is a test built from a real failure. A trace that went wrong in production becomes a case your CI must pass forever after.
Hermetic replay is what makes the test usable in CI. Recorded tool and LLM responses stand in for live calls, so the suite needs no API keys, costs nothing in tokens, and gives the same result every time.
A fail-to-pass contract is the check that the test is trustworthy. The case must fail on the old code and pass on the fix, or the test proves nothing.
Tracely (about 1.2k GitHub stars, MIT, Python/FastAPI with ClickHouse) bundles ingest, grading, clustering, promotion and a tracely-gate GitHub Action in one self-hosted stack.
Langfuse and LangSmith let you add traces to datasets and run experiments over them. Promptfoo runs assertion suites in CI. All three re-call the model live unless you add your own fixture layer.
Replay only works if the agent sends tool and model calls through a recordable seam. Plan that refactor before you pick a tool.

Why do AI agents need regression tests built from traces?

Agent failures rarely show up in the test inputs a team writes on day one. They come from real conversations: a user phrases a refund request the planner did not expect, a tool returns an empty list instead of an error, or a model upgrade changes how often the agent calls a search tool. By the time someone sees the failure, it lives in a trace (a nested record of every prompt, model response, tool call and result in one run).

Teams usually handle this in one of two ways, and both leak:

Hand-written eval sets. Someone reads the bad trace, writes a new test input and an expected output, and adds it to a YAML file or dataset. The new input is a paraphrase of what happened, not a copy. The tool responses that caused the failure are usually lost, and the test then calls live tools that now return different data.
Dashboards with no gate. The observability platform records the failure, an online evaluator scores it low, and the score sits on a chart. Nothing stops the next pull request from bringing the same failure back.

Trace-based regression testing closes that gap. The trace is the test: its exact inputs, the exact tool outputs the agent saw, and the model responses in recorded order. A grader decides pass or fail. The suite runs on every pull request and blocks merges that bring a known failure back.

This is the same discipline as "write a failing test before you fix the bug" in ordinary software. The difference is that agents are non-deterministic and depend heavily on external services, so the test needs recorded fixtures before it can be repeated.

What does a trace-to-test pipeline look like?

The loop has five stages. Every implementation, whether bought or built, needs all five.

Stage	What happens	What breaks if it is missing
Observe	Every production run emits a trace (OpenTelemetry `gen_ai.*` or OpenInference attributes) indexed by agent, conversation, turn and step	You cannot reproduce a failure you never recorded
Grade	Structural checks (schema, tool-call order, no model needed) and LLM judges score runs online	Failures are found by users, not by the system
Triage	Failing runs are clustered by structural signature and semantic similarity	31 broken runs become 31 tickets instead of one issue with a count
Promote	A representative failing trace becomes a hermetic case with its fixtures recorded	The "test" re-calls live services and drifts
Gate	CI replays the suite against the change and fails the build on regressions	The fix ships, and the next refactor quietly reverts it

The Promote stage is where most homegrown setups stop. Copying a trace's input into a dataset is easy. Copying the world the agent saw (tool results, retrieval hits, intermediate model outputs) and replaying it in order is the actual work.

How does Tracely implement trace-native CI/CD?

Tracely is an open-source project by Julien Wuthrich that describes itself as "trace-native CI/CD for AI agents." It is MIT-licensed, has about 1.2k stars at the time of writing, and is still early (a few dozen commits). Its stack is a Python backend (FastAPI, Celery, uv) and a Next.js 15 UI. It stores traces in ClickHouse, the case registry in Postgres with pgvector, the queue in Redis and blobs in MinIO/S3. A local demo starts with docker compose --profile demo up -d --build --wait.

Ingest: OTLP first, SDK second

Any OpenTelemetry exporter can post to POST {endpoint}/v1/traces with a Bearer ingest key. Tracely reads both gen_ai.* semantic-convention attributes and OpenInference attributes, so existing instrumentation keeps working. The Python SDK (tracelyai-sdk) adds auto-instrumentation for OpenAI, Anthropic, Google, Mistral and LangChain:

import tracely_sdk as tracely

tracely.init(
    endpoint="http://localhost:8000",
    api_key="tracely_dev_key",
    service_name="support-agent",
    env="prod",            # prod | staging | ci | dev — the gating axis
    instrument="auto",
)

with tracely.trace(agent="support-agent", conversation="conv-1", user="u_42"):
    client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": "Where is order ORD-4471?"}],
    )

The env field matters more than it looks. It separates production traces (the source of failures) from CI traces (the ones the gate grades), so the same pipeline can serve both.

Grade and cluster

Evaluators come in two kinds. Structural checks need no model. LLM judges run at conversation, run or span level and can output a score, boolean, number, text or JSON-schema result. Failing runs are then clustered "structurally and semantically" with a structural signature plus embedding similarity. In the project's words, "31 broken runs become one issue with a count."

Promote with a fail-to-pass contract

This is the most important idea in the project. Promoting a failing trace bundles "recorded input, tool and LLM outputs" as fixtures. The case is trusted only if it "must fail on the old code and pass on the fix." A regression test that passes on the buggy code is a test that cannot fail, which is the most common way evaluation suites lie to their owners. The contract makes that check part of the workflow instead of leaving it to discipline.

Multi-turn behaviour uses scenarios instead of single cases: either a scripted conversation, or an adversarial goal that a red-team model improvises against.

Gate in GitHub Actions

Replay runs offline against the recorded fixtures, with no API keys and no model spend. tracely gate exits non-zero, posts a commit status and upserts a PR comment. The bundled action has two modes. Tracely can call your agent through an entrypoint (tracely replay planner --entrypoint my_pkg.agent:run, or --cmd "node run.js" for non-Python agents), or it can grade the CI traces your own test job emitted:

- uses: jwuthri-cloud/Tracely-ai/.github/actions/tracely-gate@master
  with:
    mode:  gate
    agent: planner
    api:   https://tracely.your-co.dev
    key:   ${{ secrets.TRACELY_KEY }}

Pin that @master reference to a commit SHA before you rely on it. A gate that can change underneath you is not a gate.

The constraint you cannot skip

The README states it plainly: hermetic replay "requires your agent to route tool/model calls through the SDK's call_tool / call_llm seam." Auto-instrumentation is enough to record a trace. It is not enough to replay one, because replay means intercepting the call and returning the recorded value instead of making it. If your agent calls requests.get from inside a tool, or a framework makes model calls you cannot reach, those calls go live during replay.

Other caveats from the project itself: judges and failure analysis degrade without an LLM key, the default AUTH_MODE=dev with tracely_dev_key is open access and only suitable for local use, and the Celery worker does not hot-reload.

How do Langfuse, LangSmith and Promptfoo handle the same loop?

None of the established tools is called "trace-native CI," but each covers some of the stages. The question is which stages you get for free and which you build.

Stage	Tracely	Langfuse	LangSmith	Promptfoo
Production tracing	Yes (OTLP + SDK)	Yes, core product	Yes, core product	No (test runs only)
Online grading	Structural + LLM judges	Online evaluators on sampled traffic	Online evaluators	No
Failure clustering	Built in	Manual filtering and annotation	Manual filtering and annotation	No
Trace → test	Promote to hermetic case	Add trace to dataset	Add run to dataset	Copy input into YAML test case
Recorded tool fixtures	Yes, bundled on promote	Inputs/expected outputs only	Inputs/reference outputs only	No, you write mocks
CI gate	`tracely-gate` action, PR comment	Experiments via SDK in your CI job	Evaluations via SDK in your CI job	`promptfoo eval` + GitHub Action
Offline / no model spend	Yes, by design	No, experiments call the model	No, evaluations call the model	No, unless you mock providers
Maturity	Early, small community	Mature, MIT core	Mature, commercial	Mature, MIT

A few clarifications on that table.

Langfuse and LangSmith datasets store inputs and expected outputs, not the world. Adding a trace to a Langfuse dataset or a LangSmith dataset captures what went in and, optionally, what should come out. When you run an experiment, your application runs again, live. That is the right design for comparing a new prompt or model against a fixed set of inputs: you want the new model's real behaviour. It is the wrong design for a regression test of agent logic, because the tools return today's data instead of the data that triggered the bug. For the trade-offs between these two platforms, see our LangSmith vs Langfuse vs Phoenix comparison.

Promptfoo is the strongest CI gate and the weakest trace source. Its GitHub Action and assertion language are mature, and it runs red-team suites no trace tool matches. But it never sees production. Every case is something a human typed, so the "trace to test" step is copy-paste. Our Langfuse vs Promptfoo comparison covers why many teams pair the two.

Tracely's real differentiator is the fixture bundle plus the fail-to-pass contract. The tracing and judging overlap heavily with Langfuse. Indeed, the project describes parts of its design as mirroring Langfuse's. What is new is treating promotion as a test-engineering operation with a validity check, not as a "save to dataset" button.

How do you build hermetic replay without adopting a new platform?

If you already run Langfuse or LangSmith and do not want another stack, you can build the missing stages yourself. The core is a single seam that every tool and model call goes through. This is a generic sketch of the pattern, not any vendor's API:

import json, hashlib
from contextvars import ContextVar

_mode = ContextVar("mode", default="live")       # live | record | replay
_tape = ContextVar("tape", default=None)          # list of recorded calls

def _key(kind, name, args):
    blob = json.dumps([kind, name, args], sort_keys=True, default=str)
    return hashlib.sha256(blob.encode()).hexdigest()[:16]

def seam(kind, name, fn, **args):
    """Every tool call and LLM call goes through here."""
    mode, tape = _mode.get(), _tape.get()
    if mode == "replay":
        rec = tape.pop(0)                         # strict order, not lookup
        if rec["key"] != _key(kind, name, args):
            raise AssertionError(f"divergence at {kind}:{name}")
        return rec["output"]
    out = fn(**args)
    if mode == "record":
        tape.append({"key": _key(kind, name, args), "kind": kind,
                     "name": name, "output": out})
    return out

Three design choices in that sketch decide whether the tests are useful.

Replay in strict order, and fail on divergence. If the fixed agent makes a different call than the recorded one, a lookup-by-key replay silently returns nothing or the wrong record. Raising on divergence turns "the agent took a different path" into an explicit result. Sometimes that is the fix working, and you re-record. Sometimes it is a new bug.
Decide which calls are fixtures and which are under test. In a test of planner logic, the model calls are the system under test, so replaying them proves only that the code around them is deterministic. A common split: replay tools and retrieval always, and replay the model only for the steps before the step that failed, then call the model live (or a pinned cheap model) for the step being tested. Tracely's offline mode replays everything. That is cheaper and deterministic, but it catches regressions in orchestration code, not in model behaviour.
Grade the outcome, not the transcript. Exact-match on model text breaks on every model upgrade. Grade structural facts ("called refund with the original order ID", "did not call cancel_subscription") and use an LLM judge only where structure cannot express the requirement.

Then write the fail-to-pass check as an actual test step: run the case against the parent commit and require a failure, run it against the fix and require a pass. Without that step you will end up with cases that were promoted from a trace but never reproduced the bug, and they pass forever.

When is trace-based regression testing the wrong tool?

It is not a replacement for every kind of evaluation.

Model or prompt comparisons need live calls by definition. Use dataset experiments (Langfuse, LangSmith) or a Promptfoo matrix, and accept the token cost.
Security and adversarial testing need inputs no user has sent yet. A trace suite only knows past failures. Red-team generation (Promptfoo plugins, or Tracely's adversarial scenarios) fills that gap.
Agents whose tools have side effects you cannot record. Examples are a browser agent, a coding agent running shell commands, or anything that depends on wall-clock time. These need a sandbox or a simulated environment, not a tape. Replaying a recorded DOM is possible but brittle.
Very low traffic. If you have a handful of production runs a week, hand-written cases are faster than running an ingest stack.

The approach pays off when you have steady production traffic, recurring failure modes, and orchestration code that changes often. That is the profile of most customer-facing tool-calling agents.

How should a team adopt trace-to-test in practice?

A sequence that keeps the investment proportional:

Standardise tracing on OpenTelemetry gen_ai.* attributes (see the OpenTelemetry GenAI semantic conventions). Every tool in this article can ingest them, so the choice of backend stays reversible.
Introduce the call seam in the agent code. Do this first, because it is the step every option needs and the one that takes longest.
Promote by hand for a month. Take the five worst production failures, record them, write structural graders, and run them in CI. You will learn which calls leak past the seam.
Then decide on a platform. If clustering and promotion are now the bottleneck, a tool like Tracely earns its operational cost (ClickHouse, Postgres, Redis and object storage). If your observability platform's datasets plus your own fixture layer are enough, stay there.
Treat the gate as production infrastructure. Pin action versions, alert on gate failures you did not expect, and review the suite whenever you upgrade a model. A failure across the whole suite after a model change is a signal, not noise.

FAQ

What is AI agent regression testing?

AI agent regression testing checks that a change to an agent's code, prompts or model does not bring back a failure that was already fixed. The strongest form builds each test from a real failing trace and replays it on every pull request.

What is hermetic replay for LLM agents?

Hermetic replay runs an agent test with recorded tool and model responses instead of live calls. The test needs no network access, no API keys and no token spend, and it gives the same result on every run, which makes it usable as a CI gate.

What is a fail-to-pass contract?

A fail-to-pass contract is a rule that a regression case must fail on the code that had the bug and pass on the code that fixes it. It proves the test actually reproduces the failure, instead of being a test that cannot fail.

Is Tracely a replacement for Langfuse or LangSmith?

Not yet for most teams. Tracely overlaps with their tracing and judging but is an early project with a small community. Its distinctive parts are fixture-bundled promotion, failure clustering and an offline CI gate, which you can also build on top of an existing platform.

Can Promptfoo test agents built from production traces?

Promptfoo can gate CI on agent outputs and run red-team suites, but it does not ingest production traces. You copy the trace's input into a test case by hand and write your own mocks for tool responses.

Do I need OpenTelemetry to do trace-based testing?

No, but it keeps your options open. Tracely, Langfuse, LangSmith and Phoenix all accept OpenTelemetry traces, so instrumenting once with gen_ai.* attributes lets you change backends without re-instrumenting the agent.

Sources

Originally published at fp8.co. Subscribe for weekly AI engineering analysis at fp8.co/newsletters.

DEV Community