DEV Community

Cover image for Your AI agent broke silently, and every test passed
Syed Mohammed Faham
Syed Mohammed Faham

Posted on

Your AI agent broke silently, and every test passed

#llm #ai #testing #python

TL;DR: AI agents regress silently: a prompt tweak or a model bump changes behavior with no exception and no red CI. agentsnap records your agent's LLM + tool calls once as a committed "golden" snapshot, then fails your tests when behavior drifts across four dimensions: the tool sequence, the arguments, which tool the model itself chose, and semantic meaning. Run it in replay mode on every PR (deterministic, zero API calls) and live nightly (catches real model drift). Works with raw OpenAI/Anthropic and with LangChain / Pydantic AI / the OpenAI Agents SDK, with no wrappers. pip install agentsnap.

Here's a bug I kept shipping, and I bet you have too.

You've got an LLM agent in production. It works. Then you do something completely reasonable: tweak a prompt, bump gpt-4o-mini to the next version, refactor a tool function. You run your tests. Green. You ship.

And the agent is now subtly worse. It picks a different tool. It drops a step. Its answers got a little vaguer. Nothing threw an exception. Nothing turned CI red. You find out three days later from a user, or worse, from a metric.

This is the thing about agents that normal testing just doesn't cover: the failure is silent and the output is non-deterministic. You can't assert output == "expected string" when the model phrases things differently every run. And re-calling the real API on every test is slow, flaky, and costs money.

So I built agentsnap: snapshot testing for AI agents. This is the ~10-minute tour: what it is, how to use it, and how it works under the hood.

agentsnap catching a prompt change

That GIF is the whole idea in eight seconds: record a golden run, and when someone later changes the prompt, agentsnap catches it, replaying the recorded calls, with zero API calls.

The mental model: snapshots, but for agent behavior

One call drifted, nothing threw

If you've used Jest snapshots, or VCR / cassettes for HTTP tests, you already get it.

You run your agent once in a known-good state. agentsnap records every LLM call and tool call it makes (the messages sent, the responses, the tool arguments) and writes it all to a committed .json snapshot. That file is your golden run.

From then on, every test run compares the new behavior against that golden snapshot. If the agent starts behaving differently, the test fails with a structured diff telling you exactly what changed.

It's pip install agentsnap, MIT-licensed, Python 3.10+.

How to use it (the 3-minute version)

1. Install and set up:

pip install agentsnap
agentsnap init   # pick a semantic-comparison backend; offline embeddings need no key
Enter fullscreen mode Exit fullscreen mode

2. Record a golden run. The neat part: you don't wrap or change your agent code. agentsnap patches the SDK classes directly, so a raw anthropic or openai client is captured automatically.

from agentsnap import PatchSet, AgentRecorder
import anthropic

def my_agent(question):
    client = anthropic.Anthropic()          # raw client, no wrapper
    return client.messages.create(...).content[0].text

with PatchSet():
    with AgentRecorder("my_agent") as rec:
        rec.output = my_agent("What is Python?")
# writes __agent_snapshots__/my_agent.json  (commit this file)
Enter fullscreen mode Exit fullscreen mode

3. Assert on every run after that:

from agentsnap import PatchSet, AgentAsserter

with PatchSet():
    with AgentAsserter("my_agent") as a:
        a.output = my_agent("What is Python?")
# raises AgentRegressionError if behavior drifted
Enter fullscreen mode Exit fullscreen mode

Or skip the ceremony entirely and use the pytest plugin: snapshot.run() records on the first run and asserts on every run after:

def test_my_agent(snapshot, agentsnap_instrument):
    with snapshot.run("my_agent") as s:
        s.output = my_agent("What is Python?")   # captured automatically
Enter fullscreen mode Exit fullscreen mode 
pytest
Enter fullscreen mode Exit fullscreen mode

That's it. First run records, later runs guard.

How it works

How agentsnap works

Three ideas do all the work.

1. Zero-instrumentation capture (PatchSet)

Most tools make you wrap your client: wrapped = Something(client). That's annoying, and it doesn't work for agent frameworks that build their clients internally.

PatchSet monkey-patches the SDK classes themselves: anthropic.resources.messages.Messages.create, OpenAI's Completions.create, their async variants, the OpenAI Responses API. So any client, wrapped or not, created anywhere in your call stack (or inside LangChain, Pydantic AI, the OpenAI Agents SDK) is captured. No glue code.

2. Four comparison dimensions

When a new run comes in, agentsnap diffs it against the golden across four axes:

  • Structural — the sequence of tool calls (by name and order), via edit distance.
  • Arguments — the arguments each tool was called with.
  • Model tools — which tool the model itself asked to call. This is the sneaky one. Your code might handle a tool call gracefully, but if the model started choosing delete_file where it used to choose search, that's a behavior change you absolutely want to know about — even before your code runs it.
  • Semantic — the meaning of the responses and the final output.

If anything drifts past its threshold, you get an AgentRegressionError with a diff showing exactly what moved.

3. Replay vs. live — two modes for two jobs

This is the part that makes it fast enough to run constantly.

Mode LLM calls Catches Run it
replay none, recorded responses replayed code regressions (prompt edits, tool wiring, changed call counts) on every PR
live real API model/behavior drift nightly

In replay mode, agentsnap feeds the recorded response back to your agent instead of calling the API. No key, no cost, no network, fully deterministic. The comparison flips to the request side: it fails if your code sends different prompts or makes a different number of calls. That's your PR check: it runs in milliseconds and never flakes.

Live mode makes the real calls against the current model, so it catches drift that only appears when the model itself changes. That's your nightly job.

The pattern in one line: replay on pull requests, live nightly.

A real example: catching a prompt change

Say your agent summarizes text. You record a golden with the prompt "Summarize: What is Python?". A week later, a teammate "improves" it to "You are a pirate. What is Python?" (it happens). Replay catches it instantly:

Agent regression in 'summarize_agent'
=====================================

[ARGS] llm_call[0].messages:
  values_changed:
    root['messages'][0]['content']:
      'Summarize: What is Python? ...' -> 'You are a pirate. What is Python? ...'

Failed checks: ['llm_requests']
Enter fullscreen mode Exit fullscreen mode

No API call. No flake. Just a clear diff of the exact thing that changed. When the change is intentional, you approve it in one command:

agentsnap update summarize_agent   # promotes the new run to the golden
git commit -m "approve: pirate summaries, apparently"
Enter fullscreen mode Exit fullscreen mode

There's also agentsnap status (a CI-friendly overview of which snapshots pass, fail, or need re-running) and agentsnap diff to inspect a change before approving.

It works with your framework

Because capture happens at the SDK level, agents built on frameworks work through PatchSet with no per-framework code:

  • Pydantic AI
  • OpenAI Agents SDK (including the Responses API)
  • LangChain (sync and async)
  • LangGraph (node-level events)
  • plus raw OpenAI / Anthropic, and adapters for Gemini, Cohere, Mistral, and Groq.

The first three are verified in CI against the real libraries on every change, not mocks. (Fun aside: setting that up caught two genuine bugs in agentsnap that unit tests never would have; the real frameworks do surprising things with raw-response wrappers. Dogfooding pays.)

Test yourself

You've now met all three ideas: the two modes, the four dimensions, and silent tool drift. Put them together, click an answer for instant feedback:

Try it 

pip install agentsnap
Enter fullscreen mode Exit fullscreen mode

If you're building with LLM agents, I'd genuinely love to know whether these four dimensions match how you think about regressions, and what breaks on your setup. It's a solo open-source project and real-world feedback is what drives it.

Go record a golden run. Next time something silently changes, you'll actually find out.

Connect & Share

I’m Faham — currently diving deep into AI/ML. I share what I learn as I build real-world AI apps.

If you find this helpful, or have any questions, let’s connect on LinkedIn and X (formerly Twitter).

AI Disclosure

This blog post was written by Faham with assistance from AI tools for research, content structuring, and image generation. All technical content has been reviewed and verified for accuracy.

Top comments (0)