DEV Community: Nicolai Bohn

Where AI agent test cases come from: synthetic vs. production data

Nicolai Bohn — Mon, 05 Oct 2026 14:56:22 +0000

Before you can evaluate an agent, you need something to evaluate it against. Which test cases you can get depends mostly on timing. Before launch, everything you test with is something your team came up with. After launch, your users start writing test cases for you, whether you collect them or not.

This post compares the main sources and shows how the mix changes as your agent matures. There's a walkthrough video at the end.

Two sources: synthetic and production data

Synthetic test cases are the ones you create yourself. You can write them while exploring the agent in a playground, or generate them from a description of what you want to test. They are available from day one, and they are how you cover reliability (does the agent meet your requirements?) and robustness (does it hold up against jailbreaks, prompt injection and other adversarial inputs?).

Tests from production data come from conversations real users had with your agent. They include the edge cases nobody on the team thought to write down. You only get them once you are live.

What each method is good for

Playground

The fastest way to start. You chat with your agent, and when it gets something wrong, you save that exchange as a test case. Say your claims assistant tells a customer they have 30 days to file when your policy says 14. That conversation becomes a regression test in a few clicks.

The limit is you. Coverage only reaches as far as the scenarios you think of, and recording cases one at a time takes a while.

Synthesizer

Here you describe what you want to test, for example "customers asking about cancellation deadlines, including attempts to get the agent to promise refunds it can't give". You rate a few samples so the generator learns what you mean, and you get back a full test set. The same works for multi-turn simulations, where a simulated user pushes the agent over several turns instead of a single prompt.

You get broad coverage of your requirements and adversarial inputs before a single user has touched the agent. You still have to define the scope and review samples, though. And generated inputs tend to read cleaner than real ones. Real users make typos and ask several things in one message.

Real conversations

Nothing is more realistic than what your users actually typed. Production conversations show you real phrasing and real edge cases, and they tell you what people want from the agent, which is not always what the spec assumed.

They have two catches. You need to be live, and you usually want to find problems before your users do. And real conversations contain personal data, so they need cleanup before they become test cases. For teams in Europe that is a GDPR question as much as a technical one.

Working from a coding agent instead? You don't have to go through a UI for any of this. The Rhesis MCP server exposes the synthesizer and your production traces to Claude Code, Cursor and other MCP clients. Paste a product spec and the agent proposes requirements, metrics and test sets, creating them once you approve. Or ask it to read recent traces and turn the conversations worth keeping into test cases.

Your test sources shift as the agent matures

In practice, most teams end up using all three, in roughly this order:

Exploring by hand. In the first iterations, domain experts use the playground and record what they see. This is where your first expectations come from.
Generating at scale. Once requirements settle, the synthesizer and simulations carry most of the volume, covering requirements and adversarial inputs systematically.
Learning from production. After launch, real conversations show you what the synthetic sets missed. A failure from production also makes a good seed for new synthetic variations.
Automated where possible. Routine checks run automatically. Experts stay in the loop on the hard, novel cases.

See it in action

In the video, I go through each method in Rhesis, from the first case recorded in the playground to test cases built from production conversations.

Rhesis is open source. You can try it on app.rhesis.ai or self-host it from GitHub.

Originally published on the Rhesis blog.

How do you source test cases for your agents? Mostly synthetic, mostly production, or something else entirely? I'd like to hear in the comments.

Launch week day 1: drive the full AI testing workflow from inside any AI tool

Nicolai Bohn — Mon, 04 May 2026 17:17:03 +0000

Building an AI agent uses one tool. Testing it uses another. Every iteration cycle ends with you switching to your test platform UI, running tests, inspecting results, then back to your editor to fix what broke. The context switch is small, but it adds up.

Today we shipped the fix: the Rhesis Agent Skill, day 1 of Rhesis Launch Week.

If you build LLM agents and use Claude Code, Cursor, Codex, Gemini CLI, or any of 40+ other AI tools, you can now drive the full Rhesis testing workflow from inside the chat where you write the code.

What it does

The Agent Skill packages our domain knowledge into a portable skill file that any compatible AI tool can load. Once installed, your AI assistant gains:

Endpoint discovery in Quick or Comprehensive mode
Test suite design with behaviors, test sets, and metrics
Confirmation guards that wait for approval before anything is created
Test execution against your endpoints
Failure analysis with pass/fail summaries and links back to runs

All powered by the Rhesis MCP server (27 tools covering test sets, behaviors, metrics, runs, and OData queries), all in natural language.

Install

Single command across all your AI tools:

npx skills add rhesis-ai/rhesis -g

The CLI detects which AI tools you have installed and configures the skill for each one. Then set your API token:

export RHESIS_API_KEY=rhs_your_token_here

Get a token at app.rhesis.ai/tokens.

Claude Code

Claude Code uses a plugin system that bundles the skill and MCP server config together:

/plugin marketplace add rhesis-ai/rhesis
/plugin install rhesis@rhesis-ai

Cursor

Add the MCP server to your .cursor/mcp.json:

{
  "mcpServers": {
    "rhesis": {
      "url": "<https://api.rhesis.ai/mcp>",
      "headers": {
        "Authorization": "Bearer YOUR_RHESIS_API_KEY"
      }
    }
  }
}

For self-hosted backends, swap https://api.rhesis.ai/mcp for http://localhost:8080/mcp.

Use it

Type something like:

"Test my support agent on billing scenarios, run it, and rank the failures by severity."

The skill walks the conversation through a 6-step loop:

Discover: explores what your endpoint can do
Plan: proposes a test suite with behaviors and metrics
Review: waits for your approval before creating anything
Create: builds entities on the platform following the approved plan
Execute: runs tests once you confirm
Analyze: surfaces a pass/fail summary, failure patterns, and links back to results

For ad-hoc operations:

"List my existing test sets."
"Improve the Safety Compliance metric. Make the threshold stricter."
"Compare my last two test runs."

The host agent's native confirmation handles the safety guard, so destructive actions never happen without your sign-off.

Issues in conversational AI apps are so obvious right now that even John Oliver felt the need to dedicate a whole episode to the topic: https://www.youtube.com/watch?v=Ykvf3MunGf8 #AISafety #AIEvals #Testing #QA

Nicolai Bohn — Tue, 28 Apr 2026 09:24:37 +0000

youtube.com