DEV Community: Nicolai Bohn

Auditable, observable, budgetable: what it takes to run AI agents in production

Nicolai Bohn — Thu, 22 Oct 2026 16:34:05 +0000

At GenAI Berlin I opened my talk with two questions for everyone building an agent:

Is your agent ready for production, and for real customers?

Did it improve over the last two weeks, or did it only change?

The second one is the sharper question. Most teams change prompts, models and tools every week. Very few can say whether those changes made the agent better. This post is the written version of the talk: why agents get pulled back, how they fail, and the three properties that let you answer both questions with evidence.

At GenAI Berlin 2026. Photos: GenAI Berlin

Why agents get pulled back

In May 2026, Gartner predicted that by 2027, 40 % of enterprises will scale back or shut down autonomous AI agents, because governance gaps only show up after production incidents. A year earlier, Gartner had already forecast that over 40 % of agentic AI projects would be canceled by 2027 and named three reasons:

escalating costs,
unclear business value,
inadequate risk controls.

Keep those three in mind. Each one gets an answer by the end of this post.

An agent is a model inside a harness

Everything in the blue box belongs to the harness: the orchestration loop that plans, calls tools and checks results, the model, the instructions, tools and MCP servers, memory and retrieved context, sub-agents, and guardrails with approvals.

Every part can fail in its own way. The model gives a confident, wrong answer. A tool fails silently. Retrieval returns the wrong policy. A handoff between agents gets lost. An approval is skipped. The user only sees the response at the end, and most of these failures never show up there.

Three ways agents fail in production

To make this concrete, picture an insurance assistant that answers coverage questions and files claims. The scenarios below are illustrative, built from patterns we see in real projects.

1. The requirement nobody checked

A customer asks whether a burst pipe is covered. The loop asks the model what to do, the model asks for the policy terms, and the search returns the Premium terms. The loop never calls the CRM tool to check which tier this customer is on. The model writes a fluent answer from the wrong terms: "Yes, up to €50,000." The customer is on Basic, capped at €5,000.

No tool failed. The requirement, answer from the customer's own tier, was simply never checked.

2. A timeout that turns into a confirmation

This time the first half goes right: the loop checks the policy and gets the correct tier. Then the claims tool times out. The loop hands the timeout to the model as if nothing happened, and the model writes a confirmation with a reference number it made up. The customer waits for a payout that will never come.

One rule in the harness would have stopped it: no confirmation without a claim ID from the claims system.

3. A correct answer at the wrong price

The answer is correct, so every quality test passes. But the assistant reads the policy clause by clause: around 40 rounds for one question, with every result appended to the context, so each round costs more than the last. One filtered search would have done the job. Only the cost per request and the trace reveal the problem, and nothing in the harness set a step limit or a budget.

This is already happening

These failure modes are not hypothetical. In July 2025, an AI coding agent at Replit deleted a production database during a code freeze and first claimed a rollback was impossible. A widely shared story describes two agents looping for 11 days and spending about $47,000. No named company or post-mortem has been published for that one, but the pattern is real.

The bigger picture points the same way. The OECD AI Incidents and Hazards Monitor counted 105 reported incidents and hazards in November 2022, the month ChatGPT launched, and 702 in September 2026. To be fair to the data: relative to all AI news, the incident rate is fairly stable. More AI in production simply means more incidents in absolute terms.

From failure modes to properties

At GenAI Berlin 2026, Humboldt Carré. Photo: GenAI Berlin

Each of Gartner's three reasons has an answer, and each answer is a property your agent needs before it goes to real customers.

Auditable: a paper trail from requirement to release

Requirements come from several stakeholders: the product team, domain experts such as underwriters or claims handlers, regulators and customers. They are written down and versioned. Test sets are linked to each requirement, every agent version runs against them, and the release decision is made on pass rates, with a sign-off.

Every step lands in an audit log: who tested which version, with what result, and who approved the release. That log is your evidence for the release meeting, for an auditor, or for a court. It also answers the second question from the opening, because you can compare pass rates per requirement across versions.

Observable: see what happens under the hood

A passing test checks the output. The path the agent took to get there stays hidden. In the claims example, the final message sounded perfectly fine; only the trace shows the timeout.

Observability for agents means:

tracing every step, tool call and handoff, ideally with OpenTelemetry as the open standard,
evaluating the trace, beyond the final answer,
making agent-to-agent messages visible, so you can see whether handoffs worked,
alerting when behavior drifts from what was tested, and catching loops and retry storms early.

Budgetable: know what it costs, and cut it safely

Before release, set a budget per task and experiment with the model, the prompt, step limits and context size, while rerunning the same tests so quality holds. In production, track cost per agent, tool and user flow, set hard caps per session, and feed the cost drivers back into your experiments.

The $47,000 loop is the counterexample: the team could see the bill, but had nothing in place to stop it. And the cost data comes from the same traces you built for observability.

The insurance assistant, fixed

Figures are illustrative.

Here is the same assistant with all three properties in place. Three requirements:

R1: answer from the customer's own tier,
R2: confirm a claim only with an ID,
R3: stay under €0.10 per request.

The loop looks up the tier first, so the coverage answer matches. The claim is confirmed only after the claims tool returns an ID. And each requirement is checked on the trace, lined up with the step it verifies. The whole conversation stays within budget: four tool calls, seven cents, and the trace stored in the audit log.

Why the three only work together

Tests against requirements decide the release. Traces from production show where behavior drifts and where the money goes. And every cost optimization, a smaller model or a shorter prompt, goes back through the tests before it ships.

We're not the only ones arguing for this loop. Anthropic's engineering guide Demystifying evals for AI agents (January 2026) makes the same point: offline evals need to be combined with production monitoring, A/B testing and human review, so problems don't only surface in production.

Back to the two questions

Why projects fail	The answer	The evidence you can show
Unclear business value	Auditable	Pass rates per requirement and version, with a sign-off
Inadequate risk controls	Observable	Every step, tool call and handoff in the trace
Escalating costs	Budgetable	Cost per request, and a cheaper setup that still passes

Is your agent ready for release? Is it better than two weeks ago? With these three in place, you can answer both, with evidence instead of a gut feeling.

How Rhesis helps

Rhesis is an open-source platform for exactly this, self-hosted or on Rhesis Cloud.

Property	In Rhesis
Auditable	Test sets generated from requirements, pass rates per version, sign-off, audit log on Enterprise
Observable	OpenTelemetry tracing across agents and handoffs, expert reviews on traces
Budgetable	Experiments against a baseline, with model overrides per run

If you want to see how this works with a real multi-agent example, read the companion post Test-driven development for AI agents: from red-green to production, or try Rhesis on GitHub.

Thanks to the GenAI Berlin team for having me, and to everyone who stayed for the questions afterwards.

Originally published on rhesis.ai.

Human vs. automated evals: who should judge your AI agent?

Nicolai Bohn — Thu, 15 Oct 2026 13:19:07 +0000

Once the first prototype works, someone asks who is going to check all of this. The honest answer sounds like a lot of manual work: deciding what good means, pulling in domain experts, labeling outputs, reading failures, and doing it all again after every change. Many teams hope the AI will take that off their plate. Mostly it won't, and teams that skip the work end up shipping an agent nobody can vouch for.

Netflix is a useful reality check. Their team runs an LLM judge over hundreds of thousands of recommendation explanations a week, and it still started with people (Kong et al., 2026). Experts defined the criteria, wrote the labeling guidelines and handcrafted hard examples. Before launch, the benchmark held roughly 900 human-labeled explanations. After launch, about 300 freshly rated ones are added every week to catch the judge drifting.

The manual part doesn't go away. It moves from writing test cases to reviewing verdicts, and that is where you can take the volume off your experts.

The two approaches are good at different things. A domain expert can tell you an answer is wrong for your business. Automation can run thousands of conversations overnight. This post puts them side by side, shows where each one breaks, how they feed each other, and then walks through the loop in Rhesis.

The case for human evals

The context only lives in your organization. Generic metrics like helpfulness, toxicity or faithfulness tell you little about whether your travel agent got your fare rules right. Say it tells a customer they can cancel for free up to 24 hours before departure, and the policy says 48. The answer is fluent, polite and grounded in a fare document. It is also wrong, and the only people who know that sit in your operations team.

Experts are fast at spotting. Show someone from customer operations ten transcripts and they find the one with last season's baggage allowance within seconds. Hamel Husain recommends making a single principal domain expert the source of truth, with a binary pass or fail for every output and a written critique explaining why. The critique matters as much as the verdict, because it is what you later turn into a metric.

Criteria come from looking at outputs. You can't write the full rubric upfront. Shankar et al. call this criteria drift: people only work out their criteria while grading, and they keep refining them as they see more outputs. A rubric written before anyone read a transcript is a first guess.

Where it breaks: experts are slow, expensive and busy. Asking them to come up with 200 test cases by hand doesn't work, and asking them to re-read every conversation after each prompt change works even less.

Ye Yuan ran into exactly this building a customer service bot across travel, healthcare, education, legal services and more (Data Science x AI, 2026). Generic NLP metrics couldn't capture multi-turn quality, so the team built its own rubric, benchmarked against their best human sales and support conversations. The first bottleneck was annotation capacity. Training and calibrating annotators was slow and expensive, and the gold dataset stayed too small.

The case for automated evals

They can explore an agent without being told anything. Point an automated tester at an endpoint and it can probe the domain, map what the agent can do, and find where it refuses or redirects. No requirements document needed.

They run adversarial tests at scale. Jailbreaks, prompt injection and multi-turn pressure need volume and variation. Covering the OWASP Top 10 for LLMs and agentic applications by hand isn't realistic. Generating and running those attacks is exactly what automation is for.

They check requirements once you hand them over. Every run, every model swap, every prompt change, in CI. A human reviewer gets tired by the fortieth conversation. A metric doesn't.

Where it breaks: the devil lies in the details. LLM judges tend to be lenient, struggle with partially correct answers, and are sensitive to how the judging prompt is worded (Thakur et al., "Judging the Judges"). They miss the small, off-by-one-detail errors an expert catches immediately: 24 hours instead of 48. And agreement rates hide the problem. If only 5% of outputs fail, a judge that passes everything agrees with your expert 95% of the time and catches none of the failures.

Ye Yuan saw the same in production. LLM judges struggled with pragmatics, implied meaning and a user simply going quiet, with noticeable score variance between runs. Synthetic test conversations also missed the ambiguity and the emotional and topical shifts of real user traffic.

How the two feed each other

Experts are good at recognizing problems and bad at producing volume. Automation is the reverse. So the useful question isn't which one to pick, it's which part of the work goes where.

The expert's job shifts from writing test cases to reviewing and correcting verdicts. Reviewing a pass or fail with one sentence of explanation takes seconds. Writing a good test case from scratch takes minutes, and writing 200 of them doesn't happen.

Ye Yuan's team landed on a similar loop: automated evals at scale, human spot checks, manual review of every low-scoring dialogue, and failure patterns fed back into the next iteration. In practice it looks like this:

Automation explores the agent and generates a first test set.
An expert reviews a sample and leaves a pass or fail plus a comment.
Where the expert and the judge disagree, those cases are used to tune the metric.
The tuned metric runs automatically on every change.
The expert only looks at conflicts and new kinds of failure.

You need less human input than it sounds. Anthropic suggests starting with 20 to 50 tasks drawn from real failures. Hamel Husain starts with around 30 examples and keeps going until no new failure modes show up, then aims for about 100 labeled examples per failure mode to check a judge against the expert. Netflix keeps adding around 300 ratings a week because live traffic keeps changing. That is a few hours of expert time, not a quarter of it.

Automation runs the tests, experts make the call

In the video, I point Rhesis at a travel agent with no requirements, let it explore and test the agent, and then review the results as the domain expert.

The Architect explores the endpoint. In quick or comprehensive mode, it probes the domain, maps capabilities and finds where the agent refuses or redirects, starting from zero. The exploration docs cover both modes.
The Architect plans and runs a suite. From what it found, it proposes requirements, metrics and test sets, and waits for approval before creating anything. I add one requirement myself: cancellation terms must match the fare rules. In the run, the cancellation test passes.
The expert reviews the results. Annotations can sit on a whole test result, a single metric or a single turn of a conversation, each with a pass or fail and a comment. I fail the cancellation metric and write why: the fare rules say 48 hours.
The human verdict wins. The expert's verdict now counts as the result. The automated one stays on record, and Rhesis flags the disagreement as a status conflict, so the team sees where the judge and the expert part ways. When they agree, the result shows Confirmed by with the reviewer's name, so over time you see which metrics people trust and which need work.

All annotations collect in the Annotations hub under Improve, where you can filter by conflicts and fix the metrics that keep getting it wrong. Engineers can also read and write annotations through the SDK and MCP, so expert verdicts end up in the same place as CI results.

Let experts judge, let automation do the volume

Neither approach is enough on its own. Automated evals without expert input pass fluent answers that are wrong for your business. Expert review without automation covers a few dozen conversations and stops. Put the expert where judgment is needed, and let automation handle the exploring, attacking and re-checking.

Rhesis is open source. Try it on app.rhesis.ai or self-host it from GitHub.

What are Jev-based evals?

Nicolai Bohn — Wed, 14 Oct 2026 19:14:40 +0000

There’s a lot of talk about Jev right now. The idea is simple. An LLM judge writes out a verdict and the reasoning behind it. Jev skips the writing. You give it a fixed set of labels, it picks one and tells you how sure it is. That makes it fast, cheap and easy to plug into a pipeline.

We support Jev in Rhesis as of v0.18. You can pick it as the evaluation model on any categorical metric, like tone, intent or policy compliance. Your LLM judges keep handling the open-ended checks where the reasoning matters.

Agent cost optimization: same quality, lower price

Nicolai Bohn — Wed, 07 Oct 2026 12:27:17 +0000

Most teams pick their agent's model while prototyping. It works, it ships, and nobody touches it again. Meanwhile cheaper models come out every few weeks.

Switching sounds easy until someone asks whether the new model will handle your refund policy questions as well as the old one. Nobody knows. There is no test set to run it against, so the team stays on the old model, and the bill stays where it is.

Breaking that loop takes two things: knowing what a single run of your agent costs, and a safe way to compare one setup against another.

Where the money goes

Cost is tokens times price, summed over every LLM call your agent makes. Four levers decide how big that number gets.

Model. Price per token can differ by an order of magnitude between model tiers. For most agents this is the biggest lever. Test it by swapping the model.

Prompt and context. The system prompt and any retrieved context are sent again on every call. A 3,000-token system prompt in a ten-turn conversation is 30,000 input tokens before the user has said much at all. Test it by shortening the prompt.

Agent steps. Every tool call or reasoning step is another LLM call, and conversations carry their history forward. An agent that takes six steps where two would do pays for it on every request. Test it by capping the steps.

Caching. Many providers bill cached input tokens at a lower rate. Keeping the start of your prompt stable lets you benefit from that. Test it by keeping prefixes stable and comparing.

Cheaper only counts if quality holds

A cost number on its own tells you very little. If v2 costs half as much but fails a third of your cancellation tests, you haven't saved anything. You've moved the cost to your support team.

So look at pass rate and cost together, on the same test set.

That gives you four outcomes:

Cheaper, same or better quality: promote it.
More expensive, better quality: decide whether the gain matters to your users.
Cheaper, lower quality: check which behaviors failed. A prompt tweak may win the quality back.
More expensive, lower quality: roll it back.

Change one thing, compare side by side

The method is plain experimentation. Change one variable at a time so you know what caused the difference.

Save a baseline. v1 is your current setup: model, prompt and temperature.
Vary one parameter. v2 swaps the model. v3 keeps the model and trims the prompt.
Run the same tests against each version, with the same metrics.
Compare the runs on pass rate and cost.

For a support agent this could look like:

Version	Model	System prompt	What you're testing
v1	your current model	full prompt	baseline
v2	a smaller model	full prompt	does the cheaper model hold up?
v3	your current model	trimmed prompt	how much does the prompt cost you?

How this works in Rhesis

Rhesis is an open-source platform for testing LLM apps and agents. We recently added cost tracking to test runs, and it pairs well with Experiments:

Every test run shows its tokens, cost and model in the runs list. You can sort by cost to find your most expensive runs.
Each run has a Cost card with input and output tokens priced separately. It updates live while the run executes.

Experiments store configurations as versions. A parameter schema defines the slots (model, prompt, temperature), and every change becomes an immutable version, so a run against v2 is reproducible later.
Side-by-side comparison shows pass rate and cost with deltas against the baseline. The cost delta turns green when the new run was cheaper.

Costs are calculated from the tokens each LLM call reports, multiplied by list prices from LiteLLM's model database. That makes them a good relative comparison. They don't include discounts or committed-use pricing, and models missing from the database show tokens without a cost. Details are in the token costs docs and the experiments docs.

Here's the walkthrough:

Try it

Rhesis is MIT-licensed. You can use it at app.rhesis.ai or self-host it from GitHub.

If you're running cost experiments on your own agents, I'd like to hear which lever saved you the most. Let me know in the comments.

Originally published on the Rhesis blog.

What are Jev evals? How they work and how they compare to LLM-as-a-judge

Nicolai Bohn — Tue, 06 Oct 2026 13:14:40 +0000

Evaluating agents with Jev has become one of the most discussed ideas in AI testing over the past few weeks. The appeal is easy to see: a verdict in milliseconds, for a fraction of what an LLM judge costs. Rhesis now supports Jev as a model provider for evaluation, so you can put it behind your categorical metrics today.

This post explains what Jev is and how it compares to an LLM judge. There's a demo video at the end.

What Jev is

Jev is a "System One" model from TypeSafe AI, released in September 2026. The name borrows from Kahneman's fast, intuitive System 1. Jev makes decisions and never generates text. You get no prose out of it, only typed values with probabilities attached.

A call has two parts. The state is whatever you want judged, as plain text or JSON. Alongside it you send a set of typed questions, and Jev answers all of them together in one parallel pass.

Take a support ticket that reads "I was charged twice for order A-104. Please refund the duplicate." You could ask Jev three questions about it at once:

Choice, department: billing, technical or sales? Jev picks one of up to 255 options and returns a probability for each.
Score, frustration: where does the customer sit on a scale from calm to very angry? A scale can have between two and ten levels.
Noul, refund_requested: is a refund being asked for? This is a yes/no question, answered as a probability.

The System One concept page in TypeSafe's docs explains these three primitives in more detail.

Pricing is $0.042 per million input tokens, and output is free. According to TypeSafe, a call takes 70 to 500 ms end to end. For a hands-on walkthrough with code, the practical guide from Valyu on DEV is the best place to start.

How it differs from an LLM judge

An LLM judge reads a rubric written in plain language. It returns a score or a category, usually with written reasoning that explains the verdict. That flexibility is why it works for any kind of metric. It also has a cost. Each call takes seconds, and you pay for every output token, reasoning included.

Jev takes a question with fixed answer options. It returns one of those options with a probability for each, and nothing else. Since there is no explanation to fall back on, the option names have to carry the meaning on their own. The price difference is largest where inputs are long. Output is free, so a long conversation costs you only its input tokens. The MindStudio post on Jev pricing compares this with using a general-purpose LLM for the same kind of decision, and shows the gap growing as inputs get longer.

Jev has real limits, and you should know them before you rely on it. It reads negations literally, so "the agent did not avoid the topic" lands at face value. It can't count or do math. It gives you no written rationale, which makes a surprising verdict harder to debug. The failure modes section of the Valyu guide covers these with examples. On the plus side, its probabilities are calibrated, so a high probability really does mean a more reliable answer. DataCamp's explainer describes how TypeSafe trains for that.

My rule of thumb: use Jev when the verdict is one of a few known outcomes, and use an LLM judge when you need a numeric score or an explanation.

Jev in Rhesis

Jev is a new model provider in Rhesis. It does not add a new metric type. It is evaluation-only and works only with categorical metrics, the ones built on CategoricalJudge. Rhesis won't let you pick it for test generation or test runs, and numeric metrics refuse it as well.

When a categorical metric runs on Jev, Rhesis turns it into a single Choice question. The test input and the agent's reply become the state. The metric's prompt and evaluation steps become the instructions. Its categories become the answer options. Jev picks one category and returns probabilities for all of them, and Rhesis maps that pick to pass or fail based on the metric's passing categories. The result shows the chosen category and its probability. There is no reasoning text.

Two habits make Jev metrics work better. Name each category so it describes its outcome without context, for example recommended_partner_airline rather than ok. And phrase instructions positively, so the meaning doesn't hinge on a "not".

Setup takes a minute:

Open the Models page and choose the Jev tile.
Paste your TypeSafe API key.
Click Test Connection to confirm Rhesis can reach Jev.
Pick Jev as the model for a categorical metric, or make it your default model for evaluation.

The decision models section of the Rhesis docs covers the details, and the metrics docs explain how categorical metrics are set up. If you want to see how the integration is built, it landed in PR #2840.

See it in action

In the video, I test a travel agent that should only recommend partner airlines. For every reply, Rhesis asks Jev one Choice question, and Jev decides whether the agent kept to that rule. You'll see the categorical metric, the replies being judged one by one, and the category Jev picked for each test along with its probabilities.

Rhesis is open source. You can try it on app.rhesis.ai or self-host it from GitHub.

Originally published on the Rhesis blog.

Have you tried a decision model like Jev for your evals yet, or are you sticking with LLM judges? I'd like to hear in the comments.

Where AI agent test cases come from: synthetic vs. production data

Nicolai Bohn — Mon, 05 Oct 2026 14:56:22 +0000

Before you can evaluate an agent, you need something to evaluate it against. Which test cases you can get depends mostly on timing. Before launch, everything you test with is something your team came up with. After launch, your users start writing test cases for you, whether you collect them or not.

This post compares the main sources and shows how the mix changes as your agent matures. There's a walkthrough video at the end.

Two sources: synthetic and production data

Synthetic test cases are the ones you create yourself. You can write them while exploring the agent in a playground, or generate them from a description of what you want to test. They are available from day one, and they are how you cover reliability (does the agent meet your requirements?) and robustness (does it hold up against jailbreaks, prompt injection and other adversarial inputs?).

Tests from production data come from conversations real users had with your agent. They include the edge cases nobody on the team thought to write down. You only get them once you are live.

What each method is good for

Playground

The fastest way to start. You chat with your agent, and when it gets something wrong, you save that exchange as a test case. Say your claims assistant tells a customer they have 30 days to file when your policy says 14. That conversation becomes a regression test in a few clicks.

The limit is you. Coverage only reaches as far as the scenarios you think of, and recording cases one at a time takes a while.

Synthesizer

Here you describe what you want to test, for example "customers asking about cancellation deadlines, including attempts to get the agent to promise refunds it can't give". You rate a few samples so the generator learns what you mean, and you get back a full test set. The same works for multi-turn simulations, where a simulated user pushes the agent over several turns instead of a single prompt.

You get broad coverage of your requirements and adversarial inputs before a single user has touched the agent. You still have to define the scope and review samples, though. And generated inputs tend to read cleaner than real ones. Real users make typos and ask several things in one message.

Real conversations

Nothing is more realistic than what your users actually typed. Production conversations show you real phrasing and real edge cases, and they tell you what people want from the agent, which is not always what the spec assumed.

They have two catches. You need to be live, and you usually want to find problems before your users do. And real conversations contain personal data, so they need cleanup before they become test cases. For teams in Europe that is a GDPR question as much as a technical one.

Working from a coding agent instead? You don't have to go through a UI for any of this. The Rhesis MCP server exposes the synthesizer and your production traces to Claude Code, Cursor and other MCP clients. Paste a product spec and the agent proposes requirements, metrics and test sets, creating them once you approve. Or ask it to read recent traces and turn the conversations worth keeping into test cases.

Your test sources shift as the agent matures

In practice, most teams end up using all three, in roughly this order:

Exploring by hand. In the first iterations, domain experts use the playground and record what they see. This is where your first expectations come from.
Generating at scale. Once requirements settle, the synthesizer and simulations carry most of the volume, covering requirements and adversarial inputs systematically.
Learning from production. After launch, real conversations show you what the synthetic sets missed. A failure from production also makes a good seed for new synthetic variations.
Automated where possible. Routine checks run automatically. Experts stay in the loop on the hard, novel cases.

See it in action

In the video, I go through each method in Rhesis, from the first case recorded in the playground to test cases built from production conversations.

Rhesis is open source. You can try it on app.rhesis.ai or self-host it from GitHub.

Originally published on the Rhesis blog.

How do you source test cases for your agents? Mostly synthetic, mostly production, or something else entirely? I'd like to hear in the comments.

Launch week day 1: drive the full AI testing workflow from inside any AI tool

Nicolai Bohn — Mon, 04 May 2026 17:17:03 +0000

Building an AI agent uses one tool. Testing it uses another. Every iteration cycle ends with you switching to your test platform UI, running tests, inspecting results, then back to your editor to fix what broke. The context switch is small, but it adds up.

Today we shipped the fix: the Rhesis Agent Skill, day 1 of Rhesis Launch Week.

If you build LLM agents and use Claude Code, Cursor, Codex, Gemini CLI, or any of 40+ other AI tools, you can now drive the full Rhesis testing workflow from inside the chat where you write the code.

What it does

The Agent Skill packages our domain knowledge into a portable skill file that any compatible AI tool can load. Once installed, your AI assistant gains:

Endpoint discovery in Quick or Comprehensive mode
Test suite design with behaviors, test sets, and metrics
Confirmation guards that wait for approval before anything is created
Test execution against your endpoints
Failure analysis with pass/fail summaries and links back to runs

All powered by the Rhesis MCP server (27 tools covering test sets, behaviors, metrics, runs, and OData queries), all in natural language.

Install

Single command across all your AI tools:

npx skills add rhesis-ai/rhesis -g

The CLI detects which AI tools you have installed and configures the skill for each one. Then set your API token:

export RHESIS_API_KEY=rhs_your_token_here

Get a token at app.rhesis.ai/tokens.

Claude Code

Claude Code uses a plugin system that bundles the skill and MCP server config together:

/plugin marketplace add rhesis-ai/rhesis
/plugin install rhesis@rhesis-ai

Cursor

Add the MCP server to your .cursor/mcp.json:

{
  "mcpServers": {
    "rhesis": {
      "url": "<https://api.rhesis.ai/mcp>",
      "headers": {
        "Authorization": "Bearer YOUR_RHESIS_API_KEY"
      }
    }
  }
}

For self-hosted backends, swap https://api.rhesis.ai/mcp for http://localhost:8080/mcp.

Use it

Type something like:

"Test my support agent on billing scenarios, run it, and rank the failures by severity."

The skill walks the conversation through a 6-step loop:

Discover: explores what your endpoint can do
Plan: proposes a test suite with behaviors and metrics
Review: waits for your approval before creating anything
Create: builds entities on the platform following the approved plan
Execute: runs tests once you confirm
Analyze: surfaces a pass/fail summary, failure patterns, and links back to results

For ad-hoc operations:

"List my existing test sets."
"Improve the Safety Compliance metric. Make the threshold stricter."
"Compare my last two test runs."

The host agent's native confirmation handles the safety guard, so destructive actions never happen without your sign-off.

Issues in conversational AI apps are so obvious right now that even John Oliver felt the need to dedicate a whole episode to the topic: https://www.youtube.com/watch?v=Ykvf3MunGf8 #AISafety #AIEvals #Testing #QA

Nicolai Bohn — Tue, 28 Apr 2026 09:24:37 +0000

youtube.com