Emanuele de Rossi
July 27, 2026 • 9 min read
If you are developing an LLM application (be it a chatbot, an agent, or something else), you probably already have some way of watching it: an observability tool, a logging setup, something that shows you what's happening inside. This is a one-way connection: your process emits data (a completed request, a token count, a latency number) and a collector on the other end ingests it. The traffic only ever moves in one direction, out of your app.
It's cheap to bolt on precisely because of that directionality: your app doesn't need a listener, an open port, or a request handler for anything coming back. It just calls exporter.export(span) and moves on. You can layer online evals on top of that exhaust to score production traffic automatically.
However, if you want to catch problems before they reach a user, you need offline evals: something outside your application that can hand it a new prompt (a user journey you want covered, a regression test, an edge case, an adversarial probe) and grade the reply before any user is exposed to it. You need a two-way connection that can call on demand.
This post covers that difference: why testing requires a connection that can originate a request, how to wire one up using Rhesis, and what's happening underneath at the protocol level.
Setting it up takes a few minutes more than plugging in a logger, but once it exists everything downstream becomes automatic and repeatable, instead of manual copy-paste-and-eyeball forever.
Why you need a two-way connection
The connection you already have (the observability tool, the logging setup) is a one-way connection: data flows out of your LLM application into a dashboard. It's easy to set up, because it's outgoing traffic only: your application just emits logs, it doesn't need to accept or respond to anything coming in. That asymmetry is what makes it cheap: the collector can go down, be slow, be wrong, and your app never notices.
A two-way connection breaks that asymmetry on purpose. Concretely, that means one of two things: your app exposes a synchronous request/response cycle (an HTTP endpoint that accepts a call and returns a result before closing it out), or it opens and holds a connection itself so a remote caller can reach in on demand (a persistent socket). Either way, you've added a listener, a contract for what a valid inbound request looks like, and (if the caller shouldn't be anonymous) some form of auth on the way in. That's real surface area you didn't have before.
Testing means picking a prompt (single-turn) or a conversation (multi-turn) to simulate, sending it to your app, getting the reply back, and grading it, automatically, hundreds of times over. That requires something that can call your app and wait for the answer. In other words: a two-way connection.
Introducing Rhesis: how we do two-way connections
Rhesis is the collaboration layer where domain experts and engineers work together to generate test cases, run them against your application, and evaluate the results. The connection point between your LLM application and Rhesis is called an Endpoint.
Rhesis works like an orchestrator. In single-turn testing, it sends a test prompt to your application, waits for the reply, and records it for evaluation. In multi-turn testing, it drives a full conversation instead, sending follow-ups based on what your application just said, so that the whole exchange gets evaluated.
There are two ways to build an Endpoint: REST Endpoints, set up entirely through the UI with no code, or the Python SDK, for LLM applications that run locally, sit behind a firewall, or where you want tighter control from your own codebase. The decision of what to pick mostly depends on whether your application is already running as its own reachable service, or whether it lives inside your own codebase.
1. REST Endpoints (via the UI)
If your application is already exposed as an HTTP API (a deployed service, a microservice, anything already running on its own) you can connect it straight from the Rhesis platform, no code required.
Go to the /endpoints page in Rhesis
Give it your URL
Rhesis calls into your application directly: it sends a request, your application responds, Rhesis reads the reply.
This is the right choice if your AI feature already runs as its own reachable service, and the easiest to set up. Full details: Endpoints guide.
2. Rhesis Connector (Python SDK)
If your LLM application runs locally, sits behind a firewall, or isn't cleanly exposed as its own service at all, use the Python SDK instead.
The @endpoint decorator does two things: it registers run_chatbot as something Rhesis can invoke remotely, and it opens a persistent WebSocket from your app out to Rhesis. The connection is outbound from your app, so it works through firewalls and from local laptops without exposing a public URL.
When a test run starts, Rhesis sends each test case's input down the WebSocket; your application runs the function locally and sends the output back up the same connection. Full details: SDK Connector docs.
The mapping: teaching Rhesis your app's shape
Whichever of the two you pick, there's one more thing to sort out: every AI application is shaped differently. Yours might take {"query": "..."} for the request and reply with {"data": {"answer": "..."}}. Someone else's might look different. So there needs to be a mapping layer that translates your app's shape (both request and response) into a small set of standard fields. This is the same idea whether you connected via REST or the SDK; only how you do the mapping changes.
On the way in, Rhesis needs to know where to place the input. On the way out, it needs to know where to find the reply.
On the way in, Rhesis needs to know where to place input, the prompt it's sending (plus, optionally, things like files or conversation_id if your application needs them). On the way out, it needs to know where to find output, the reply it should evaluate (plus, optionally, context or metadata).
Once that mapping exists, Rhesis can talk to your app automatically, for any number of tests.
Let's use one example throughout: a simple chatbot. For each message, it takes a query and returns an answer. The mapping for it looks like this:
That's it. {{ input }} drops in the test prompt, $.data.answer pulls the reply back out. Want the full mapping options (JSONPath, Jinja2, complex payloads)? See the Endpoints documentation.
Single-turn vs. multi-turn
That mapping above is enough for a single-turn Endpoint: one prompt in, one reply out, nothing to track between calls. But a lot of what you're testing isn't single-turn, it's a conversation. For that, Rhesis needs to track state across messages, and it handles two flavors of it: stateful, where your app tracks the session itself and Rhesis just passes a conversation_id back and forth (so that the multi-turn conversation gets tracked across calls), and stateless, where your app has no memory of its own and Rhesis resends the full message history with every turn.
What you unlock once you're connected
Once your endpoint exists, everything that follows builds on it. Roughly, it breaks down into three things: generating and running the tests themselves, understanding what happened once they run, and using what you learn to actually improve your LLM application's configuration.
Create and run tests
Once Rhesis can reach your application, the first thing you get is more ways to actually generate and run tests. Instead of writing prompts by hand, Rhesis can generate relevant test cases automatically, based on a custom description or a predefined template. The Playground lets you chat with any connected endpoint directly (no code, just a conversation window) attach files, run two endpoints side by side to compare configs, and save any conversation as a test case on the spot. From there, testing goes beyond a single reply: conversation simulation covers multi-turn dialogue, adversarial testing probes for jailbreaks and unwanted behavior, and multi-modal testing covers endpoints that take images, PDFs, or audio rather than just text.
Understand what happened
Once tests run, you need to know not just whether they passed, but why. Metrics score every response automatically against the behaviors you care about, using built-in judges like DeepEval and Ragas, or your own custom code metrics. Tracing goes one level deeper: every call to your endpoint is automatically captured as a full span tree (every LLM call, tool invocation, and retrieval step) so a failure can be traced to the exact step that caused it. Insights rolls all of this up into a dashboard, tracking pass rates and trends across runs over time.
Iterate on configuration
And because testing an LLM application usually means testing which configuration of it performs best, Experiments give you something like version control for your app's settings: every change becomes an immutable version, you promote the one you trust to an environment, and Rhesis shows you the pass-rate delta between versions. This way, a regression is traceable to the exact parameter that caused it.
The trade-off
A two-way connection is the same mechanism, however simple or demanding the test is. Whether Rhesis is sending one prompt and waiting for one reply, driving a full multi-turn conversation, running an adversarial probe, or sending an image or a PDF instead of text, it's still one thing calling your application and waiting for the answer. No separate wiring needed per test type, and the same connection that powers a single-turn test is what a conversation simulation or an adversarial run builds on too.
Setting it up takes a bit more than plugging in a logger: either an HTTP endpoint that can accept a call and answer it, or an outbound connection your app opens itself, plus a small mapping layer telling Rhesis where to find your app's input and its reply.
Frequently asked questions
REST Endpoint or Python SDK, which should I pick?
If your LLM application is already running as a reachable HTTP service, use a REST Endpoint, point-and-click in the UI, no code required. If it runs locally, sits behind a firewall, or you want tighter control from your own codebase, use the Python SDK instead. The @endpoint decorator opens an outbound connection from your app, so it works without exposing a public URL.
What do I unlock once the endpoint is connected?
Once Rhesis can reach your application, you can test and evaluate it: generate test sets, run them against your application, simulate conversations, run adversarial and multi-modal tests, and score every reply with metrics.
What is the mapping between my application and Rhesis?
Every application takes a different shape for its request and its reply, so Rhesis needs a small mapping layer to translate one into the other. On the way in, it needs to know where to place the input, the prompt it is sending. On the way out, it needs to know where to find the output, the reply it should evaluate. For a simple chatbot, that is often just {{ input }} for the request and $.data.answer for the response, and once it exists, Rhesis can talk to your application automatically for any number of tests.
I don't have an application to connect yet. Can I still try it?
Yes. Rhesis ships with a default chatbot already connected, so you can explore test generation, evaluation, and the rest of the platform before wiring up your own.
Top comments (0)