Part 3 of 3 — building and testing MCP agents
If you've done any performance testing, you know the JMeter mindset: "is my system fast enough" is a question you answer by measuring, not guessing. Define a test plan, throw load at the system, read off the numbers, and now you know whether a change made things better or worse — not "it felt faster."
MCP agents don't have an equivalent, and they badly need one. Most agent development I've seen — including my own, until recently — looks like: tweak the system prompt, run it a few times by hand, eyeball whether the answers seem better, ship it. That's not measurement. That's vibes with extra steps.
What I built to fix that
MAVERIK, the MCP host from the earlier parts of this series, has grown a built-in testing framework that treats agent definitions the way JMeter treats endpoints: something you define once, run systematically, and compare on hard numbers instead of impressions.
The mapping is almost one-to-one:
| JMeter | MAVERIK |
|---|---|
| Test plan | Test suite — a set of questions, each with a pass criterion |
| Sampler firing one request | A question firing one prompt at an agent |
| Assertion on the response | A criterion: exact match, substring, regex, or an LLM judge with a rubric |
| Thread group running N times | Repetitions — because LLM answers aren't deterministic, and one lucky run tells you nothing |
One detail on judge criteria worth calling out: the judge is a second model, so it burns tokens of its own. MAVERIK tracks the judge's cost separately, so it never pollutes the consumed budget of the agent being tested. If you don't do that, every rubric-scored suite silently makes your agents look more expensive than they are.
Point a suite at a set of agent definitions and MAVERIK fires every question at every one of them, spinning up a fresh agent instance per case, and records what Part 2 called the outcome parameters: whether the instance passed, how long it took (the whole turn — LLM and tool time both), how many input and output tokens it burned across every round trip, and how many tool calls it made — which tools, specifically, since some tools are far more expensive to call than others.
The two-parameter split, operationalized
This is where the framing from Part 2 actually earns its keep: you tune the definition, and you measure the instances. Before you run anything, you write the outcome parameters down as criteria — what counts as correct, what rubric a judge should apply. That's the task contract made executable. Then you sweep the tunable parameters, one definition field at a time: a different system context, a cheaper model, a parallel-tool-calling strategy instead of sequential, a trimmed MCP server set. MAVERIK tells you, in the units you defined up front, whether the change helped or just moved the cost somewhere else.
That's the actual payoff. Not "did the agent answer the question" in isolation, but "does the v2 system context answer just as well for a third of the tokens" or "is the cheap model actually good enough for this class of question, or does it only look that way on the easy cases."
If that sounds abstract, I published a worked example last week: two agent definitions differing only in their MCP server set, one question, and a 28% cost difference that had nothing to do with answer quality — the whole experiment was a config duplication and a one-question suite. That post is this loop, run once by hand. This part is about the loop itself.
Built the hard way, on purpose
The reason any of this is possible is the design choice from Part 1: the tool-calling loop is driven by hand, not by an SDK's auto-invocation mode. Every tool call is visible, every round trip is countable, every token is attributable — and critically, the exact same loop code runs whether you're chatting with an agent interactively or running it through a full benchmark suite. Test results only mean something if they predict real behavior, and that's only true if the test path and the production path are the same code, not a simulation of it.
There's more to build: cost prediction that accounts for which tools are expensive to call, and parameter sweeps that generate a matrix of definitions from ranges instead of hand-authoring each one. But the core loop already works: define what "good" means, run it against real agent definitions, and let the numbers do the arguing instead of intuition.
The next step after that is the obvious one for anyone with a CI/CD reflex: if a suite can tell you whether a change helped, it can also tell you whether a change broke something — which means it belongs in the pipeline, running on every commit, not just when someone remembers to check. That's the next article.
If you're building on MCP and you've ever shipped a prompt change on vibes, I'd love to hear how you're currently deciding whether it actually helped. The project is open source — https://github.com/langens-jonathan/maverik.
Top comments (0)