DEV Community: Jonathan Langens

MAVERIK: Why MCP Agents Need Their Own JMeter

Jonathan Langens — Sun, 02 Aug 2026 12:03:26 +0000

Part 3 of 3 — building and testing MCP agents

If you've done any performance testing, you know the JMeter mindset: "is my system fast enough" is a question you answer by measuring, not guessing. Define a test plan, throw load at the system, read off the numbers, and now you know whether a change made things better or worse — not "it felt faster."

MCP agents don't have an equivalent, and they badly need one. Most agent development I've seen — including my own, until recently — looks like: tweak the system prompt, run it a few times by hand, eyeball whether the answers seem better, ship it. That's not measurement. That's vibes with extra steps.

What I built to fix that

MAVERIK, the MCP host from the earlier parts of this series, has grown a built-in testing framework that treats agent definitions the way JMeter treats endpoints: something you define once, run systematically, and compare on hard numbers instead of impressions.

The mapping is almost one-to-one:

JMeter	MAVERIK
Test plan	Test suite — a set of questions, each with a pass criterion
Sampler firing one request	A question firing one prompt at an agent
Assertion on the response	A criterion: exact match, substring, regex, or an LLM judge with a rubric
Thread group running N times	Repetitions — because LLM answers aren't deterministic, and one lucky run tells you nothing

One detail on judge criteria worth calling out: the judge is a second model, so it burns tokens of its own. MAVERIK tracks the judge's cost separately, so it never pollutes the consumed budget of the agent being tested. If you don't do that, every rubric-scored suite silently makes your agents look more expensive than they are.

Point a suite at a set of agent definitions and MAVERIK fires every question at every one of them, spinning up a fresh agent instance per case, and records what Part 2 called the outcome parameters: whether the instance passed, how long it took (the whole turn — LLM and tool time both), how many input and output tokens it burned across every round trip, and how many tool calls it made — which tools, specifically, since some tools are far more expensive to call than others.

The two-parameter split, operationalized

This is where the framing from Part 2 actually earns its keep: you tune the definition, and you measure the instances. Before you run anything, you write the outcome parameters down as criteria — what counts as correct, what rubric a judge should apply. That's the task contract made executable. Then you sweep the tunable parameters, one definition field at a time: a different system context, a cheaper model, a parallel-tool-calling strategy instead of sequential, a trimmed MCP server set. MAVERIK tells you, in the units you defined up front, whether the change helped or just moved the cost somewhere else.

That's the actual payoff. Not "did the agent answer the question" in isolation, but "does the v2 system context answer just as well for a third of the tokens" or "is the cheap model actually good enough for this class of question, or does it only look that way on the easy cases."

If that sounds abstract, I published a worked example last week: two agent definitions differing only in their MCP server set, one question, and a 28% cost difference that had nothing to do with answer quality — the whole experiment was a config duplication and a one-question suite. That post is this loop, run once by hand. This part is about the loop itself.

Built the hard way, on purpose

The reason any of this is possible is the design choice from Part 1: the tool-calling loop is driven by hand, not by an SDK's auto-invocation mode. Every tool call is visible, every round trip is countable, every token is attributable — and critically, the exact same loop code runs whether you're chatting with an agent interactively or running it through a full benchmark suite. Test results only mean something if they predict real behavior, and that's only true if the test path and the production path are the same code, not a simulation of it.

There's more to build: cost prediction that accounts for which tools are expensive to call, and parameter sweeps that generate a matrix of definitions from ranges instead of hand-authoring each one. But the core loop already works: define what "good" means, run it against real agent definitions, and let the numbers do the arguing instead of intuition.

The next step after that is the obvious one for anyone with a CI/CD reflex: if a suite can tell you whether a change helped, it can also tell you whether a change broke something — which means it belongs in the pipeline, running on every commit, not just when someone remembers to check. That's the next article.

If you're building on MCP and you've ever shipped a prompt change on vibes, I'd love to hear how you're currently deciding whether it actually helped. The project is open source — https://github.com/langens-jonathan/maverik.

Do unused MCP tools cost you money?

Jonathan Langens — Thu, 30 Jul 2026 18:18:36 +0000

A short case study from my "building and testing MCP agents" series — it stands on its own, but the method behind it is laid out in https://dev.to/langensjonathan/the-parameters-that-actually-matter-when-youre-tuning-an-ai-agent-2agd.

TL;DR: I benchmarked two agents that are identical except for one thing — how many MCP servers they're connected to — on the exact same question. Both got the right answer, both called the same single tool. The one with more MCP servers attached still cost 28% more per question, purely from the extra tool schemas the model has to be told about on every single call, whether it uses them or not.

The setup

MAVERIK is my open-source MCP test bench: define a suite of questions with pass criteria, run it against one or more agent configurations, and compare the results on hard numbers. This post is one deliberately tiny experiment with it: change exactly one thing about an agent, hold everything else fixed, and see what the numbers attribute to that one change.

I have a small "GitHub summarizer" agent: one system prompt, one job — answer questions about my GitHub account by calling the GitHub MCP server. I duplicated its configuration (MAVERIK supports this directly — same model, same prompt, same everything) and changed one field on the copy: the set of attached MCP servers, adding deepwiki, microsoft-learn, and context7. Neither agent needs any of those three for the question I was about to ask; they were attached because that's what the "kitchen sink" version of this agent had accumulated over a few sessions of general-purpose use.

Then I wrote the simplest possible test suite — one question:

"How many repositories do I have?"

with a contains criterion checking the answer includes the correct count. No judge model, no subjectivity — it either says the right number or it doesn't. I ran both agents against it, 2 repetitions each, same model (claude-haiku) for both, and pulled up MAVERIK's Agent Comparison report.

	Agent A — `github` only	Agent B — `github` + 3 more MCP servers
MCP servers attached	1	4
Pass rate	100%	100%
Tool calls per question	1 (`get_me`)	1 (`get_me`)
Iterations per question	2	2
Avg input tokens	16,628	21,467
Avg output tokens	138	144.5
Avg peak context tokens	8,494	10,919.5
Avg duration	3,659 ms	3,061 ms
Token cost / question	$0.034636	$0.044379
Tool cost / question	$0.0000	$0.0000
Overall cost / question	$0.034636	$0.044379

(Raw CSV, straight out of the report's export button, is at the bottom of this post if you want to check my arithmetic.)

What actually moved

Three numbers moved together, consistently, across both repetitions:

Input tokens: +29.1% (16,628 → 21,467) — and it's exactly the same number on both repetitions for each agent. That's not noise; that's the tool catalog being serialized into every request, deterministically, regardless of which tools actually get called.
Peak context tokens: +28.6% (8,494 → 10,919.5) — same story, since peak context is dominated by that same input payload.
Cost per question: +28.1% ($0.034636 → $0.044379) — tracking the input-token increase almost 1:1, because output tokens barely moved (138 vs. 144.5) and no priced tools were called by either agent. The three unused MCP servers weren't free — they were billed on every single request, whether the model reached for them or not.

One caveat I want to make before someone else does: 28% is a function of how small this task is. The tax itself is absolute — ~4,839 extra input tokens on every request — so on a task with 50k tokens of real context the percentage would look much smaller. But that's the wrong comfort: the absolute tax is paid on every iteration of every run this agent ever makes, forever. Short tasks are where the tax is most visible, not where it's worst.

What didn't move — and why that's the more interesting part

Pass rate: identical (100%/100%). Extra tools didn't confuse the model into a worse answer.
Tool calls and iterations: identical (1 call, 2 iterations, same tool — get_me — both times). The model didn't waste a call poking at an irrelevant server "just in case." Having more options didn't make it indecisive here.
Duration: actually lower for the agent with more servers attached (3,061 ms vs. 3,659 ms) — the opposite of what the token numbers would predict. Looking at the individual repetitions, agent A's first call was a slow outlier (4,402 ms) that dragged its average up; its second call (2,916 ms) was actually faster than either of agent B's. With only 2 repetitions this is provider-side latency noise, not signal — which is exactly why MAVERIK has a repetitions knob in the first place. I'd want a much larger n before drawing any conclusion about latency here. Token counts, by contrast, needed zero repetitions to trust — they were bit-for-bit identical across both runs of the same agent, because they're a function of the prompt and tool schemas, not the network.

That asymmetry is the real lesson: cost is a much cleaner signal than latency. If I'd only looked at duration, I'd have concluded the bigger agent was better. It took breaking the run down by token count — something a stopwatch can't tell you — to see the actual, deterministic tax of the extra servers.

The takeaway

It's a reminder that "which MCP servers does this agent have access to" is a tunable parameter with a real, measurable cost, not a one-time architectural decision you make and forget. Every tool a server exposes gets its name, description, and JSON schema stuffed into the model's context on every call. A server with a large or verbose tool catalog is a standing tax on every request an agent makes, independent of whether that agent ever calls it.

There's a design consequence hiding in that, too. Restricting which tools an agent may use only saves you money if the host enforces the restriction before serialization: a disallowed tool has to be left out of the request entirely, not merely rejected when the model tries to call it. Enforce it at call time and you keep paying the token tax for tools the agent was never allowed to touch. The same trimming shrinks your actual attack surface, in the security sense — every unused tool schema in context is one more thing a prompt injection can try to steer the model toward.

The practical version of this: if you're building agents against MCP servers, don't default to "attach everything, it might be useful." Give each agent only the servers its job actually requires, and — if you're not sure whether a broader toolset is worth its overhead — measure it instead of guessing. That's a two-line duplication of an agent config and a one-question test suite, and the answer was sitting in a CSV five minutes later.

Reproducing this

This whole thing was: duplicate an agent config, write a one-question suite with a contains criterion, run both agents against it a couple of times, open the built-in Agent Comparison report (or export it to CSV, like I did above). If you're evaluating your own MCP servers or agent configs, the project is open source — https://github.com/langens-jonathan/maverik.

Raw CSV (Reporting → Agent Comparison → Export → To CSV)

suiteId,agentId,timestamp,sourceRunId,passRate,avgDurationMs,avgInputTokens,avgOutputTokens,avgToolCalls,avgPeakContextTokens,tokenCost,toolCost,overallCost
github-summarizer,github-test-agent,2026-07-26T10:07:18.9720785+00:00,github-summarizer-20260726-100718,1,3659,16628,138,1,8494,0.034636,0,0.034636
github-summarizer,github-test-agent-all-mcp-servers,2026-07-26T10:07:18.9720785+00:00,github-summarizer-20260726-100718,1,3061,21467,144.5,1,10919.5,0.044379,0,0.044379

The Parameters That Actually Matter When You're Tuning an AI Agent

Jonathan Langens — Thu, 30 Jul 2026 07:40:28 +0000

Part 2 of 3 — building and testing MCP agents

Every AI agent is a bundle of decisions, most of which get made once, informally, and never revisited: which model, what system prompt, which tools it's allowed to touch, how many steps it gets before you give up on it. Nobody writes these down as a config. They just happen, in code, and then everyone forgets they were choices at all.

In Part 1 I argued those decisions have a shape: an agent definition (the class — loop, system context, capability boundary, model, limits) and an agent instance (a running occurrence — goal, context window, credentials, consumed budget). That split turns out to be exactly what you need for tuning, because the two kinds of parameters people mix up when "tuning an agent" live on opposite sides of it: you tune the definition, and you measure the instances.

Tunable parameters: the fields of the agent definition

These are the levers — change any one and you have a different agent, even if everything else stays the same:

system context — how the agent is instructed
LLM — which model answers, and at what price
MCP server set and capability boundary — which tool sources exist, and which of their tools this agent may actually reach for
the host loop — sequential vs parallel tool execution, retry behavior
termination limits — how many iterations and tokens it gets before you cut it off
context-size strategy — how it manages a growing conversation history as a task gets longer

That's the agent definition from Part 1, field for field — with one deliberate absence. The task contract is the only definition field that isn't a lever. You don't tune it; you tune everything else against it. Hold that thought.

The consequence of seeing these as fields is simple but real: if you don't track which field you changed, you can't attribute why the results changed.

Outcome parameters: what you measure on the instances

This is the half people skip. An agent isn't "good" in the abstract — it's good or bad against criteria you define yourself, and every one of them is a measurement taken from agent instances after they run:

Does it reach an acceptable answer? Correctness isn't binary — sometimes it's "contains the right fact," sometimes it needs a rubric and a second model to judge it. Notice what this is: the task contract made testable. The definition says what the agent must return; the outcome criterion checks whether an instance actually did.
How long does it take? Wall-clock time, including every tool call, not just the model's "thinking."
How many tokens does it burn? Input and output, summed across every round trip, not just the final message. This is the instance's consumed budget, read back out.
Which tools does it reach for, and how often? Underrated. Some tools are cheap to call, others are expensive — rate-limited APIs, slow external services, tools with real dollar cost per call. An agent that gets the right answer by using an expensive tool once is worse than one that gets there in five cheap calls, even if both "pass."

The order matters more than the list

Here's the part that's easy to get backwards: you have to define the outcome parameters before you start sweeping the tunable ones. Decide what a correct answer looks like, what latency is acceptable, what token budget you're willing to spend, and which tools should be used sparingly — write it down as a criterion, a rubric, a budget. Only then start trying a tighter system context, a cheaper model, a different loop strategy.

Skip that step and "tuning" becomes vibes — you nudge a prompt, the answer looks a little better in the three examples you tried, you ship it, and you have no idea whether it's actually better or just different. Do it in the right order and every change becomes a testable hypothesis: does this specific change to one definition field move the numbers I said mattered, in the direction I wanted, without moving the ones I didn't touch?

Why this matters: an agent definition is data, not code

The reason this framing pays off in practice is that once the system context, model, capability boundary, and loop settings are configuration rather than something baked into code — an agents.json, in my host — comparing two agents stops being a rewrite and becomes a diff. Point the same set of test questions at two definitions that differ in exactly one field, run a batch of instances against each, and any difference in the measured outcomes is attributable to that one field.

That's the whole game: tune the class, measure the instances. And it's exactly the setup you need for what comes next — running these comparisons systematically instead of by hand.

The link to part 1 of this series: https://dev.to/langensjonathan/mcp-agents-explained-what-actually-makes-an-llm-an-agent-12a0

MCP Agents, Explained: What Actually Makes an LLM an "Agent"

Jonathan Langens — Wed, 29 Jul 2026 11:24:27 +0000

Part 1 of 3 — building and testing MCP agents

"Agent" has become one of those words that means everything and nothing. So let's ground it.

On its own, a large language model does one thing: it turns text into text. Ask it a question, it predicts an answer. It can't look anything up, check a fact against a live system, or take an action in the world. Everything it "knows" is frozen at training time.

An agent changes that by wiring the model into a loop with tools. Give it a set of functions it can call — read a file, hit an API, query a database — and a loop that feeds each tool's result back to it, and something different happens. The model can now decide it needs information, ask for a tool, read what comes back, decide whether that's enough or whether it needs another tool, and only then answer.

That loop — call a tool, or answer, repeat until done — is the core of every system people call an agent, dressed up in different frameworks. But to me it seems the loop alone is not the whole definition. A loop with no stop condition is a runaway process; a loop with no tool policy is a security incident waiting to happen. When I tried to pin down what I actually configure when I set up an agent, I ended up with two separate things.

Agent definition and agent instance

I split the concept in two parts, like a class and its instances. An agent definition is the template; an agent instance is one running occurrence of it. (I wrote a separate article going deeper into this definition and how the newest MCP spec stress-tests it — link at the bottom.)

The agent definition consists of:

the host loop — the core, as above
system context — persona, instructions. Static and authored.
MCP server set — which tool sources exist for this agent
capability boundary / tool policy — which servers and tools this agent may use, and what needs approval. This is a role, not a login.
LLM — the actual model we are using
termination limits — max iterations, token budget, stop conditions
task contract — what this agent accepts and what it must return
context-size strategy — how it keeps the conversation inside the model's window

An agent instance is what you get when a definition picks up real work:

a goal — what the user actually asked
a working context window — conversation history, tool results
credentials — whose tokens, on whose behalf
consumed budget — iterations elapsed, tokens spent

Note the symmetry: the definition holds limits, contracts and capabilities; the instance holds counters, goals and credentials. Static in the class, dynamic in the instance. Keep this split in mind — the rest of the series leans on it.

Where MCP fits in

The Model Context Protocol (MCP) standardizes one half of the picture: how a host discovers and calls tools in the first place. Before MCP, every "agent plus data source" pairing needed its own bespoke integration — a Slack agent, a GitHub agent, a database agent, each wired by hand. MCP turns that into a protocol: any MCP server exposes typed, discoverable tools over a standard interface, and any MCP-aware host can connect to any MCP server without custom glue code.

Think of it as the plug standardizing, not the appliance. The agent — definition, instance, and the loop between them — still has to exist on the host side. MCP just means the host doesn't have to reinvent "how do I talk to this particular tool" every single time.

What this looks like in practice

I've been building a small MCP host that does exactly this: it connects to one or more MCP servers, aggregates whatever tools they expose, and drives that call-a-tool-or-answer loop on behalf of an LLM. The interesting design decision wasn't the happy path — it was resisting the temptation to let the SDK auto-run tools invisibly. Most agent frameworks offer a mode where you hand the model a tool list and it just handles the rest, opaquely, in one call.

I deliberately don't do that. The host drives the loop by hand: send the conversation, inspect the response for tool-call requests, dispatch each one, feed the results back, repeat — capped by the termination limits from the agent definition, so nothing spirals. It's a few dozen more lines of code than the "just let the SDK do it" version. What it buys you is visibility: every tool call, every round trip, every token is something you can actually see and measure, instead of a black box between "here's a question" and "here's an answer."

That visibility turns out to matter a lot once you start asking harder questions. Not just "does the agent work," but "is this system prompt actually better than that one," "is the cheaper model good enough," "what will this cost at scale." Those are measurement questions, not build questions — and they're what the rest of this series is about.

Next up: the levers you have when configuring an agent — which, not coincidentally, are exactly the fields of the agent definition above — and the metrics that tell you whether pulling one made things better or worse.

Deeper dive into the definition itself: https://dev.to/langensjonathan/what-is-an-agent-a-classinstance-definition-stress-tested-against-the-2026-07-28-mcp-spec-5akp

What is an 'agent'? A class/instance definition, stress-tested against the 2026-07-28 MCP spec

Jonathan Langens — Tue, 28 Jul 2026 07:28:30 +0000

Industry standards

In the Gen AI / LLM space, two standards have emerged as the de facto choice in their domains:

A2A (v1.0, Linux Foundation) defines how agents integrate and communicate with their peers. It defines the interface of an agent, but not the agent itself.
MCP (2026-07-28 spec, finalizes today) defines how an agent uses tools. It defines the internal plumbing of an agent, but not the agent itself.

One nuance to be fair to both: MCP can also act as an agent's public interface, when an agent is exposed as an MCP server inside a single trust domain. So A2A covers the cross-organization contract, and MCP covers everything inside it. Either way, the conclusion stands: neither protocol defines what an agent is.

To me it seems the terms 'agent' and 'agentic' are still not well defined, and this article is my attempt at an operational definition: one precise enough that you could build a host around it. That claim is not hypothetical — I am building an MCP host, and this definition is what fell out of that work.

Prior art (and why it is not enough)

"An agent is an LLM using tools in a loop" (Anthropic's framing) is directionally right, but it is not operational: it does not tell you what belongs in your agents.json. The academic definitions (rational agents, BDI, FIPA) predate LLMs and do not map to context windows, token budgets or MCP servers. I want something in between: a definition you can serialize.

Scope: this is a definition for agents that live in an MCP host. Frameworks like LangGraph organize things differently, but I believe the same components show up under different names.

The definitions

I split the concept in two parts, like a class and its instances. Both live in the MCP host. From here on: an agent definition is the class, an agent is the instance.

The agent definition consists of:

the host loop — the core. The LLM deciding actions, observing results and iterating. Without a loop you have a chatbot with tools, not an agent.
system context — persona, instructions. Static and authored.
MCP server set
capability boundary / tool policy — which servers and tools this agent may use, and what needs approval. This is a role, not a login.
LLM — the actual model we are using
termination limits — max iterations, token budget, stop conditions
task contract — the schema of what this agent accepts and what it must return
context-size strategy — compaction vs top-x

An agent is an instance of that class. In C# terms:

record AgentDefinition(
    HostLoop Loop,
    SystemContext Context,
    McpServerSet Servers,
    CapabilityBoundary Capabilities,
    Model Llm,
    TerminationLimits Limits,
    TaskContract Contract,
    ContextStrategy Strategy);

record Agent(
    AgentDefinition Definition,
    AgentTask Goal,
    ContextWindow WorkingContext,
    Principal Credentials,
    Budget Consumed);

goal — what the user actually asked
working context window — conversation history, tool results
credentials — the principal: whose OAuth tokens, on whose behalf. Two instances of the same definition, running for two different users, differ only here.
consumed budget — iterations elapsed, tokens spent

Note the symmetry: the definition holds limits, contracts and capabilities; the instance holds counters, goals and credentials. Static in the class, dynamic in the instance.

Enter the 2026-07-28 spec, or more precisely: the stateless core

The new MCP spec removes protocol sessions entirely. A server no longer remembers you between calls. All state now travels in explicit handles that the model itself can see: task handles for long-running work, workflow ids, and the requestState blob a paused call hands back.

This breaks my 'working context window' as defined above. Those handles land in the context window, which means compacting — or worse, top-x-ing — becomes a correctness concern instead of a cost concern. Summarize away a task handle and the agent has orphaned remote work it can never resume, because the stateless server has no session through which to remind it.

So the instance needs a split:

compressible context — conversation history, safe to compact
load-bearing context — outstanding handles and request state, never compacted

The host loop also stops being free-form. It must now support:

the multi-round-trip pause — a tool call returns input-required plus request state; the loop surfaces the question and re-issues the call with the answer
task polling (tasks/get)
unsolicited task handles arriving in results

Serializable Agents

Here is where the stateless core points. If instance state is exactly {goal, context, handles, credentials, consumed budget}, and the protocol holds no hidden session on your behalf, then an agent is a serializable value. You can suspend it, persist it, and resume it in a different host process. Not a running process you must keep alive — a record you can store.

Production reality will take a while to catch up (connection reuse, servers caches, ...), but the direction is set by the spec itself.

I'm looking for feedback on this — especially from people running MCP hosts in production: what is in your agent config that this definition misses?