Part 2 of 3 — building and testing MCP agents
Every AI agent is a bundle of decisions, most of which get made once, informally, and never revisited: which model, what system prompt, which tools it's allowed to touch, how many steps it gets before you give up on it. Nobody writes these down as a config. They just happen, in code, and then everyone forgets they were choices at all.
In Part 1 I argued those decisions have a shape: an agent definition (the class — loop, system context, capability boundary, model, limits) and an agent instance (a running occurrence — goal, context window, credentials, consumed budget). That split turns out to be exactly what you need for tuning, because the two kinds of parameters people mix up when "tuning an agent" live on opposite sides of it: you tune the definition, and you measure the instances.
Tunable parameters: the fields of the agent definition
These are the levers — change any one and you have a different agent, even if everything else stays the same:
- system context — how the agent is instructed
- LLM — which model answers, and at what price
- MCP server set and capability boundary — which tool sources exist, and which of their tools this agent may actually reach for
- the host loop — sequential vs parallel tool execution, retry behavior
- termination limits — how many iterations and tokens it gets before you cut it off
- context-size strategy — how it manages a growing conversation history as a task gets longer
That's the agent definition from Part 1, field for field — with one deliberate absence. The task contract is the only definition field that isn't a lever. You don't tune it; you tune everything else against it. Hold that thought.
The consequence of seeing these as fields is simple but real: if you don't track which field you changed, you can't attribute why the results changed.
Outcome parameters: what you measure on the instances
This is the half people skip. An agent isn't "good" in the abstract — it's good or bad against criteria you define yourself, and every one of them is a measurement taken from agent instances after they run:
- Does it reach an acceptable answer? Correctness isn't binary — sometimes it's "contains the right fact," sometimes it needs a rubric and a second model to judge it. Notice what this is: the task contract made testable. The definition says what the agent must return; the outcome criterion checks whether an instance actually did.
- How long does it take? Wall-clock time, including every tool call, not just the model's "thinking."
- How many tokens does it burn? Input and output, summed across every round trip, not just the final message. This is the instance's consumed budget, read back out.
- Which tools does it reach for, and how often? Underrated. Some tools are cheap to call, others are expensive — rate-limited APIs, slow external services, tools with real dollar cost per call. An agent that gets the right answer by using an expensive tool once is worse than one that gets there in five cheap calls, even if both "pass."
The order matters more than the list
Here's the part that's easy to get backwards: you have to define the outcome parameters before you start sweeping the tunable ones. Decide what a correct answer looks like, what latency is acceptable, what token budget you're willing to spend, and which tools should be used sparingly — write it down as a criterion, a rubric, a budget. Only then start trying a tighter system context, a cheaper model, a different loop strategy.
Skip that step and "tuning" becomes vibes — you nudge a prompt, the answer looks a little better in the three examples you tried, you ship it, and you have no idea whether it's actually better or just different. Do it in the right order and every change becomes a testable hypothesis: does this specific change to one definition field move the numbers I said mattered, in the direction I wanted, without moving the ones I didn't touch?
Why this matters: an agent definition is data, not code
The reason this framing pays off in practice is that once the system context, model, capability boundary, and loop settings are configuration rather than something baked into code — an agents.json, in my host — comparing two agents stops being a rewrite and becomes a diff. Point the same set of test questions at two definitions that differ in exactly one field, run a batch of instances against each, and any difference in the measured outcomes is attributable to that one field.
That's the whole game: tune the class, measure the instances. And it's exactly the setup you need for what comes next — running these comparisons systematically instead of by hand.
The link to part 1 of this series: https://dev.to/langensjonathan/mcp-agents-explained-what-actually-makes-an-llm-an-agent-12a0
Top comments (0)