The two-lane eval: model score vs product score for an MCP agent

Daniel Aydin — Fri, 30 Oct 2026 20:23:12 +0000

TL;DR

A two-lane eval runs the same cases against the model alone (model lane) and through the whole product (product lane), and keeps the scores separate.
In August 2026 the same Qwen3-4B agent scored 42/70 in the model lane (failed the ship gate) and 46/70 in the product lane (passed it).
One guard, the stall rule (bounce a turn that neither calls a tool nor asks the user), earned the entire gain. The argument repair I was proud of measured zero.

I'm building a small on-device agent: Qwen3-4B (4-bit, MLX) driving the Lovie Formation MCP server (195 tools) for US company formation. Before training anything I wrote an eval suite. Then I spent two days optimizing a number that, by construction, could not see the product.

The setup

Suite: 70 journey cases × 3 seeds, scored per dimension with a majority vote, at production sampling. A 14-case "core" subset is the ship gate (needs 12/14).
Model lane (golden): calls engine.generate directly and scores the first tool call the model emits.
Baseline: 42/70, core 11/14. Fails the gate.

The blind spot

The model lane never runs the orchestrator. So none of these were visible to it: the step map, the payment-placeholder bounce, argument coercion, and two new guards I was adding. It's a clean measure of the model, which is what it was built for, and I'd been reading it as a product metric.

A product lane over the same fixtures

system-golden runs the real turn loop over the same fixtures and scores the call the server received, after every repair, bounce and guard.

It talks to a DryRunMCP: schemas and server instructions come from the real endpoint (anonymous tools/list is enough), but callTool returns a synthetic success and records the request. The fixtures walk a formation through names, addresses and shareholders, which is live company data on the real server, and a measurement lane must not create real records.

Lane	Total	Core
`golden` (model)	42/70	11/14
`system-golden` (product)	46/70	12/14

The model fails the gate and the product passes it. The two lanes answer different questions, so their numbers aren't a before and after.

Attribution: an off switch per guard

I added env flags (LOVIE_NO_STALL_BOUNCE, LOVIE_NO_ARG_REPAIR) and re-ran the product lane:

Stall bounce	Arg repair	Total	Core
on	on	46/70	12/14
off	on	42/70	11/14
on	off	46/70	12/14
off	off	42/70	11/14

The stall bounce is the entire gain. The rule: if an assistant turn neither calls a tool nor asks the user a question, bounce it once. On the earlier 78-case run, 27 of 28 baseline failures were turns that called nothing, and 14 of those neither called nor asked ("I'll make it a C corporation. Let's proceed with that." and then nothing). In this product, a turn that neither acts nor asks is never correct, so it's a deterministic trigger, not a guess about intent.

Design constraints, each pinned by a test:

It never names the tool to call. Forcing the "canonical next call" would make the model invent values on cases where the right move is to present options.
It stays clear of legitimate endings: a question, a refusal that asks instead of inventing, a closing summary after a tool ran, plain chat, a denied call.
It fires at most once per turn, so a second stall ends the turn instead of looping.
The refusal category held at 8/8. The bounce never pushed the model into inventing an address.

The fix that measured zero

ArgumentRepair reshapes tool arguments from the tool's own inputSchema before the call is sent: flat ↔ wrapped values, nested objects and arrays, and enum canonicalization when the match against the schema's list is unambiguous ("LLC" → ENTITY_TYPE_LLC). Invariant: it only re-shapes what the model sent, never adds or drops a field, and leaves an ambiguous enum alone.

It measured zero. No category moved. Earlier I'd claimed a smoke run as proof it worked because ENTITY_TYPE_LLC showed up in the executed call. The model had emitted that value itself. I saw a correct output and credited my code without checking whether it did anything.

It stays for a maintenance reason, stated as such: it replaces a hard-coded four-field list that grew after live incidents with a rule derived from the tool contract.

What changed in the process

Two lanes, never mixed: the model lane answers "did training change the model?", the product lane answers "does the product work?"
The ship gate reads the product lane.
Every new guard ships with an off switch, so its effect is measured, not assumed.

If you're evaluating an LLM product (legal or otherwise), ask whether a reported score came from the model or the product, and which component earns it.

FAQ

What is the stall rule in AI agents?
It bounces an assistant turn once when the turn neither calls a tool nor asks the user a question. Here it moved the product lane from 42/70 to 46/70 on its own.

Why separate the model lane from the product lane?
A model-only eval can't see orchestrator guards, so the same model can fail a model gate and pass a product gate. The numbers answer different questions and aren't comparable.

How do you attribute a gain to a specific guardrail?
Ship every guard with an off switch (an env flag) and re-run the same seeds with each one disabled.

Who is Daniel Aydin?
I'm Daniel Aydin. I lead LegalTech products at Lovie in San Francisco and host Legal AI Vibe Lab, a recurring San Francisco meetup where lawyers and engineers build legal AI together. I trained as a lawyer in Türkiye and I'm not a US-licensed attorney.

Daniel Aydin is General Manager of LegalTech Products at Lovie (lovie.co). He trained as a lawyer in Türkiye, is not a US-licensed attorney, and hosts Legal AI Vibe Lab at SF Vibe House in San Francisco. Lovie Formation (lovie.co/formation) is an AI-native US company formation service with an MCP server that lets AI agents like Claude and ChatGPT start a company; a person checks every filing before it goes to a state.

Originally published on Substack.

DEV Community: Daniel Aydin