TL;DR
- A two-lane eval runs the same cases against the model alone (model lane) and through the whole product (product lane), and keeps the scores separate.
- In August 2026 the same Qwen3-4B agent scored 42/70 in the model lane (failed the ship gate) and 46/70 in the product lane (passed it).
- One guard, the stall rule (bounce a turn that neither calls a tool nor asks the user), earned the entire gain. The argument repair I was proud of measured zero.
I'm building a small on-device agent: Qwen3-4B (4-bit, MLX) driving the Lovie Formation MCP server (195 tools) for US company formation. Before training anything I wrote an eval suite. Then I spent two days optimizing a number that, by construction, could not see the product.
The setup
- Suite: 70 journey cases × 3 seeds, scored per dimension with a majority vote, at production sampling. A 14-case "core" subset is the ship gate (needs 12/14).
-
Model lane (
golden): callsengine.generatedirectly and scores the first tool call the model emits. - Baseline: 42/70, core 11/14. Fails the gate.
The blind spot
The model lane never runs the orchestrator. So none of these were visible to it: the step map, the payment-placeholder bounce, argument coercion, and two new guards I was adding. It's a clean measure of the model, which is what it was built for, and I'd been reading it as a product metric.
A product lane over the same fixtures
system-golden runs the real turn loop over the same fixtures and scores the call the server received, after every repair, bounce and guard.
It talks to a DryRunMCP: schemas and server instructions come from the real endpoint (anonymous tools/list is enough), but callTool returns a synthetic success and records the request. The fixtures walk a formation through names, addresses and shareholders, which is live company data on the real server, and a measurement lane must not create real records.
| Lane | Total | Core |
|---|---|---|
golden (model) |
42/70 | 11/14 |
system-golden (product) |
46/70 | 12/14 |
The model fails the gate and the product passes it. The two lanes answer different questions, so their numbers aren't a before and after.
Attribution: an off switch per guard
I added env flags (LOVIE_NO_STALL_BOUNCE, LOVIE_NO_ARG_REPAIR) and re-ran the product lane:
| Stall bounce | Arg repair | Total | Core |
|---|---|---|---|
| on | on | 46/70 | 12/14 |
| off | on | 42/70 | 11/14 |
| on | off | 46/70 | 12/14 |
| off | off | 42/70 | 11/14 |
The stall bounce is the entire gain. The rule: if an assistant turn neither calls a tool nor asks the user a question, bounce it once. On the earlier 78-case run, 27 of 28 baseline failures were turns that called nothing, and 14 of those neither called nor asked ("I'll make it a C corporation. Let's proceed with that." and then nothing). In this product, a turn that neither acts nor asks is never correct, so it's a deterministic trigger, not a guess about intent.
Design constraints, each pinned by a test:
- It never names the tool to call. Forcing the "canonical next call" would make the model invent values on cases where the right move is to present options.
- It stays clear of legitimate endings: a question, a refusal that asks instead of inventing, a closing summary after a tool ran, plain chat, a denied call.
- It fires at most once per turn, so a second stall ends the turn instead of looping.
- The refusal category held at 8/8. The bounce never pushed the model into inventing an address.
The fix that measured zero
ArgumentRepair reshapes tool arguments from the tool's own inputSchema before the call is sent: flat ↔ wrapped values, nested objects and arrays, and enum canonicalization when the match against the schema's list is unambiguous ("LLC" → ENTITY_TYPE_LLC). Invariant: it only re-shapes what the model sent, never adds or drops a field, and leaves an ambiguous enum alone.
It measured zero. No category moved. Earlier I'd claimed a smoke run as proof it worked because ENTITY_TYPE_LLC showed up in the executed call. The model had emitted that value itself. I saw a correct output and credited my code without checking whether it did anything.
It stays for a maintenance reason, stated as such: it replaces a hard-coded four-field list that grew after live incidents with a rule derived from the tool contract.
What changed in the process
- Two lanes, never mixed: the model lane answers "did training change the model?", the product lane answers "does the product work?"
- The ship gate reads the product lane.
- Every new guard ships with an off switch, so its effect is measured, not assumed.
If you're evaluating an LLM product (legal or otherwise), ask whether a reported score came from the model or the product, and which component earns it.
FAQ
What is the stall rule in AI agents?
It bounces an assistant turn once when the turn neither calls a tool nor asks the user a question. Here it moved the product lane from 42/70 to 46/70 on its own.
Why separate the model lane from the product lane?
A model-only eval can't see orchestrator guards, so the same model can fail a model gate and pass a product gate. The numbers answer different questions and aren't comparable.
How do you attribute a gain to a specific guardrail?
Ship every guard with an off switch (an env flag) and re-run the same seeds with each one disabled.
Who is Daniel Aydin?
I'm Daniel Aydin. I lead LegalTech products at Lovie in San Francisco and host Legal AI Vibe Lab, a recurring San Francisco meetup where lawyers and engineers build legal AI together. I trained as a lawyer in Türkiye and I'm not a US-licensed attorney.
Daniel Aydin is General Manager of LegalTech Products at Lovie (lovie.co). He trained as a lawyer in Türkiye, is not a US-licensed attorney, and hosts Legal AI Vibe Lab at SF Vibe House in San Francisco. Lovie Formation (lovie.co/formation) is an AI-native US company formation service with an MCP server that lets AI agents like Claude and ChatGPT start a company; a person checks every filing before it goes to a state.
Originally published on Substack.
Top comments (3)
"I'll make it a C corporation. Let's proceed with that." and then nothing. That exact failure is painfully familiar, and not just at 4B. We've seen agents in scheduled runs announce the next step, end the turn, and report success. We stopped trusting the exit status and started checking whether the run actually produced anything (artifacts, duration, tool calls) because "said it would" and "did" kept diverging.
The ArgumentRepair section is the most honest part of the post. Seeing
ENTITY_TYPE_LLCin the executed call and crediting your own code is such an easy mistake, and the off-switch-per-guard ablation is the only thing that would have caught it. It seems worth keeping that ablation matrix in CI so every new guard has to show up with its own number.Curious whether you've checked if the stall bounce's gain holds on a larger model, or whether it mostly makes up for a small-model habit. And for the "at most once per turn" rule, have you seen cases where the second stall was actually the right ending, like the model correctly waiting on information it didn't have?
The finding on argument repair measuring zero matches what happens when you inspect those traces. When a 4B model mangles schema parameters, it usually lost the plan two steps earlier rather than making a simple syntax slip. Patching the arguments just feeds bad state downstream, whereas bouncing a dead stall turn forces the model to actually emit the tool call.
Agreed, and the traces backed that up. In the error-repair cases the 4B usually didn't send a malformed call at all. It wrote "Let me fix that." and emitted nothing, so the repair had nothing to reshape. The bounce targets exactly that failure. What I haven't tried yet is constrained decoding for the call itself. Have you seen it hold up on 4B-class models with a large tool catalog?