I found a bug in my own stack last month that had already been "fixed" for three weeks.
One of my tools takes a timeout in milliseconds. The model had been sending "30s". The call succeeded anyway, because somewhere between the model and the tool there was a coercion shim I'd written in a hurry and then forgotten about. It parsed the string, multiplied, passed the int through. Green check. Nobody looked at it again.
That shim is the whole myth in one line: a proxy that silently repairs malformed tool calls does not make your agent more reliable. It moves the failure from a visible error into an invisible correction. Your success rate goes up. Your tool-call accuracy does not move at all.
Why it feels like it works
Because the dashboard agrees with you. Before the gateway, you had a wall of validation errors and a handful of broken trajectories. After the gateway, the errors are gone and the trajectories complete. Every signal you have says the system got better.
But nothing about the model changed. It still guesses wrong at the same rate. You just stopped counting the guesses. What you measured wasn't accuracy — it was the width of your repair heuristics. Those are different numbers, and only one of them is about the model.
I've watched this bite in three places.
The schema stops being a contract. Once a layer downstream will accept "30s", 30, and "30000", your schema is no longer a specification. It's a suggestion with a parser behind it. And that parser is now the least-reviewed, least-tested code in the entire stack, running on every single call. I've never once seen a repair heuristic with a test suite. I have seen plenty with a comment that says # temp.
The model never learns it guessed. This is the part people skip. Inside a single trajectory, the error message is the feedback signal. You return {"error": "timeout_ms must be an integer"} as a tool result, the model reads it, and on the next turn it sends an integer. That's the loop working. A silent repair cuts the loop. The model gets a success result for a call it got wrong, and it will make the same guess for the rest of the session, and in every session after that, forever. You've trained your prompt into a shape that only works because of the shim.
You lose the ability to tell whether a model upgrade helped. Swap in a newer model, run your evals, watch the numbers. They'll look identical, because the repair layer is absorbing the difference. You can't distinguish "this model emits better tool calls" from "my heuristics got broader last Tuesday." That's the expensive one. You're paying for evals that can't see the thing they're evaluating.
The honest version of the argument
There is a real case for lenient parsing. Dates. Numeric strings from HTML forms. Anything where the ambiguity is in the encoding, not in the model's understanding of the task. If a tool legitimately accepts 2024-03-01 and 2024-03-01T00:00:00Z, normalizing both is not a repair, it's just parsing. Fine. Do it.
The distinction I care about is whether the correction is loud and counted. Coerce the date, but increment a counter and log the raw input. If that counter is at zero for a week, delete the coercion. If it's climbing, you have a prompt problem you can now actually see.
Silent repair is different in kind. It's a correction with no counter, no log line, and no owner. That's not a reliability feature. That's a metric laundering service.
And yes — there's a trade I'd take. Customer-facing chat where a failed tool call is worse than a slightly wrong one, demo day, a migration window where you need the old clients to keep working. Silent repair is defensible there. But say it out loud, put an expiry date on it, and stop calling it reliability. It's a tourniquet. Tourniquets have a place. Nobody puts one on a healthy arm and calls it fitness.
What I do instead
Strict validation at the tool boundary. No coercion unless it's declared in the schema as an accepted union.
On failure, return the validation error to the model as a normal tool result, in plain language, with the expected shape. One retry. Then fail the step loudly and let the trajectory die where it died.
Log the raw arguments before validation, always. The bug report you'll eventually get is "it did something weird yesterday," and raw args are the only thing that answers it.
Count validation failures as a first-class metric, next to latency and cost. If that number is flat while your task success rate climbs, something else is doing the work and you should find out what.
And if you already have a repair layer you can't remove today: run it in shadow mode. Let it compute the correction, log what it would have fixed, and don't apply it. Two weeks of that tells you exactly how much of your reliability is real and how much is the shim. My guess is you'll be surprised in the bad way. Mine was.
The fixes that actually moved my agents' reliability this year were all subtractions — narrower schemas, fewer tools, stricter types, more failures surfaced. I haven't added a single forgiving layer that I didn't later regret.
Top comments (0)