I spend most of my time on the seam between a model and the tools it calls. Over the last year, the failure mode that has cost me the most debugging hours is a tool that breaks quietly while the model reports back as if the call had gone fine.
I want to be careful about the claim, because it is easy to overstate. The model is not "lying," and it is (usually) not hallucinating in the way people mean when they say that word. It is reading a tool result that does not clearly say this failed, and it fills the gap the way it fills every gap: by continuing. The fix is almost never in the prompt. It lives in how the tool boundary encodes failure.
Here is the number that made me stop treating this as anecdotal. Over one 30-day window (roughly 41,000 tool invocations across two agents in production), 1,142 calls returned something other than a clean success: a timeout, an error body, a partial write. In 331 of those, about 29%, the model continued the plan as if the call had succeeded. That 29% is the figure I care about, because every one of those is a silent wrong answer with no exception in the logs to catch it.
Four cases, same shape each time.
Case 1: The timeout that came back as an empty string
Our retrieval tool had a 5-second budget. When it blew past that, the wrapper caught the timeout and returned "" (an empty string) rather than raising. The intent was defensive. The effect was that the model read an empty result as the search ran and found nothing, and it confidently told the user there were no matching records.
There were matching records. The search never completed.
The tell (in hindsight) is that "found nothing" and "did not run" collapsed into the same token stream. A human on-call would notice a 5-second gap and a suspiciously empty payload. The model has no clock and no baseline, so it cannot notice either.
Case 2: The 200 that carried an error body
This one is almost a genre. An upstream API returned HTTP 200 with a body of {"status": "error", "message": "rate limited, retry after 30s"}. Our tool wrapper checked the HTTP status code (200, so "success"), serialized the body, and handed it back. The model saw a JSON object with fields in it, treated the fields as data, and reasoned over message as though it were a result.
I would flag the general rule here, because it burned us more than once: transport-level success (the request completed) and application-level success (the thing you asked for happened) are different questions, and a lot of wrappers only answer the first one. If your success check is response.ok, you are trusting the upstream service to never return an error inside a 200. In my experience that trust is misplaced roughly as often as you would expect, which is to say: often enough to matter, rarely enough that you forget about it between incidents.
Case 3: The partial write that reported as whole
A multi-step tool plan wrote three records: two committed, the third failed on a constraint violation. The orchestration layer returned a single top-level success: true because it read one sub-call's status instead of aggregating all three, and that sub-call was one of the two that committed. The model summarized the operation as complete. Downstream, one record was missing, and nothing in the transcript hinted at it.
Partial failure is the case I think teams underinvest in, because it does not look like a failure from either end. The tool did not throw. The model did not confabulate. The plan just had a hole in the middle that neither side was responsible for noticing. (I will concede this one is as much an orchestration bug as a model-boundary bug. But the model happily papered over it, and that is the part I can actually harden.)
Case 4: The exception stringified into the result field
The one I find hardest to defend against. A tool caught its own exception and did return {"result": repr(e)}. So the result field, the field the model is trained to read as the answer, now contained the text of the exception. The model, gamely, tried to use it. In one trace it read KeyError('user_id') and reported to the user that their ID was KeyError.
In a test that reads as a bug. In front of a customer it reads as us not knowing our own data.
The shared cause, and the thing that actually fixed it
The four cases look different (a timeout, a 200, a partial commit, a swallowed exception), but they share one property: failure was encoded in-band, in the same channel and often the same field as a real result. The model had no structural way to tell "here is your answer" from "here is why you have no answer."
What moved the 29% was not a better system prompt. It was forcing every tool to return an explicit status envelope, and putting the failure signal somewhere the payload can never live:
{
"ok": false,
"error_kind": "timeout", // timeout | upstream_error | partial | exception | not_found
"retriable": true,
"partial_results": null, // present only when error_kind == "partial"
"data": null, // populated ONLY when ok == true
"message": "retrieval exceeded 5s budget; no results fetched"
}
Two rules make the envelope earn its keep. First, data is populated only when ok is true, so the model cannot accidentally read an error message as an answer (Case 4 dies here). Second, not_found is its own error_kind, distinct from timeout, so "ran and found nothing" and "never ran" stop collapsing into the same thing (Case 1 dies here). Cases 2 and 3 need the wrapper to actually check application-level status and to aggregate every sub-call, which the envelope does not do for you, but it at least gives them a place to report the truth once you do.
After we rolled this out across the tool layer, the silent-continue rate over the next comparable window dropped from 29% to about 4% (roughly 46 of 1,180 non-success calls). I do not think 4% is a floor anyone should be proud of, and I have not fully chased down what is left. But going from "one in three quiet failures sails through" to "one in twenty-five" changed which bugs made it to users.
Where I'd push back on this
If I were reviewing my own argument, here is the strongest version of the counter-case, because I think it is partly right.
The steelman: envelopes push complexity onto every tool author, and they invite a false sense of safety. You can hand the model a perfectly structured ok: false and it can still barrel ahead and ignore it, because nothing forces it to branch on that field. The status envelope makes the failure legible, not respected. And a determined team could get most of the same benefit with strict typing and a retry layer that never lets a malformed result reach the model at all, no envelope required. That is a fair hit. The envelope is a convention, and a convention only holds while every tool author keeps honoring it.
My concession: yes. On our own numbers, the residual 4% is mostly the model reading a clean ok: false and continuing anyway, which is exactly the failure the envelope was supposed to prevent. So the envelope solved the encoding problem (the model can no longer confuse an error for data) and only dented the compliance problem (the model still sometimes ignores a well-formed error). Those are two different problems and I conflated them for longer than I would like to admit.
Objections I'd accept: that this is really a contract-design problem, not a model problem; that a hard retry-or-halt layer outside the model is stronger than any envelope the model can choose to ignore; that my 4% is under-investigated and might be papering over a subclass I have not named yet.
Objections I wouldn't accept: that better prompting ("if a tool fails, stop and report it") fixes this. We tried the prompt-only version first. It moved the 29% by a couple of points and drifted back within a week. If the failure is not structurally distinguishable from a success, no amount of instruction makes the model reliably see a difference that is not encoded in what it reads.
The short version I would give a teammate: assume every tool will fail silently at least once, and design the boundary so that a silent failure is impossible to read as a success. You still have to get the model to branch on the signal. That is a separate problem, and it only starts once the signal is unambiguous in what the model reads.
Top comments (0)