A data-sync agent I maintain reported one morning that it had reconciled 1,400 records. The trace was green from the first span to the last. No exceptions, duration in the normal band, every span closed clean. It had reconciled nothing. One of its tools had failed three times in a row, the retry wrapper around that tool handed back the third failed attempt as if it were an ordinary result, and the agent wrote a confident one-paragraph summary over an empty payload.
Nothing in that chain raised an error. That is the point I keep relearning: the worst agent bugs do not throw. The harness catches the failure for me, which is its job, and catching it is exactly what buries it.
A caught error is not a recorded error
The mechanism is almost dull once you see it. No code, just the shape. Tracing marks a span as failed when an exception escapes that span's block. That part is automatic, and it is the behavior everyone leans on. The moment you wrap a risky call in a try and handle the error inside the block, nothing escapes, so nothing gets marked. The span keeps its default status, the one that means neither success nor failure, and every trace UI I have used paints that default green.
So the retry wrapper that swallowed my tool failure did not lie, exactly. It just never recorded that anything went wrong. An error-rate dashboard built on span status drew a flat, clean line straight through the incident. The failure was real. The signal was gone before it ever reached me.
The harness hides failures in five predictable places
Once I started hunting the pattern instead of the symptom, the same handful of recovery habits showed up every time. Each one is reasonable on its own. Each one deletes the exact signal you would have used to find the problem.
Retry wrappers that return the last attempt. A call that failed three times becomes a normal return value, and the attempt count usually lives nowhere the trace can see.
Step caps that report partial work as finished. A run that burned through its step budget returns the same shape as a run that actually completed, so "it stopped early" and "it succeeded" look identical downstream.
Exceptions stringified into the model's context. The error turns into a line of text the model reads and reasons over as if it were data, and the run keeps going in a state your monitoring reads as healthy.
Output parsers that coerce instead of failing. A malformed response gets its missing fields filled with defaults, and downstream code cannot tell a field the model produced from a field the parser invented.
Fallbacks that quietly downgrade the model. When the primary trips and a weaker model silently takes over, the quality drop looks exactly like a prompt problem, and you lose a day editing prompts that were never the issue.
The fix is to record before you recover
I am not arguing the harness should stop catching errors. Keeping the run alive is the whole reason it exists. The problem is that recovery and reporting are two separate jobs, and almost every default above collapses them into one, then drops the reporting half.
So I changed the defaults, not the behavior. The wrapper still retries, but it records the attempt count and marks the span on the final failure before it returns anything. A step cap hands back a distinct finish reason instead of a success-shaped object. The broad catch around a tool call becomes a narrow one that records the exception and then recovers. Parsers run in strict mode, so an unknown or missing field fails loudly instead of getting patched. A fallback records which model actually answered. None of that stops the agent. It stops the agent from lying about what it did.
You catch these in evaluation, not on a dashboard
What finally worked was giving up on the idea that a dashboard would ever surface this. If the output is the only thing you check, a swallowed failure walks right through every time the model writes a plausible summary over a bad result. Checking the output is necessary and it is nowhere near sufficient.
What catches them is asserting on the shape of the run itself, which you can only do where you already know what the run was supposed to look like. That means evaluation, not production. Three checks do most of the work. Compare the actual step count against an expected range, and a run that hit its cap separates cleanly from one that finished in four steps. Check the tool calls, and a tool that was invoked but returned nothing usable, or one that should have fired and never did, stops hiding. Check the trajectory, the sequence of actions, and you catch reruns, skipped steps, and loops that a final-answer check cannot see.
There is a real ceiling here, and I want to be honest about it. If the harness already erased the signal, no downstream check recovers the original error. What these assertions give you is the symptom, loud and early, which is enough to send you to the trace where the swallowed exception gets diagnosed. Every silent failure I have chased got caught the same way: a person finally noticed a weird answer, never a test and never an alert, and a couple of them had been running for weeks by the time anyone did. Run-shape checks are how you stop depending on that person.
The defaults that keep lying to you
If you change only five things, change these.
Make retry wrappers record the attempt count and mark the span when the final try fails.
Give a step cap its own finish reason so cap exhaustion never wears the shape of success.
Catch narrowly around tool calls, record the exception, then recover.
Run parsers in strict mode so a missing field fails loudly instead of getting invented.
Record which model actually answered when a fallback quietly takes over.
If you want the deeper version, this piece walks through all five patterns and the exact span-status behavior that keeps them invisible.
I would like to know which of these has bitten you. For me it is almost always the first one, a retry wrapper handing back the last thing that failed as though it were an answer.
Top comments (0)