This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.
Third entry in the DEV x Sentry Bug Smash. Entry 1 was a cras...
For further actions, you may consider blocking this person and/or reporting abuse
"They hand the model a plausible-but-wrong message and let it fail politely, on repeat." That's the truest thing I've read about agent failure this month. The red stack trace is mercy — it's the polite, well-handled, misleading error that eats your afternoon.
I hit the exact structure outside Python. On my server a firewall rewrites a legitimate 4xx into a 405, so my agent gets handed "405" — plausible, wrong, pointing at the wrong cause. Left alone it will "fix" the routing three times, same as your model retrying valid syntax, because the message names a problem that isn't the problem. The retry loop isn't the model being dumb; it's the model faithfully acting on a message that was already a lie.
Two things I'm taking from your fix. One: fuzz the sandbox with ordinary valid input and treat every false rejection as a bug — stealing that as a test for my own guardrails. Two: this week I gave my agents a rule that's basically your lesson in prose — when you hit a familiar failure signal, translate what it means in THIS environment before you act on it, because a politely wrong message will happily let you loop forever. You found the version that burns tokens; mine burns a junior's afternoon. Same silent killer.
thanks man
The 405 rewrite is the perfect parallel: the message names a problem that isn't the problem, and a faithful agent will "fix" the wrong thing on loop. Love your rule of translating a familiar failure signal into what it means in THIS environment before acting on it. Mine burns tokens, yours burns a junior's afternoon, same silent killer. And yes, fuzz your guardrails with ordinary valid input, every false rejection is a bug.
Already did it — your last line went straight into code the same afternoon. I planted six known-good snippets into my scanner's test suite: escaped output, a prepared query, a hashed password, the safe patterns my regex rules could plausibly misfire on. Ran it: caught everything it should, false-flagged none of the six. Now if I ever "tighten" a rule and it starts biting clean code, that test goes red instead of me hearing it from an annoyed teammate.
The part I didn't expect: one snippet — the string "eval(" sitting inside a comment — does still trip it. Not a regression, just an honest known blind spot now, labeled as one instead of pretending it isn't there. Your framing turned a vague "should test that" into a standing tripwire. Thanks man — this one earned its keep.
This is the best possible outcome: a vague "should test that" turned into a standing tripwire. And the eval-inside-a-comment case being labeled a known blind spot instead of hidden is exactly the win. A blind spot you can see beats a silent one you can't, and now the day you tighten a rule too far, the test goes red instead of a teammate. That is the whole thesis in one test file. Glad it earned its keep.
"A blind spot you can see beats a silent one you can't" — that's the entire corner of my brain that's been busy this week, in one line. The test doesn't catch the comment-eval case; it never will, that's still a false positive. What changed isn't detection, it's status: the same failure went from silent to signed. It's the difference between a field that's absent and a field that's present-and-null — the null fixes nothing, it just proves someone looked, and that this is a known, chosen limit rather than an accident nobody noticed.
That's the pattern I keep finding everywhere lately: the win usually isn't fixing the failure, it's dragging the failure into a place where it announces itself. A red test on the day I over-tighten is exactly that — the cost shows up early and loud, instead of late and quiet on a teammate. You put the thesis better than my commit message did. Glad it landed, and thanks for the line that started the whole thing.
The rule that any rejected valid Python counts as a bug seems like it would also flag things the sandbox blocks on purpose, like imports outside the authorized list. How did you tell intentional restrictions apart from real bugs when triaging, an allowlist of expected errors or just by hand?
Good question, and it was mostly by construction rather than an allowlist. The fuzzer only fed pure language constructs (operators, comprehensions, unpacking, f-strings, argument forms) with no imports, no dunder access and no I/O, so the intentional restrictions never entered the input set in the first place. Then the triage line was: where does the rejection come from? smolagents keeps its deliberate blocks in named policy (DANGEROUS_MODULES, the authorized_imports check, the dunder guard). Anything that instead fails inside evaluate_ast's structural handling, with no policy involved, is a real bug. All four I found were in the evaluator core, not the policy layer. So an import outside the allowlist is the policy doing its job; {**d} dying is the interpreter failing to handle valid syntax.
The (code, error) hash guard you described in the reply to Zenovay is the part that generalizes best. It doesn't need to understand the error — it just needs to notice that the error isn't changing, which is the signature of a loop rather than a retry.
This connects to something I keep seeing in agent systems: the difference between a retry and a loop is whether the state changes between attempts. An agent that gets a different error each time is learning (or at least exploring). An agent that gets the same error from the same input is stuck, and the stuckness is invisible to the agent because the error message looks like actionable feedback.
The expensive part isn't the token cost — it's that the retry budget is being consumed on a problem the agent can't solve, which means it has less budget left for problems it can. Your fuzz-the-sandbox-with-valid-input approach catches the cause. The hash guard catches the symptom. You need both, because the cause fix only prevents known bugs, and the symptom guard catches any loop regardless of root cause.
Retry versus loop is whether the state changes between attempts. That is the cleanest framing of it I have seen, I'm keeping it. And you put your finger on the real cost: it isn't the tokens, it's that a fixed budget gets spent on an unsolvable problem, starving the solvable ones. Agreed you need both layers. The cause fix only covers known bugs; the hash guard catches any loop regardless of cause. Sandbox fix plus loop detector is belt and suspenders.
Really interesting failure mode. The code was valid, but the misleading error made retrying look like the correct action.
Besides fixing the sandbox, did you consider adding a guard that detects identical code and identical errors across consecutive attempts? That seems like a useful last line of defense against silently burning the entire step budget.
Yes, and I think that guard belongs in the agent loop rather than the sandbox. The executor's job is to run one snippet correctly; the loop is what can see history. A cheap version: hash (code, error) per step and if the same pair repeats N times, stop retrying and escalate (replan, ask for help, or fail fast) instead of feeding the identical lie back to the model. It works as a last line of defense precisely because it doesn't need to understand the error, just notice it isn't changing.
This is a great example of why the most expensive AI agent bugs aren't always crashes they're the ones that produce believable but misleading feedback. Once an agent gets the wrong error signal, retries become wasted tokens instead of recovery. We've run into similar patterns while building production AI workflows at IT Path Solutions, where improving error classification and adding retry guards often had a bigger impact than changing the model itself. Really enjoyed the deep dive into the AST behavior and the reviewer feedback around duck typing - nice catch.
Thanks, glad the AST and duck-typing parts landed. Better error classification over model-swapping matches what I saw here too: the retry loop was a signal problem, not a capability problem.
The AST detail here is the part I keep hitting too: a **mapping spread stores None as the key, and a catch-all isinstance fallthrough turns that into a misleading "NoneType is not supported." What makes it expensive is that the executor hands that wrong message straight back to the model as feedback, so the agent burns retries chasing a null it never wrote. Fuzzing the sandbox with plain valid Python to surface these is smart; did you log retry count per error type, since a handled-but-misleading error never shows up in a crash report?
The fix is interesting, but the broader lesson is even better: fuzz the harness with ordinary valid inputs.
Agent bugs are not always exotic prompt-injection cases. Sometimes they are normal Python syntax hitting a sandbox edge case and turning into a retry loop.
That's the real lesson, yeah. The exotic prompt-injection cases get all the attention, but plain valid syntax hitting a sandbox edge case and turning into a retry loop is more common and much quieter.
This is a great reminder that agent retries need a semantic success check, not just "code ran without exception."
I've seen the same class of bug when a tool returns a soft failure wrapped as a 200 — the loop treats it as progress and burns tokens redoing identical work. Explicit idempotency keys / content hashes on tool results would have saved me more than one bill.