This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.
Third entry in the DEV x Sentry Bug Smash. Entry 1 was a cras...
For further actions, you may consider blocking this person and/or reporting abuse
"They hand the model a plausible-but-wrong message and let it fail politely, on repeat." That's the truest thing I've read about agent failure this month. The red stack trace is mercy — it's the polite, well-handled, misleading error that eats your afternoon.
I hit the exact structure outside Python. On my server a firewall rewrites a legitimate 4xx into a 405, so my agent gets handed "405" — plausible, wrong, pointing at the wrong cause. Left alone it will "fix" the routing three times, same as your model retrying valid syntax, because the message names a problem that isn't the problem. The retry loop isn't the model being dumb; it's the model faithfully acting on a message that was already a lie.
Two things I'm taking from your fix. One: fuzz the sandbox with ordinary valid input and treat every false rejection as a bug — stealing that as a test for my own guardrails. Two: this week I gave my agents a rule that's basically your lesson in prose — when you hit a familiar failure signal, translate what it means in THIS environment before you act on it, because a politely wrong message will happily let you loop forever. You found the version that burns tokens; mine burns a junior's afternoon. Same silent killer.
thanks man
The 405 rewrite is the perfect parallel: the message names a problem that isn't the problem, and a faithful agent will "fix" the wrong thing on loop. Love your rule of translating a familiar failure signal into what it means in THIS environment before acting on it. Mine burns tokens, yours burns a junior's afternoon, same silent killer. And yes, fuzz your guardrails with ordinary valid input, every false rejection is a bug.
Already did it — your last line went straight into code the same afternoon. I planted six known-good snippets into my scanner's test suite: escaped output, a prepared query, a hashed password, the safe patterns my regex rules could plausibly misfire on. Ran it: caught everything it should, false-flagged none of the six. Now if I ever "tighten" a rule and it starts biting clean code, that test goes red instead of me hearing it from an annoyed teammate.
The part I didn't expect: one snippet — the string "eval(" sitting inside a comment — does still trip it. Not a regression, just an honest known blind spot now, labeled as one instead of pretending it isn't there. Your framing turned a vague "should test that" into a standing tripwire. Thanks man — this one earned its keep.
The rule that any rejected valid Python counts as a bug seems like it would also flag things the sandbox blocks on purpose, like imports outside the authorized list. How did you tell intentional restrictions apart from real bugs when triaging, an allowlist of expected errors or just by hand?
Good question, and it was mostly by construction rather than an allowlist. The fuzzer only fed pure language constructs (operators, comprehensions, unpacking, f-strings, argument forms) with no imports, no dunder access and no I/O, so the intentional restrictions never entered the input set in the first place. Then the triage line was: where does the rejection come from? smolagents keeps its deliberate blocks in named policy (DANGEROUS_MODULES, the authorized_imports check, the dunder guard). Anything that instead fails inside evaluate_ast's structural handling, with no policy involved, is a real bug. All four I found were in the evaluator core, not the policy layer. So an import outside the allowlist is the policy doing its job; {**d} dying is the interpreter failing to handle valid syntax.
This is a great example of why the most expensive AI agent bugs aren't always crashes they're the ones that produce believable but misleading feedback. Once an agent gets the wrong error signal, retries become wasted tokens instead of recovery. We've run into similar patterns while building production AI workflows at IT Path Solutions, where improving error classification and adding retry guards often had a bigger impact than changing the model itself. Really enjoyed the deep dive into the AST behavior and the reviewer feedback around duck typing - nice catch.
Thanks, glad the AST and duck-typing parts landed. Better error classification over model-swapping matches what I saw here too: the retry loop was a signal problem, not a capability problem.
Really interesting failure mode. The code was valid, but the misleading error made retrying look like the correct action.
Besides fixing the sandbox, did you consider adding a guard that detects identical code and identical errors across consecutive attempts? That seems like a useful last line of defense against silently burning the entire step budget.
Yes, and I think that guard belongs in the agent loop rather than the sandbox. The executor's job is to run one snippet correctly; the loop is what can see history. A cheap version: hash (code, error) per step and if the same pair repeats N times, stop retrying and escalate (replan, ask for help, or fail fast) instead of feeding the identical lie back to the model. It works as a last line of defense precisely because it doesn't need to understand the error, just notice it isn't changing.
The AST detail here is the part I keep hitting too: a **mapping spread stores None as the key, and a catch-all isinstance fallthrough turns that into a misleading "NoneType is not supported." What makes it expensive is that the executor hands that wrong message straight back to the model as feedback, so the agent burns retries chasing a null it never wrote. Fuzzing the sandbox with plain valid Python to surface these is smart; did you log retry count per error type, since a handled-but-misleading error never shows up in a crash report?
The fix is interesting, but the broader lesson is even better: fuzz the harness with ordinary valid inputs.
Agent bugs are not always exotic prompt-injection cases. Sometimes they are normal Python syntax hitting a sandbox edge case and turning into a retry loop.
That's the real lesson, yeah. The exotic prompt-injection cases get all the attention, but plain valid syntax hitting a sandbox edge case and turning into a retry loop is more common and much quieter.