This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.
Third entry in the DEV x Sentry Bug Smash. Entry 1 was a crash with a confusing message. Entry 2 was a freeze the timeout could not catch. This one is quieter and sneakier: valid Python that the sandbox rejects with an error pointing at the wrong thing entirely.
When the open issues run out, fuzz
By entry 3 every obvious open smolagents bug was already claimed or had a competing PR. So instead of reading the issue tracker I pointed a small fuzzer at the piece of smolagents that runs the most untrusted code: LocalPythonExecutor, the sandbox that executes model-generated Python.
The method is boring and effective: feed it ordinary, valid Python one snippet at a time, and flag anything that raises InterpreterError. Valid Python that the sandbox refuses to run is, by definition, a bug, because the model writes valid Python and expects it to work.
That surfaced four unreported bugs in one afternoon. This post is about the one I shipped: dict unpacking.
The bug
config = {**{"temperature": 0.7, "max_tokens": 512}, "top_p": 0.9}
Merging dicts with ** is one of the most common things an LLM writes. Under smolagents it fails with:
InterpreterError: NoneType is not supported.
There is no None anywhere in that line. The message sends you looking for a null value that does not exist.
Why it happens
In Python's AST, a dict literal keeps its keys and values in two parallel lists. For a normal entry the key is an AST node. For a **mapping spread entry, the key is literally None, a signal that says "this is a spread, not a key/value pair."
smolagents evaluated every key by walking expression.keys and calling evaluate_ast(key, ...) on each one. When the key is None, that call falls through every isinstance branch to the catch-all raise InterpreterError(f"{type} is not supported"). So the spread marker got evaluated as if it were an expression, and the model got blamed for a None it never wrote.
Why silence is the expensive part
Here is the part the Sentry view made obvious. The error is handled: the agent catches it and feeds it back to the model as "here is what went wrong, try again." But the message names NoneType, and the model's code has no None, so the model cannot act on it. It retries the exact same valid syntax. And again. Every step burns a real LLM call and a slot in the step budget until the run gives up.
One bug, one misleading message, three identical failures:
Three events on a single issue is not noise. It is the agent stuck in a loop, and without Sentry counting the events you would never see the loop, only a run that quietly underperformed. Sentry's Seer read the same event and reached the exact root cause:
smolagents' LocalPythonExecutor doesn't handle dict unpacking (
**) syntax: None keys in ast.Dict cause an unsupported type error. [...]evaluate_ast(None, ...)matches no isinstance branch and falls to the else clause. The interpreter raises InterpreterError: NoneType is not supported, the agent retries with identical code, burning steps in a loop.
The fix
Evaluate the dict pairwise instead of evaluating keys blindly. A None key means "merge this mapping":
result = {}
for key_node, value_node in zip(expression.keys, expression.values):
if key_node is None:
value = evaluate_ast(value_node, *common_params)
if not hasattr(value, "keys"):
raise InterpreterError(f"'{type(value).__name__}' object is not a mapping")
result.update(value)
else:
key = evaluate_ast(key_node, *common_params)
result[key] = evaluate_ast(value_node, *common_params)
return result
This matches CPython exactly: spreads merge in order, later keys win, and unpacking a non-mapping raises 'list' object is not a mapping.
A reviewer caught my fix being too strict
I first gated the spread on isinstance(value, Mapping). Minutes after the PR opened, OpenAI's Codex reviewer flagged it (P2): CPython does not require the Mapping ABC, it only requires an object with a keys() method. Since the sandbox lets users define their own classes, a duck-typed mapping with keys() and __getitem__() would have been wrongly rejected. I switched the check to hasattr(value, "keys") and added a test for exactly that case. AI wrote the code, AI reviewed the code, I kept score.
After
On the patched build the same line just runs:
app: step 1 ok, config = {'temperature': 0.7, 'max_tokens': 512, 'top_p': 0.9}
One step, no loop, no phantom None.
Numbers
- 4 unreported bugs found by fuzzing valid Python through the sandbox; this is the first fix
- Misleading
NoneTypeerror reproduced on currentmainand 1.26.0 - 3 wasted agent steps per occurrence, visible only because Sentry counts events
- 9 new tests: spreads, double spreads, override order both ways, a duck-typed mapping class, empty spread, non-mapping rejection
- 406 passing, ruff clean
Links
- Issue: https://github.com/huggingface/smolagents/issues/2552
- PR: https://github.com/huggingface/smolagents/pull/2553
The pattern across all three entries: the worst agent bugs do not throw a red stack trace at you. They hand the model a plausible-but-wrong message and let it fail politely, on repeat. Count your events.
Top comments (14)
"They hand the model a plausible-but-wrong message and let it fail politely, on repeat." That's the truest thing I've read about agent failure this month. The red stack trace is mercy — it's the polite, well-handled, misleading error that eats your afternoon.
I hit the exact structure outside Python. On my server a firewall rewrites a legitimate 4xx into a 405, so my agent gets handed "405" — plausible, wrong, pointing at the wrong cause. Left alone it will "fix" the routing three times, same as your model retrying valid syntax, because the message names a problem that isn't the problem. The retry loop isn't the model being dumb; it's the model faithfully acting on a message that was already a lie.
Two things I'm taking from your fix. One: fuzz the sandbox with ordinary valid input and treat every false rejection as a bug — stealing that as a test for my own guardrails. Two: this week I gave my agents a rule that's basically your lesson in prose — when you hit a familiar failure signal, translate what it means in THIS environment before you act on it, because a politely wrong message will happily let you loop forever. You found the version that burns tokens; mine burns a junior's afternoon. Same silent killer.
thanks man
The 405 rewrite is the perfect parallel: the message names a problem that isn't the problem, and a faithful agent will "fix" the wrong thing on loop. Love your rule of translating a familiar failure signal into what it means in THIS environment before acting on it. Mine burns tokens, yours burns a junior's afternoon, same silent killer. And yes, fuzz your guardrails with ordinary valid input, every false rejection is a bug.
Already did it — your last line went straight into code the same afternoon. I planted six known-good snippets into my scanner's test suite: escaped output, a prepared query, a hashed password, the safe patterns my regex rules could plausibly misfire on. Ran it: caught everything it should, false-flagged none of the six. Now if I ever "tighten" a rule and it starts biting clean code, that test goes red instead of me hearing it from an annoyed teammate.
The part I didn't expect: one snippet — the string "eval(" sitting inside a comment — does still trip it. Not a regression, just an honest known blind spot now, labeled as one instead of pretending it isn't there. Your framing turned a vague "should test that" into a standing tripwire. Thanks man — this one earned its keep.
The rule that any rejected valid Python counts as a bug seems like it would also flag things the sandbox blocks on purpose, like imports outside the authorized list. How did you tell intentional restrictions apart from real bugs when triaging, an allowlist of expected errors or just by hand?
Good question, and it was mostly by construction rather than an allowlist. The fuzzer only fed pure language constructs (operators, comprehensions, unpacking, f-strings, argument forms) with no imports, no dunder access and no I/O, so the intentional restrictions never entered the input set in the first place. Then the triage line was: where does the rejection come from? smolagents keeps its deliberate blocks in named policy (DANGEROUS_MODULES, the authorized_imports check, the dunder guard). Anything that instead fails inside evaluate_ast's structural handling, with no policy involved, is a real bug. All four I found were in the evaluator core, not the policy layer. So an import outside the allowlist is the policy doing its job; {**d} dying is the interpreter failing to handle valid syntax.
This is a great example of why the most expensive AI agent bugs aren't always crashes they're the ones that produce believable but misleading feedback. Once an agent gets the wrong error signal, retries become wasted tokens instead of recovery. We've run into similar patterns while building production AI workflows at IT Path Solutions, where improving error classification and adding retry guards often had a bigger impact than changing the model itself. Really enjoyed the deep dive into the AST behavior and the reviewer feedback around duck typing - nice catch.
Thanks, glad the AST and duck-typing parts landed. Better error classification over model-swapping matches what I saw here too: the retry loop was a signal problem, not a capability problem.
Really interesting failure mode. The code was valid, but the misleading error made retrying look like the correct action.
Besides fixing the sandbox, did you consider adding a guard that detects identical code and identical errors across consecutive attempts? That seems like a useful last line of defense against silently burning the entire step budget.
Yes, and I think that guard belongs in the agent loop rather than the sandbox. The executor's job is to run one snippet correctly; the loop is what can see history. A cheap version: hash (code, error) per step and if the same pair repeats N times, stop retrying and escalate (replan, ask for help, or fail fast) instead of feeding the identical lie back to the model. It works as a last line of defense precisely because it doesn't need to understand the error, just notice it isn't changing.
The fix is interesting, but the broader lesson is even better: fuzz the harness with ordinary valid inputs.
Agent bugs are not always exotic prompt-injection cases. Sometimes they are normal Python syntax hitting a sandbox edge case and turning into a retry loop.
That's the real lesson, yeah. The exotic prompt-injection cases get all the attention, but plain valid syntax hitting a sandbox edge case and turning into a retry loop is more common and much quieter.
The AST detail here is the part I keep hitting too: a **mapping spread stores None as the key, and a catch-all isinstance fallthrough turns that into a misleading "NoneType is not supported." What makes it expensive is that the executor hands that wrong message straight back to the model as feedback, so the agent burns retries chasing a null it never wrote. Fuzzing the sandbox with plain valid Python to surface these is smart; did you log retry count per error type, since a handled-but-misleading error never shows up in a crash report?
Some comments may only be visible to logged-in visitors. Sign in to view all comments.