DEV Community

Himanshu Kumar
Himanshu Kumar Subscriber

Posted on

The smolagents sandbox broke 'a, *b = list', one of Python's most common lines

#devchallenge #bugsmash #ai #python

Summer Bug Smash: Clear the Lineup 🐛🛹

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Fourth entry, and the fuzzer that found entry 3 is still paying out. Same failure family, different corner of the language: valid Python the sandbox refuses to run, with an error that lies about why.

The one-liner every developer writes 

best, *rest = scores
Enter fullscreen mode Exit fullscreen mode

Splitting a list into "the first one" and "the rest" is about as ordinary as Python gets. It is PEP 3132, shipped in 2008. Under smolagents' sandbox it fails with:

InterpreterError: Cannot unpack tuple of wrong size
Enter fullscreen mode Exit fullscreen mode

There is no wrong size. scores has four items and the pattern accepts any length of two or more. The message describes a problem that does not exist, so the agent does the only thing a faithful agent can do with a message that is already a lie: it retries the identical, valid code.

And it is not just the starred form. All of these are standard Python and all of them were broken:

first, *rest = "hello"   # Cannot unpack tuple of wrong size
a, b = "hi"              # Cannot unpack non-tuple value
[a, b] = [1, 2]          # silently assigns nothing at all
Enter fullscreen mode Exit fullscreen mode

Why it happens

smolagents runs model-generated code in its own AST interpreter, and assignment targets go through one function, set_value. It handled exactly one shape: a fixed-size ast.Tuple.

elif isinstance(target, ast.Tuple):
    if not isinstance(value, tuple):
        if hasattr(value, "__iter__") and not isinstance(value, (str, bytes)):
            value = tuple(value)
        else:
            raise InterpreterError("Cannot unpack non-tuple value")
    if len(target.elts) != len(value):
        raise InterpreterError("Cannot unpack tuple of wrong size")
    ...
Enter fullscreen mode Exit fullscreen mode

Three separate holes hide in those few lines:

  1. Starred targets. a, *b has two target elements but the value has three items, so len(target.elts) != len(value) fires. The * marker is never even looked at.
  2. Strings and bytes. They are explicitly excluded from the iterable path, so a, b = "hi" is rejected even though CPython unpacks strings happily.
  3. List-pattern targets. [a, b] = ... is an ast.List, not an ast.Tuple, so it matches no branch, falls through, and silently assigns nothing. No error, no values, the worst kind of quiet.

The Sentry angle: count the retries

Same lesson as the last two entries, and Sentry keeps making it visible. The error is handled: the agent catches it and feeds it back to the model as guidance. But the guidance is wrong, so the model cannot act on it, so it loops. One bug, one misleading message, three identical failures burning three steps:

Sentry issue showing InterpreterError Cannot unpack tuple of wrong size, 3 events, environment before, transaction CodeAgent list split task

Three events on one issue is the retry loop made countable. Without it you would see a slow run, not a stuck one. Sentry's Seer read the same event and landed on the exact cause:

smolagents' custom Python interpreter does not support starred unpacking (e.g. best, *rest = scores), treating it as a fixed-size tuple unpack. set_value checks len(target.elts) != len(value) and raises, without handling ast.Starred targets.

The fix

Rewrite the branch to match CPython instead of guessing:

elif isinstance(target, (ast.Tuple, ast.List)):
    elts = target.elts
    starred = [i for i, e in enumerate(elts) if isinstance(e, ast.Starred)]
    if len(starred) > 1:
        raise InterpreterError("multiple starred expressions in assignment")
    if not hasattr(value, "__iter__"):
        raise InterpreterError(f"cannot unpack non-iterable {type(value).__name__} object")
    values = list(value)
    if starred:
        i = starred[0]
        n_after = len(elts) - i - 1
        if len(values) < i + n_after:
            raise InterpreterError(
                f"not enough values to unpack (expected at least {i + n_after}, got {len(values)})"
            )
        split = len(values) - n_after
        # assign head, then the starred target gets the middle as a list, then the tail
        ...
    else:
        if len(values) < len(elts):
            raise InterpreterError(f"not enough values to unpack (expected {len(elts)}, got {len(values)})")
        if len(values) > len(elts):
            raise InterpreterError(f"too many values to unpack (expected {len(elts)})")
        ...
Enter fullscreen mode Exit fullscreen mode

Any iterable now unpacks, a single starred target absorbs the surplus into a list in any position (a, *b, *a, b, a, *b, c), list-pattern targets work, and the size errors read exactly like CPython's, so when the model genuinely does pass the wrong number of values it gets an actionable message instead of a dead end.

After 

app: step 1 ok, output = (90, [82, 71, 65])
Enter fullscreen mode Exit fullscreen mode

One step. No loop. best is 90, rest is the tail, the way the model expected all along.

Numbers

  • 4 valid unpacking forms fixed: starred targets, string unpacking, list-pattern targets, and the CPython error messages
  • Reproduced on current main and 1.26.0; 3 wasted agent steps per occurrence, visible only because Sentry counts events
  • 14 new tests plus one existing test updated to the improved message
  • 411 passing, ruff clean

Links

The through-line across all four entries has not changed: the dangerous agent bugs are not the loud crashes, they are the polite, well-handled, wrong messages that let the model fail on repeat. Fuzz the sandbox with ordinary valid code, and count your events.

Top comments (1)

Collapse
 
xinyangwuethz profile image
Xinyang Wu •

I think I need to track (tool_name, normalized_args, error_type) in my tool loop to prevent same error from happening multiple times.Thanks