DEV Community

Himanshu Kumar
Himanshu Kumar Subscriber

Posted on

One line of math froze my AI agent forever. The timeout watched and did nothing.

#devchallenge #bugsmash #python #ai

Summer Bug Smash: Clear the Lineup 🐛🛹

This is my second entry for the DEV x Sentry Bug Smash challenge. Entry #1 was a crash with a confusing error message. This one is the opposite and it is scarier: no crash, no message, no event. Just silence.

The bug that sends you nothing

smolagents runs LLM-generated Python in a sandboxed executor with a timeout. Issue #2473 claims that one line of model-generated math defeats it completely:

from smolagents.local_python_executor import LocalPythonExecutor

executor = LocalPythonExecutor(additional_authorized_imports=[], timeout_seconds=2)
executor.send_tools({})
executor("10 ** 10 ** 8")  # 2 second timeout. Should be fine, right?
Enter fullscreen mode Exit fullscreen mode

I ran this with a 2 second timeout and a faulthandler bomb set for 20 seconds. The timeout never fired. The process sat frozen until the external kill. An agent that generates this expression (and "compute this huge number" is exactly the kind of thing agents try) freezes its host process forever.

Zero comments on the issue, zero PRs. Mine now.

Why the timeout lies to you

smolagents' timeout is thread based: a worker thread runs the code, the main thread waits in future.result(timeout=2). That design is fine for almost everything, because CPython switches threads between bytecode instructions.

But 10 ** 10 ** 8 is not "almost everything". CPython computes arbitrary precision **, << and * inside a single C call that holds the GIL from start to finish. No bytecode boundary, no thread switch, no timeout. The result would have about 400 million bits. The computation takes somewhere between minutes and hours. Your watchdog needs the GIL to wake up, and it never gets it.

The faulthandler dump made it concrete, and it was worse than the issue described:

Thread 0x16d1f3000 (worker):
  File "local_python_executor.py", line 753 in evaluate_binop   <- computing the pow

Thread 0x1f00d5e80 (main):
  File "threading.py", line 999 in start
  File "concurrent/futures/thread.py", line 180 in submit       <- never returned!
Enter fullscreen mode Exit fullscreen mode

The main thread was still stuck inside ThreadPoolExecutor.submit. It never even reached future.result. The 2 second timer never armed at all.

The existing MAX_OPERATIONS guard (10 million AST operations) does not help either. This is a handful of AST nodes. The entire cost lives inside one of them.

The Sentry angle: monitoring for absence

Entry #1 was about a noisy failure. This bug is the opposite. I pointed a fresh Sentry project at a simulated agent run on the unpatched PyPI release (1.26.0) and got the most unsettling result possible: nothing. No event, no transaction, an empty project. The process was frozen mid-transaction and the SDK never got a chance to flush.

The lesson: for freeze-class bugs you need a supervisor. I added a small watchdog process that gives the worker 25 seconds, then kills it and reports what it saw, with the worker's faulthandler stack attached as evidence:

watchdog: starting worker (before) with 25s budget
worker: smolagents 1.26.0
worker: step 1 executing 'result = 10 ** 10 ** 8'...
watchdog: worker FROZE, killed it, reporting to Sentry
Enter fullscreen mode Exit fullscreen mode

The resulting Sentry issue carries the whole story in one place: environment tagged before, handled by the supervisor, with the frozen frame (evaluate_binop, line 753 of local_python_executor.py) sitting in the attached worker_faulthandler_stack extra.

Sentry issue AgentFrozenError, resolved, environment before, reported by the watchdog after the worker froze

Sentry's Seer ran root cause analysis on that issue and independently landed on the same conclusion: signal and thread interruption need the GIL, and a single C-level big-int operation never releases it.

Seer's verdict, verbatim:

CodeAgent's 2s timeout uses Python signal-based interruption, which cannot fire during uninterruptible C-level big-int operations that hold the GIL. [...] A single large big-integer arithmetic operation runs entirely as one C-level call that holds the GIL continuously without yielding. CPython cannot deliver signals or switch threads during an uninterruptible C extension call, so no timeout callback fires for the duration of that operation.

Sentry Seer Autofix root cause panel identifying the GIL-holding big-int operation as the reason the timeout never fired

The fix: you cannot interrupt it, so refuse to start it

Killing the computation mid-flight is impossible from Python. But predicting the damage is O(1). Before executing **, << or * on integers, the executor now estimates the result's bit length from the operands' bit lengths:

if op == "**":
    estimated_bits = left.bit_length() * right       # upper bound
elif op == "<<":
    estimated_bits = left.bit_length() + right
elif op == "*":
    estimated_bits = left.bit_length() + right.bit_length()
Enter fullscreen mode Exit fullscreen mode

Above 1 million bits (about 300k digits, still generous) it raises an informative InterpreterError:

Operation '**' would produce an integer of around 400000000 bits, exceeding
the maximum of 1000000 bits allowed. Use smaller operands, or
pow(base, exp, mod) for modular exponentiation.
Enter fullscreen mode Exit fullscreen mode

That message matters. The agent surfaces it to the model, and the model can actually act on it: use pow(base, exp, mod), which stays unrestricted because modular exponentiation is fast and legitimate. The agent recovers on the next step instead of hanging the host.

The after run on the patched build: the guard rejects the expression in 0.0 seconds, the error lands in Sentry as a normal actionable issue and step 2 executes fine.

watchdog: starting worker (after) with 25s budget
worker: smolagents 1.27.0.dev0
worker: step 1 error surfaced to the model: InterpreterError: ...
worker: step 2 executing '2 + 2'...
worker: step 2 ok
worker: DONE
watchdog: worker exited with code 0
Enter fullscreen mode Exit fullscreen mode

Sentry issue showing the informative InterpreterError captured on the patched build, environment after, transaction CodeAgent math task, resolved

A robot reviewed my robot fix

Minutes after I opened the PR, OpenAI's Codex reviewer flagged a real hole: my guard checked type(x) is int, which lets bool and int subclasses slip through. True << 10**9 and class BigInt(int) still reached the uninterruptible C calls. Fixed with isinstance, added both as regression tests. AI found the bug class, AI fixed it, AI reviewed the fix. I just steered.

Numbers

  • Freeze reproduced at 20+ seconds (would have run for hours), external kill required
  • Fix rejects the same expression in 0.0 seconds
  • 9 explosive patterns blocked: **, <<, chained *, all augmented forms, pow(a, b), bool and int subclass variants
  • 10 legitimate operations verified untouched: 100! via repeated *=, pow(7, 2**64, 97), float pow, 1 ** 10**9
  • 19 new tests, full test file 416 passed, ruff clean
  • The blocked-pattern tests hang forever on unpatched main. I verified that the honest way, with a stash and a kill switch.

Links

The scariest bugs are not the ones that page you at 3am. They are the ones that make sure nothing ever pages you at all. Instrument for silence.

Top comments (6)

Collapse
 
nazar-boyko profile image
Nazar Boyko •

Does the guard also catch sequence repetition? Something like ('a' * 10**6) * 10**4 builds the result in one C call too, so I'd expect it to hold the GIL the same way the big ints do, just with memory as the bomb instead of CPU.

Collapse
 
himanshu_748 profile image
Himanshu Kumar •

Great catch, and you're exactly right. The guard only checked int operands, so str/bytes/list/tuple repetition slipped straight through and could freeze the process the same way, with memory as the bomb instead of CPU. I just pushed a follow-up commit to the PR that extends the check to sequence repetition (seq * n in either operand order, plus *=), capping the result element count, with tests for the bypass you described. Thanks for spotting it.

Collapse
 
mads_hansen_27b33ebfee4c9 profile image
Mads Hansen •

Excellent example of a freeze-class failure. The bit-length estimate is the right fix for these known operators, but it should not become the only safety boundary. An executor running model-generated code still needs process isolation, a wall-clock supervisor outside that process, CPU/memory limits (rlimits or cgroups), and reliable kill/reap behavior. That bounds the next pathological C extension or memory bomb that static guards do not predict. For monitoring silence, I would also emit a correlated “started” event and require exactly one terminal outcome; a missing terminal event then becomes an alertable fact rather than an empty dashboard.

Collapse
 
himanshu_748 profile image
Himanshu Kumar •

Completely agree. The bit-length estimate is a cheap in-process mitigation, not a sandbox. smolagents says the same about the local executor: for untrusted model output you want the docker or remote executor plus an out-of-process wall-clock supervisor and cgroup CPU/memory limits, which is the watchdog role the before-run in the post demonstrates. This guard just stops the trivial single-expression freeze from taking the host down before any of that kicks in.

Collapse
 
krupali_gadhiya profile image
Krupali Gadhiya •

The "instrument for silence" takeaway is something I'll remember. Great lesson beyond just this bug.

Collapse
 
himanshu_748 profile image
Himanshu Kumar •

Thanks, that line was the whole reason I wanted to write it up. Monitoring for the absence of signal is underrated.