The sandbox in an agent loop: from the model's text to a reliable observation

The sandbox is the part of the write-test-fix loop that takes the code the model wrote and runs it. But calling it "run" sells it short: its real job is to turn the model's text output into a reliable observation, one the loop can base its next turn on. In the previous post that was a three-line runTests function—write the code to a file, run the tests, capture the output—and it worked because the example was a toy. The moment the code comes from a real model, four phases slip in between those three lines that the naive runner skips over, and each one has its own way of breaking.

TL;DR

• The sandbox is the loop's run step, but its job is to turn the model's text into a reliable observation. It goes through four phases: entry, preparation, execution, and isolation.
• Extract the code from the prose, check that it parses, run it with a timeout and in a separate process with a clean environment. Watch out: the timeout only stops code that takes too long—not code that consumes too much—and a subprocess is not a security boundary.
• What moves the needle most is classification: having the sandbox return a status (not a boolean) and keep stdout and stderr separate, so each failure feeds back to the model as a distinct message.

The four phases of the sandbox

For the three-line runTests from the previous post to work, four assumptions have to hold: that the model's reply is code, that the code parses, that it terminates, and that it's safe to run. In the toy example all four hold because you wrote the case to behave well. In production none of them holds on its own, and each broken assumption is a phase of the sandbox: a filter the model's output has to pass before moving on.

Here's the map for the rest of the post. It helps to have the whole thing before diving into each piece:

Phase	What it receives	What can break	The defense
Entry	The model's text	It comes wrapped in prose or a fence	Extract the code
Preparation	The extracted code	It doesn't parse	Syntax check
Execution	The valid code	It doesn't terminate, or runs away	Timeout (with its limits)
Isolation	The running process	It touches your environment	Separate process, clean environment

Seen this way, the sandbox isn't a single block but a pipeline of four filters, and the model's output crosses them one after another:

model's text
      │
 [ Entry ]        extract the code        → prose / fence?
      │
 [ Preparation ]  does it parse?          → syntax_error
      │
 [ Execution ]    run with a timeout      → timeout · test_failed · pass
      │
 [ Isolation ]    separate process, clean env
      │
      ▼
  classified observation that feeds back to the loop

The idea that ties the four phases together, and the one I come back to at the end: in each phase, a distinct outcome has to be able to feed back to the model as a distinct observation. That's what separates a runner that merely executes from one that classifies.

---

Keep reading

That is the first half. The full walkthrough — with the rest of the implementation, the trade-offs and the things that only show up in production — is on my blog:

Read the full post on ramonchancay.me →

Originally published at www.ramonchancay.me/blog/sandbox-running-llm-generated-code.