I have pasted agent-generated code straight into my terminal without reading it. More times than I want to admit.
Usually it’s fine. I build something most weeks and the loop is always the same. Describe what I want, let something write it, run it, find out what breaks.
The step I skip past every time is “run it.” On my own machine. With my project files sitting right there and my API keys loaded in the shell.
What that code can actually do
The generated script that reads a .csv and writes a summary is also a Python process with your user’s permissions. While it runs, it can:
Read any file your user can read, including .env, ~/.ssh, and your browser’s cookie store
Write or delete anywhere in your home directory
Open outbound network connections to anywhere
Pull packages at runtime, which means code you never saw gets executed too
Spin until it exhausts memory or disk
None of that requires malice. A model that hallucinates a path and a model that targets one produce the same rm. The failure mode you should plan for isn’t an attacker. It’s a confident wrong guess running with your credentials.
This mattered less when the loop had a human reviewing every line before execution. It matters a lot more when the agent is the one deciding to run it, in a loop, unattended.
The options people reach for
Run it locally and read it first. Works until the code is 200 lines, or the agent is running ten steps and you’re not watching step seven. Reviewing every line defeats the point of delegating.
Docker. Real isolation, and the right instinct. But you own the Dockerfile, the mounted volumes, the network mode, and the resource limits. Most people mount the project directory read-write and run with default networking, which gives back most of what they were trying to contain. It also doesn’t solve anything if the agent is the one generating the container config.
A throwaway VM. Good isolation, slow to start, annoying to wire into an agent loop that needs to execute something in under a second and read the result back.
A hosted sandbox. Execution happens somewhere that isn’t your filesystem, with boundaries set before the code arrives rather than configured per run.
What I’d want from one
Whatever you pick, the questions are the same:
What can the process read, and is my source tree in that set
Can it reach the network, and can I turn that off per run
What are the memory, CPU, and wall clock limits, and what happens when it hits them
What do I get back when it fails on step six of nine, and do I know it was step six
Are credentials scoped to the run, or is it inheriting my whole environment
That last one catches people. An agent given “the API key” usually gets the same key you use for everything, with the same scope, because that’s the key that was sitting in the environment.
Try this on Saturday
CreateOS is running a one-day hackathon on exactly this problem. Build an AI agent that writes its own code and executes it in a CreateOS Sandbox, with your stack set up through Stripe Projects.
Sat Oct 17, 12 to 7 PM PT, online on Discord. Solo or teams of up to 3, one person registers per team. $10,000 prize pool. Kickoff is a Stripe Projects walkthrough from the Stripe team, then a Sandbox brief from the CTO, @pratikbin. Demos and awards at 6 PM PT.
Spots are approved one by one, so apply early and say what you want to build.
If you’re already running agent-generated code, where does it execute right now, and what would it reach if it went wrong?
Top comments (0)