Don't Trust-Execute AI-Generated Code: A Sandbox Harness for Evaluating Coding Models Safely

In my last post I shared a reproducible harness for comparing free hosted coding models against a local setup. A few people asked the obvious follow-up: when a model writes code for you, where does that code actually run?

If your answer is "on my machine, with my shell, my environment variables, and my network" — that's the part worth fixing before you scale up any AI-assisted workflow. This week there's an active discussion on DEV about what happens when agent tool boundaries fail, and it maps directly onto my own experience benchmarking coding models: the riskiest moment isn't the generation, it's the execution.

This post is the missing piece of that harness: a small, boring, reproducible sandbox so that whatever a model emits — a fix, a migration script, a suspiciously eager curl | bash — runs somewhere that can't hurt you.

The actual problem

When I evaluate a coding model, the loop looks like this:

1. Feed a task prompt to the model.
2. Get back a patch or a script.
3. Run it and measure the result.

Step 3 is where evaluation hygiene and security hygiene collide:

• Generated code can be wrong in destructive ways. A model asked to "clean up build artifacts" once proposed rm -rf ./$BUILD_DIR with BUILD_DIR unset. On my host, that's a bad afternoon.
• Generated code can be over-privileged by default. Dependency installation, network calls, telemetry — a model optimizing for "make the test pass" will happily pull packages from anywhere.
• Benchmarks need determinism anyway. Even ignoring safety, comparing models is meaningless if each run has different network conditions, caches, and leftover state.

A sandbox solves the determinism problem and the trust problem with the same mechanism. That's why I consider it part of the evaluation harness, not a separate security project.

Design constraints

I wanted something that is:

• Reproducible: same image, same flags, same result, on my laptop or a cheap cloud VM.
• Isolated: no network, no host filesystem, capped CPU/memory, non-root user.
• Disposable: one command in, one result out, nothing persists.
• Auditable: the exact artifact that ran is saved alongside the output.

Docker gets me all of this with flags I can put in version control. No exotic tooling.

The artifact: a minimal sandbox runner

The script below takes a file (typically something a model just generated) and runs it inside a locked-down container. It works for Python in this form; extending it to Node is a matter of swapping the image and the run command.

#!/usr/bin/env bash
# sandbox_run.sh — execute an untrusted script with no network, no host mounts,
# capped resources, and a non-root user.
# Usage: ./sandbox_run.sh path/to/candidate.py [timeout_seconds]

set -euo pipefail

CANDIDATE="${1:?usage: sandbox_run.sh <file> [timeout]}"
TIMEOUT="${2:-60}"
IMAGE="python:3.12-slim"   # pin a digest in real pipelines

ABS_PATH="$(realpath "$CANDIDATE")"
BASENAME="$(basename "$CANDIDATE")"

docker run --rm \
  --network none \                 # no egress: no package installs, no exfil
  --read-only \                    # container filesystem is immutable
  --tmpfs /tmp:rw,noexec,nosuid,size=64m \
  --memory 512m --memory-swap 512m \
  --cpus 1.0 \
  --pids-limit 128 \
  --user 1000:1000 \               # never root inside the box
  --cap-drop ALL \                 # no Linux capabilities
  --security-opt no-new-privileges \
  -v "$ABS_PATH":/work/"$BASENAME":ro \  # the candidate, read-only, nothing else
  -w /work \
  "$IMAGE" \
  timeout "$TIMEOUT" python "$BASENAME"

The flags that matter most, and why:

Flag	Failure it prevents
--network none	Model-generated pip install/fetch of arbitrary URLs; data exfiltration
--read-only + --tmpfs /tmp	Persistence tricks, tampering with the runtime
--memory / --cpus / --pids-limit	Fork bombs and accidental infinite allocation ruining the host and the benchmark timing
--user 1000:1000, --cap-drop ALL	Privilege escalation inside the container
-v ...:ro (single file)	Accidental access to your home directory, .env, SSH keys
timeout	Hangs from generated infinite loops

One caveat worth stating plainly: this is a convenience sandbox, not a security boundary against a motivated adversary. Containers share the host kernel. For evaluating "code a model wrote in good faith that might be buggy or sloppy," this is a reasonable bar. For running genuinely hostile code, use a real VM or a microVM (Firecracker, gVisor) instead.

Wiring it into a model-evaluation loop

Here's the pattern I use to grade a model's output without ever executing it on the host. Pseudocode-adjacent Python, trimmed to the skeleton:

import subprocess, json, hashlib, pathlib, time

def grade(candidate_code: str, task_id: str) -> dict:
    run_dir = pathlib.Path("runs") / task_id
    run_dir.mkdir(parents=True, exist_ok=True)
    candidate = run_dir / "candidate.py"
    candidate.write_text(candidate_code)

    started = time.time()
    proc = subprocess.run(
        ["./sandbox_run.sh", str(candidate), "60"],
        capture_output=True, text=True,
    )
    elapsed = time.time() - started

    record = {
        "task_id": task_id,
        "code_sha256": hashlib.sha256(candidate_code.encode()).hexdigest(),
        "exit_code": proc.returncode,   # 124 = timeout killed it
        "elapsed_s": round(elapsed, 3),
        "stdout_tail": proc.stdout[-2000:],
        "stderr_tail": proc.stderr[-2000:],
    }
    (run_dir / "result.json").write_text(json.dumps(record, indent=2))
    return record

Two details I consider non-negotiable:

1. Hash and store the exact artifact you executed. If you can't point to the bytes that produced a result, you don't have a benchmark, you have an anecdote.
2. Grade on exit codes and test output, not on whether the code "looks right." The sandbox makes this safe; take advantage of it.

Where free hosted models and a free server fit

A practical cost note, because this loop is only useful if you can afford to run it often.

I use MonkeyCode for the generation side of this workflow. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Concretely, two things are relevant here: it offers free access to hosted coding models, and a free server option that I use as the machine that runs the sandbox loop. That maps cleanly onto the architecture above:

• Generation (calling the model, collecting candidate code) happens through the free model access — this is the part that would otherwise meter against a paid API on every iteration of a benchmark sweep.
• Execution (the Docker sandbox) lives on the free server, which doubles as isolation: even in a worst-case container escape scenario, the blast radius is a disposable VM with no personal data on it, not my daily-driver laptop.

I deliberately keep the sandbox configuration identical between my laptop and that server, so a result is portable — the "reproducible" part of the harness only holds if the environment is pinned, which is why the image tag and flags live in the repo rather than in my memory.

If you want to try this shape of workflow, MonkeyCode's free tier is one way to get both halves (model access and a runner) without a billing account; any equivalent combination works just as well with the harness above.

A realistic failure this caught

During one comparison run, a model solving a "resize images in this folder" task generated a script that called subprocess.run(["convert", ...]) and, on failure, fell back to downloading a static ImageMagick binary from a hardcoded URL. On my host, pre-sandbox, that would have executed silently and "passed" the task. In the sandbox it failed loudly: --network none blocked the download, the exit code was non-zero, and the stderr_tail in the result JSON showed the attempted fetch.

Two lessons from that one log line: the model's answer was incorrect for my environment in a way a human skim would likely miss, and the correct grade for it was "fail," which only a sandboxed run could assign safely. This is the boundary problem from the agent-security discussion, at the smallest possible scale — and it's exactly where you want to catch it.

Limitations, honestly stated

• Not a hostile-code boundary. As noted: shared kernel. Treat this as protection against buggy/sloppy/generated code, not against exploits.
• --network none changes what you can test. Anything requiring package installation or live APIs needs a different design: a pre-baked image with dependencies installed, or an explicit allowlist proxy. That's more setup, and it reintroduces trust decisions you should make consciously.
• Resource caps distort benchmarks. If you're comparing models on performance-sensitive tasks, the 1-CPU/512MB ceiling is part of your measurement environment. That's fine — arguably good, since it's realistic for cheap CI — but report it.
• Free tiers are free tiers. Availability, quotas, and performance of free hosted models and free servers can change; design the harness so the provider is swappable, because eventually you'll want to swap it.

Who should not use this approach

• Anyone evaluating autonomous multi-step agents that need to install tools, browse, or maintain state across steps — a single-shot sandbox won't model that, and you'll need a heavier environment (dedicated VMs, snapshot/restore).
• Anyone whose threat model includes deliberately malicious inputs — step up to gVisor, Firecracker, or separate hardware.
• Anyone who just wants a quick answer to one coding question — this is harness infrastructure; the payoff only appears when you're running evaluations repeatedly.

Closing

The more code generation you let into your workflow, the more your execution environment becomes the real security boundary. A ~15-line Docker wrapper, pinned and version-controlled, turned my model-comparison harness from "hope the output is benign" into "the output is graded, hashed, and contained." If you're benchmarking or iterating against coding models — free hosted ones included — build the sandbox first. The benchmark numbers you get afterward will be both safer and more believable.