DEV Community

Charlie Zhu
Charlie Zhu

Posted on

A Reproducible Sandbox Loop for AI-Generated Code: Generate, Isolate, Assert

#ai #testing #security #tutorial

AI coding assistants are getting more tool access — shell commands, file writes, network calls. The recent discussions on DEV about agent boundaries failing resonate with a practical problem I keep running into: how do you evaluate code an AI just wrote without trusting it on your own machine?

This article describes a repeatable, boring-on-purpose workflow: generate code with an AI assistant, run it inside a locked-down container, and assert on the output. No trust required. The artifact below is a working harness you can adapt; treat it as a starting point, not production policy.

The problem

When an assistant produces a script, the usual options are:

  1. Read it carefully, then run it locally — slow, and you will eventually miss something.
  2. Run it and hope — fast, and eventually catastrophic.
  3. Never run generated code — which throws away most of the value.

Option 4 is a fixed evaluation loop with a hard isolation boundary. The point is not that containers are a perfect security boundary (they are not — see limitations). The point is that the process becomes reproducible: same prompt, same harness, same assertions, comparable results across models and sessions.

Where the free tier fits

The loop has two costs: model calls and a machine to run untrusted code on. Both can gatekeep experimentation. MonkeyCode currently offers free model access and a free server option, which removes those two gates for this kind of workflow — you can iterate on prompts and run the generated code somewhere that is not your laptop.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A few honest caveats: I am not going to quote specific model names, quotas, or hardware specs here, because availability on free tiers changes and you should check the current offering yourself. The workflow below is tool-agnostic — it works with any assistant that can emit code, and any host where you can run containers. If the free options disappear tomorrow, the harness still stands.

The artifact: a minimal isolation harness

The core idea: every generated artifact runs in a container with no network, a memory cap, a CPU cap, a read-only root filesystem, and a wall-clock timeout. Here is the runner (bash, requires Docker):

#!/usr/bin/env bash
# run_sandboxed.sh — execute an untrusted file in a locked-down container.
# Usage: ./run_sandboxed.sh <file> <image> <timeout_seconds>
set -euo pipefail

FILE="$1"
IMAGE="${2:-python:3.12-slim}"
TIMEOUT="${3:-10}"

if [ ! -f "$FILE" ]; then
  echo "No such file: $FILE" >&2
  exit 64
fi

# --network none   : no exfiltration, no package installs mid-run
# --read-only      : root fs immutable; /tmp is a small tmpfs
# --memory / --cpus: resource caps so runaway loops don't eat the host
# --pids-limit     : blunts fork bombs
# timeout          : hard wall-clock kill
exec timeout "$TIMEOUT" docker run --rm \
  --network none \
  --read-only \
  --tmpfs /tmp:rw,noexec,nosuid,size=64m \
  --memory 256m \
  --cpus 0.5 \
  --pids-limit 64 \
  --cap-drop ALL \
  --security-opt no-new-privileges \
  -v "$(realpath "$FILE")":/work/task:ro \
  "$IMAGE" /work/task
Enter fullscreen mode Exit fullscreen mode

And a tiny evaluation driver. The point of this file is not the code — it is that the assertions are written before you look at the generated solution, so you are testing the model's output rather than rationalizing it:

# eval_loop.py — proposal-grade harness; extend for your own tasks.
import json, subprocess, sys
from pathlib import Path

TASKS = json.loads(Path("tasks.json").read_text())
# tasks.json: [{"id": "fizzbuzz", "prompt": "...", "stdin": "", "expect": "1\n2\nFizz\n..."}]

results = []
for task in TASKS:
    candidate = Path(f"gen/{task['id']}.py")
    if not candidate.exists():
        results.append({"id": task["id"], "status": "missing"})
        continue
    proc = subprocess.run(
        ["./run_sandboxed.sh", str(candidate), "python:3.12-slim", "10"],
        input=task["stdin"], capture_output=True, text=True,
    )
    ok = proc.returncode == 0 and proc.stdout == task["expect"]
    results.append({
        "id": task["id"],
        "status": "pass" if ok else "fail",
        "stderr_tail": proc.stderr[-300:] if not ok else "",
    })

print(json.dumps(results, indent=2))
Enter fullscreen mode Exit fullscreen mode

The loop in practice:

  1. Write the task spec and expected output yourself, first.
  2. Have the assistant (in my case, running through MonkeyCode's free model access) produce gen/<task>.py.
  3. Run the driver. Every execution is isolated, time-boxed, and resource-capped.
  4. Record pass/fail. Now you can meaningfully compare prompt variants or models, because the measurement is stable.

Which sandbox level do you actually need?

Threat you care about Minimum sensible setup The harness above enough?
Accidental damage (rm in wrong dir, huge allocation) Restricted container, resource caps Yes
Nosy generated code (reading env vars, home dir) No network + dropped capabilities + clean image Mostly
Deliberately adversarial code Separate VM or host you can rebuild No — escalate isolation
Compliance / customer data in scope Dedicated infra, audit trail, legal review Absolutely not

Limitations

  • Containers share the host kernel. A kernel exploit escapes this sandbox. For genuinely adversarial code, use a separate VM or a disposable machine — which is one reason running this on a free remote server instead of your daily driver is attractive.
  • No network means no pip installs. If generated code needs dependencies, prebuild an image containing them rather than opening the network.
  • Timeouts cut both ways. A 10-second limit will kill legitimate slow tasks; tune per task, and treat timeouts as a signal, not noise.
  • The harness does not judge code quality, only behavior against your assertions. Tests you wrote badly will pass bad code.

Who should skip this

If your assistant only ever produces single functions you fully review before use, this is overhead. If you need to run generated code that must access production credentials or live APIs, do not adapt this harness — that is a secrets-management and permissions problem, not a sandboxing problem. And if your organization has an approved execution environment for AI output, use that instead.

Closing thought

The useful shift is treating AI output like any other untrusted artifact: fixed inputs, isolated execution, assertions written in advance. Once the loop is reproducible, model and prompt comparisons stop being vibes. If you want to try this without paying for model calls or a throwaway host while you experiment, MonkeyCode's free model access and free server option are a reasonable place to start — but the harness matters more than the vendor.

Top comments (0)