Quarantine the Output: Running Model-Generated Code Without Handing It Your Laptop

Most pipelines that evaluate LLM output share an awkward secret: at some point, the pipeline executes whatever the model emitted. Sometimes that's fine. Sometimes the model hallucinates a shutil.rmtree with the wrong path, or faithfully follows an instruction hidden in a poisoned prompt, and now the thing running on your machine is doing something you never asked for.

I hit this while wiring up a small offline evaluation loop: prompt a model, save the completion, run it, compare the result to a known-good answer. The scoring part was easy. The part that kept me up was step three — run it — because my first version was literally subprocess.run(["python", snippet]) on the same box where my dotfiles, tokens, and SSH keys live. That's not a harness, that's a trust fall.

This post is the fix I landed on: a two-layer quarantine you can reproduce in an afternoon, plus an honest accounting of what it does not protect against.

Start from what actually goes wrong

Skip the Hollywood scenarios. When generated code hurts you, it's almost always one of these:

• It never finishes. A while-loop the model swore would converge. This is 90% of incidents and it's pure accident.
• It touches the wrong files. Writes into the wrong directory, reads something sensitive and prints it to stdout you then log somewhere.
• It talks to the network. Usually innocuous (pip install, a stray requests.get), but it's also the exfiltration channel if the prompt was adversarial.
• It eats the box. Memory ballooning until the OOM killer starts shooting your other processes.

A targeted kernel exploit is not on this list. If you genuinely face that, you need dedicated infrastructure and people whose job title includes "security." Everything below is aimed at the realistic accidents-plus-mild-malice zone, and I'll mark exactly where the line is.

Picking an isolation posture

I think about this as four postures, ordered by effort:

Posture	What it is	Contains accidents	Contains malice	Cost
Bare	Run it directly, hope	❌	❌	nothing
Bounded	rlimits + timeout + scrubbed env + scratch dir	✅ mostly	❌	~an hour
Namespaced	Bounded + unshare for network/PID/mount	✅	⚠️ partial	a CLI flag more
Off-box	Run the whole loop on a separate disposable machine	✅	✅ for practical purposes	one spare machine

The heuristic I use: Bounded is enough when you control the prompts and the model. Namespaced the moment any prompt text comes from outside. Off-box whenever a bad outcome would be more than an inconvenience.

The important insight is that these stack. Bounded + namespaced + off-box is cheap and dramatically better than any one alone.

Layer one: a bounded, namespaced runner

Here's my current runner. It's different from the naive version in five specific ways, which I'll explain after.

#!/usr/bin/env bash
# quarantine.sh <snippet.py> — run untrusted code with no network,
# no environment, no persistent disk, and hard ceilings.
set -euo pipefail

SNIPPET="$(realpath "$1")"
WORK="$(mktemp -d /tmp/quarantine.XXXXXX)"
trap 'chmod -R u+w "$WORK" 2>/dev/null; rm -rf "$WORK"' EXIT

env -i PATH=/usr/bin:/bin HOME="$WORK" \
  unshare --user --map-root-user --net --pid --fork --mount-proc \
  prlimit --nproc=32 --nofile=64 --fsize=$((16*1024*1024)) \
          --as=$((512*1024*1024)) --cpu=8 \
  timeout --signal=KILL 10 \
  python3 -I "$SNIPPET"

Why each piece earns its place:

1. env -i starts the child with a completely empty environment. This is the single highest-value line in the file. The classic "generated code leaked my API key" incident is just os.environ being readable — an empty environment makes the most valuable target not exist.
2. unshare --net gives the process a private network namespace with no interfaces up. No sockets, no DNS, no exfiltration channel. --pid --fork --mount-proc means the snippet can't even see your other processes.
3. prlimit ceilings bound CPU seconds, address space, process count, open files, and file sizes. The infinite loop dies at 8 CPU-seconds; the memory bomb dies at 512 MiB; nothing it writes can exceed 16 MiB.
4. timeout --signal=KILL 10 is the outer backstop — rlimits cover CPU time, but a process blocked in certain syscalls can outlive them, so wall-clock gets its own enforcer.
5. A mktemp scratch dir as HOME, deleted on exit via trap. Whatever the snippet writes evaporates, and it never runs in a directory that contains anything of yours.

And python3 -I (isolated mode) deserves its own sentence: it ignores PYTHONPATH and user site-packages, which shrinks the attack surface and makes runs reproducible — two wins from one flag.

The eval loop around this is unglamorous: write completion to a temp file, invoke quarantine.sh, capture exit code and bounded stdout, compare against the oracle, append one row to a CSV. Non-zero exit isn't a failure of the loop — it is the signal you're measuring.

Layer two: put the loop on a machine you don't care about

Here's the uncomfortable truth about layer one: namespaces and rlimits are strong, but they're all enforced by the same kernel your daily driver uses. A bug in that enforcement, or a distro that disabled unprivileged user namespaces, and your margin is thinner than you thought.

So the second layer is physical, not technical: run the generate → quarantine → score cycle on a separate box whose worst-case fate is "reimage it." Mine is the free server tier from MonkeyCode — handy here because the same account covers the model side of the loop via their free model access, so generation and execution both happen off my laptop.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The provider is genuinely interchangeable — a retired laptop, a Pi in the corner, any cloud free tier you'd shrug at wiping. What matters is the blast-radius math: layer one contains the common failures, layer two ensures that anything escaping layer one lands somewhere with nothing worth stealing.

Where this breaks (read before trusting)

• Linux only. unshare and prlimit are util-linux. On macOS or Windows, skip straight to containers or a separate host.
• User namespaces aren't guaranteed. Hardened distros and most managed CI runners disable unprivileged userns. Check unshare --user --net true before depending on it; Docker with --network=none and --read-only is the usual fallback.
• This is not adversary-grade. --map-root-user is not a real security boundary against someone actively trying to escape, RLIMIT_NPROC is per-UID, and kernel CVEs exist. Code derived from untrusted user input at scale needs containers minimum, isolated hosts ideally.
• Good code gets caught. Anything needing network, real memory, or more than a few seconds will be killed by design. If your workload legitimately needs those, tune the ceilings deliberately — silently strangled snippets will poison your eval scores and you'll never know why.
• GPUs break the story. Device passthrough pokes holes in the clean isolation model; if your snippets need CUDA, this exact setup isn't for you.

Skip this if

You're executing code shaped by strangers' prompts in production (get real isolation engineering), your evaluation needs hardware devices, or you're hoping execution safety substitutes for actually reading the code. It doesn't — it makes the reading survivable to get to.

Closing thought

"Generate, run, score" is quietly becoming standard plumbing for anyone working with code models, and its default security posture is optimism. The fix isn't exotic: an empty environment, hard ceilings, a network namespace, and a machine you're willing to lose. That combination turns "the model wrote something catastrophic" from an incident into a log line.

If you adapt this — container variant, tighter ceilings, a macOS equivalent — I'm curious which limit fires first in your workload. That number usually tells you something interesting about what your model is actually generating.