Sandbox First: A Safer Local Harness for Evaluating Free Coding Models on Your Own Codebase

In my last post I walked through a reproducible baseline for comparing free LLM coding models against my own repository. A few people asked a follow-up question I should have answered the first time: how do you let a model run its own suggestions — compile, execute, attempt a fix — without giving an experimental model free rein over your working directory?

That question got sharper this week as the community chewed on what happens when AI agents get more tools and their boundaries fail. It's not hypothetical. A model asked to 'fix the failing test' will cheerfully edit the test until it passes. A model asked to 'clean up temp files' might glob something you care about. None of this requires malice — just a model, a shell, and no fence between them.

So here's the companion piece: a small, boring, auditable harness that lets a coding model attempt tasks against a real checkout of your repo while keeping the blast radius to a disposable directory. It pairs naturally with free model access (I'm using MonkeyCode's free models and its free server option for the runs below, so nothing here needs a paid API key), but the harness itself is model-agnostic — you can point it at anything that speaks an OpenAI-style chat endpoint.

The threat model (kept small and honest)

I'm not trying to defend against a hostile model. I'm defending against three boring failure modes I've actually hit:

1. Over-eager edits. The model 'fixes' the assertion instead of the bug.
2. Runaway shell commands. A suggested cleanup command with a wider glob than intended.
3. State pollution. A half-applied patch that makes the next model run look better or worse than it should, quietly corrupting a comparison.

The third one is the sneaky one for benchmarking. If you're comparing models, run isolation isn't just safety — it's measurement hygiene.

The harness

The core idea: every task attempt happens in a fresh copy of the repo, under a temporary directory, with no network assumptions and a hard timeout. The model proposes, the harness disposes.

#!/usr/bin/env python3
"""sandbox_eval.py — run one model task attempt in a throwaway repo copy.

Usage:
    python sandbox_eval.py --repo /path/to/repo --task tasks/01_fix_off_by_one.md

Requires: git, python 3.10+. No third-party deps for the harness itself.
"""
import argparse
import shutil
import subprocess
import sys
import tempfile
import time
from pathlib import Path

TASK_TIMEOUT_S = 120          # hard cap on any model-triggered command
ALLOWED_CMDS = {"pytest", "python", "node", "npm", "go", "cargo"}


def fresh_checkout(repo: Path, workdir: Path) -> Path:
    """Copy the repo into workdir. Uses `git worktree`-free plain copy so
    uncommitted files don't leak in and the original is never touched."""
    dest = workdir / "checkout"
    shutil.copytree(
        repo, dest,
        ignore=shutil.ignore_patterns(".git", "node_modules", "__pycache__", ".venv"),
    )
    return dest


def run_guarded(cmd: list[str], cwd: Path) -> tuple[int, str]:
    """Run a command the model asked for, with a denylist-by-default stance."""
    prog = Path(cmd[0]).name
    if prog not in ALLOWED_CMDS:
        return 126, f"REFUSED: '{prog}' is not in ALLOWED_CMDS"
    try:
        out = subprocess.run(
            cmd, cwd=cwd, capture_output=True, text=True, timeout=TASK_TIMEOUT_S,
        )
        return out.returncode, (out.stdout + out.stderr)[-4000:]
    except subprocess.TimeoutExpired:
        return 124, f"TIMEOUT after {TASK_TIMEOUT_S}s"


def diff_summary(checkout: Path, original: Path) -> str:
    """What did the attempt actually change? Report, don't trust."""
    out = subprocess.run(
        ["diff", "-ru", "--exclude=.git", str(original), str(checkout)],
        capture_output=True, text=True,
    )
    return out.stdout[:8000] or "(no changes)"


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--repo", type=Path, required=True)
    ap.add_argument("--task", type=Path, required=True)
    args = ap.parse_args()

    task_text = args.task.read_text()
    started = time.time()

    with tempfile.TemporaryDirectory(prefix="modeleval-") as td:
        checkout = fresh_checkout(args.repo.resolve(), Path(td))

        # --- model interaction happens here ---
        # ask_model(task_text, checkout) -> list of commands + file edits
        # Apply edits inside `checkout` only; execute via run_guarded().
        # Left as a seam: plug in whatever endpoint you're evaluating.
        proposed = ask_model(task_text, checkout)   # see note below
        for cmd in proposed.get("commands", []):
            code, log = run_guarded(cmd, checkout)
            print(f"$ {' '.join(cmd)}\n  exit={code}\n{log}\n")

        print("=== diff vs original ===")
        print(diff_summary(checkout, args.repo.resolve()))

    print(f"\nelapsed: {time.time() - started:.1f}s (temp dir destroyed)")
    return 0


def ask_model(task_text: str, checkout: Path) -> dict:
    """Stub. In my runs this posts the task plus a file listing to a chat
    endpoint and parses a small JSON contract:
        {"edits": [{"path": ..., "content": ...}], "commands": [[...], ...]}
    Keep the contract tiny. Models follow a tiny contract far more reliably
    than a clever one."""
    raise NotImplementedError("wire up your endpoint here")


if __name__ == "__main__":
    sys.exit(main())

Three design choices worth explaining:

• Copy, don't share. A full copytree minus .git and dependency folders is slower than a worktree, but it means the model physically cannot git push, amend history, or read your credentials out of .git/config. Isolation by file boundary, not by policy.
• Allowlist commands, never parse intent. ALLOWED_CMDS is short on purpose. If a task genuinely needs something else, I add it consciously for that task and note it in the results. rm, curl, and git remote operations are permanently out — the model works on files and runs tests, nothing more.
• Always print the diff. The most useful output of an evaluation run isn't pass/fail, it's what the model actually changed. When a model 'solves' a task by weakening a test, the diff shows it immediately. Automated scoring can miss this; eyes on a diff usually don't.

For the model endpoint, I've been running these against MonkeyCode's free models through its free server option, which means the eval loop costs nothing per run and I can rerun the whole task suite as often as I tweak the harness. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The harness itself doesn't depend on that service — swap ask_model for any OpenAI-compatible endpoint — but free access is what makes the rerun-early-rerun-often workflow practical rather than something you ration.

A test plan you can steal

If you want to reproduce this on your own repo:

1. Pick 5–8 tasks from your real history. Closed bugs with a known-good fix are ideal — you already have ground truth. Write each as a short markdown file: symptom, expected behavior, where to look.
2. Include one adversarial-ish task where the lazy solution is editing the test. This is your integrity probe.
3. Run each model 3× per task in fresh sandboxes. Single runs lie; temperature and sampling make one attempt unrepresentative.
4. Score three things separately: did the guarded test run pass, does the diff touch only plausibly-relevant files, and did the model attempt anything outside the allowlist (a signal about how it behaves when unsure).
5. Keep the raw diffs, not just the scores. When two models tie on pass rate, the diffs tell you which one you'd actually want in your review queue.

Limitations and who should skip this

• This is a fence, not a fortress. A temp directory plus command allowlist stops accidents and mild misbehavior. It is not a security boundary against a genuinely adversarial model or a poisoned dependency — for that you'd want actual container or VM isolation, network egress rules, and read-only mounts. Treat this harness as the 'don't trip over the cord' layer, not the vault door.
• Copy-based isolation gets expensive on huge repos. If your checkout is several gigabytes, use a git worktree against a dedicated clone with no credentials, accepting the weaker boundary.
• Results don't transfer across repos. A model that's strong on my Python service tasks may be mediocre on your Rust CLI. That's the entire point of running this on your codebase — resist generalizing.
• If you only ever run one model and never compare, this is overkill. The harness pays off when you're rerunning suites across models or prompts; for a single one-off question, just use the tool in your editor.

The broader lesson from this week's boundary discussions holds at this small scale too: the safest agent setup isn't the one with the smartest model, it's the one where a bad answer is cheap. Fresh directory, short allowlist, diff at the end — and experimental models stop being something you supervise and start being something you measure.

If you build on this, I'd genuinely like to hear what your task suite looks like — the integrity-probe task in particular. What catches models cheating on your codebase?