Your Coding Agent Is Only as Good as the Machine It Can Wreck

The patch worked everywhere except on a clean machine

Last month an agent closed a failing test for me in about ninety seconds, and for a moment I felt genuinely clever about my tooling. Then I cloned the same branch into an empty container, and the suite died before it printed a single line of output. Was the model somehow slower or dumber on the second machine? No. The room was different, and the room had been doing a lot of quiet work for me.

The patch leaned on a virtualenv that existed only on my laptop, a Postgres container I had started three weeks earlier and forgotten about, and a .env file that never made it into version control. The agent answered my request precisely. I had simply made the request inside a room where all the furniture was already arranged, and portability was never part of what I asked for.

Nothing in that output was wrong, and nothing in it was portable either. That gap is what I want to argue about, because I think most teams are measuring the wrong component.

The position: grade the room, not the prompt

Almost every discussion about AI coding quality circles the same question — is the model good enough yet? I think that question has quietly become the less interesting one, because on the work I actually ship, the model is rarely the failing part. The failing part is the handoff: the moment work leaves a machine where everything was already warm and lands on a machine where nothing is.

If you accept that framing, the useful measurement becomes almost brutally simple. Hand the agent a clean machine, give it a repository URL and nothing else, and observe which step dies first. I call this a cold-clone survival test, and it takes an afternoon to build, which is roughly the same afternoon you would otherwise spend reading benchmark leaderboards.

And yes, this is an opinion, so let me say it plainly: most teams should stop shopping for a better model and start buying themselves a disposable machine. Model improvements lift every task a little. A reproducible environment fixes the specific task that keeps failing, and it keeps working after the next model release resets your intuition.

The harness I run before I trust any agent patch

The driver below runs inside a fresh container, records each step as a tab-separated row, and never short-circuits on failure — that last property matters, because a suite that stops at the first error hides how many things were actually broken. Note the --no-cache-dir, the throwaway virtualenv, and the deliberately tiny default smoke command.

#!/usr/bin/env bash
# coldstart.sh — run this ONLY inside a fresh, disposable container.
set -uo pipefail

: "${REPO_URL:?export REPO_URL=[e-mail]:you/app.git}"
WORK="$(mktemp -d)"
REPORT="${REPORT:-$WORK/steps.tsv}"
STEP_TIMEOUT="${STEP_TIMEOUT:-900}"
SMOKE_CMD="${SMOKE_CMD:-.venv/bin/python -c 'import app; print(app.__name__)'}"

run_step() {
  local name="$1" cmd="$2" log="${WORK}/$1.log"
  timeout "$STEP_TIMEOUT" bash -lc "$cmd" >"$log" 2>&1
  printf '%s\t%s\t%s\n' "$name" "$?" "$log" >>"$REPORT"
}

run_step clone   "git clone --depth 1 '$REPO_URL' '$WORK/app'"
run_step install "cd '$WORK/app' && python3 -m venv .venv && .venv/bin/pip install --no-cache-dir -r requirements.txt"
run_step test    "cd '$WORK/app' && .venv/bin/python -m pytest -q"
run_step smoke   "cd '$WORK/app' && $SMOKE_CMD"

python3 "$(dirname "$0")/score.py" "$REPORT"

The scorer is fail-closed on purpose: a missing step is not a pass, it is a blocked run, and the distinction saves you from celebrating a suite that never executed.

#!/usr/bin/env python3
"""score.py — fail-closed verdict for a cold-clone survival run."""
import pathlib, sys

REQUIRED = ("clone", "install", "test", "smoke")
EXIT = {"PASS": 0, "FAIL": 1, "BLOCKED": 2}

def read_report(path):
    rows = {}
    for line in pathlib.Path(path).read_text().splitlines():
        step, code, log = line.split("\t")
        rows[step] = {"exit": int(code), "log": log}
    return rows

def main(path):
    rows = read_report(path)
    missing = [s for s in REQUIRED if s not in rows]
    if missing:
        print(f"BLOCKED  never ran: {', '.join(missing)}")
        return EXIT["BLOCKED"]
    failed = next((s for s in REQUIRED if rows[s]["exit"] != 0), None)
    if failed:
        tail = pathlib.Path(rows[failed]["log"]).read_text().splitlines()[-15:]
        print(f"FAIL     first broken step: {failed} (exit {rows[failed]['exit']})")
        print("\n".join(tail))
        return EXIT["FAIL"]
    print("PASS     clean clone reproduces the test suite")
    return EXIT["PASS"]

if __name__ == "__main__":
    sys.exit(main(sys.argv[1]))

Running it is one command, and the only thing you need locally is Docker and a read-only deploy key:

docker run --rm -it \
  -e REPO_URL=[e-mail]:you/app.git \
  -e SMOKE_CMD=".venv/bin/python -m pytest -q tests/test_smoke.py" \
  -v "$PWD/harness:/harness:ro" \
  -v "$PWD/out:/out" \
  -e REPORT=/out/steps.tsv \
  python:3.12-slim bash /harness/coldstart.sh

The failures it surfaces are rarely the interesting-looking ones. It finds implicit global installs the lockfile never captured, services that something assumes are listening on localhost, shell variables exported so long ago that nobody remembers exporting them, and filename collisions that only appear when a case-insensitive laptop meets a case-sensitive container. My first several runs failed on step two, not step three, which is the whole point: the agent's code was fine, and the workspace contract around it was fiction.

The machine you are willing to destroy

Here is where the sandbox question stops being abstract. A cold-clone loop reruns the same three commands many times a day, and doing that on your own machine means contending with your IDE, your database, and your own muscle memory. What you want is a box you are happy to rm -rf, plus model calls that do not drain a personal budget while you iterate.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project that advertises free model access and a free server option, which is exactly the shape of thing this workflow needs: an environment where the agent can be handed a repository and permission to break things. The operator currently describes a free token allowance on the order of ten million tokens, and I would verify that number on the project page before you build a habit around it, because promotional allowances and server lifetimes move. Treat the server as ephemeral by default: no secrets you cannot rotate, no customer data, no production database URL, and no assumption that yesterday's box is still there today. If you want somewhere to run the harness above without wrecking your own workstation, that is the one use I would genuinely recommend it for.

Before you hand anything over, write down the blast radius. This table is the part I would copy into your repo and argue about in review, because it is the cheapest alignment you will ever buy.

Capability	Disposable agent box	Your laptop / protected CI
Clone the repo with a read-only key	yes	yes
Install public dependencies, no cache	yes	yes
rm -rf the working tree freely	yes	no
Cloud credentials, prod DB URL, real customer rows	never	via secret manager, human-approved
Push to a protected branch or merge	no	no

What this test cannot tell you, and who should skip it

A green cold-clone run does not mean the code is correct. It means the environment is reproducible, which is a smaller and much more defensible claim. It will not catch load-dependent races, migration ordering problems, or a third-party API that fails one request in fifty, because a single sequential run simply does not produce those conditions. It also cannot tell you whether the agent solved the right problem, and I have shipped patches that passed every step while quietly implementing my misread of the ticket.

So treat a passing run as an admission ticket, not a merge approval. The protected branch still needs a human, the diff still needs a reader, and the smoke command you choose is a statement about what you actually care about — if it only imports a module, you have tested your import statements and nothing else.

Some teams should not adopt this at all. If your build genuinely needs a physical device farm or a licensed toolchain, a container cannot reproduce it, and you will burn days chasing a false negative. If you operate under a compliance regime that requires vetted hardware, a shared free box is the wrong place for that workload, no matter how convenient it looks on a Tuesday. And if you have not yet decided what the agent is allowed to touch, build the table before you build the harness, because the harness will happily execute whatever you accidentally authorized.

The question I would actually like answered

Which layer of your handoff breaks first when the environment changes? Is it the install, the test run, the smoke command, or the moment a service you assumed was up turns out to be a container somebody stopped last Friday? Paste the failing step and the exit code, or the HTTP status if it is a request rather than a command, and I will tell you where I would put the assertion.

Because the model is going to keep getting better whether or not any of us write a script this week. The machine underneath it, though, will keep being whatever we happened to leave lying around, and that is the part I have finally stopped trusting.