48-Hour Field Notes: My Sandbox Said PASS, and Zero Tests Had Run

I spent 48 hours letting a free model generate fixes I never applied, and the most useful thing I learned was that my harness was the bug. The sandbox told me PASS while zero tests had run, then I hashed the wrong artifact, and later I blamed a patch for a timezone problem I had written myself. Here is the loop I ended up with, the three false signals I had to remove, and the escalation rule I now use before I pay for anything.

Why one generation was never enough evidence

A single generated patch is an anecdote, and anecdotes are extremely good at convincing you that you have finished. I wanted the opposite: a cheap way to generate many candidates, a disposable place to run them, and a receipt that survives my own optimism. If a candidate never touches a real branch, then "it looked right" stops being a claim I can honestly make to myself.

For the generate step I used MonkeyCode's free model access, and for the replay step its free server option as a throwaway box. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

So the loop has three stages: generate (text output only, never applied in place), replay (one candidate per disposable run), and triage (a machine-readable receipt that decides what deserves my attention). Why three stages instead of one script? Because each stage fails differently, and I want every failure to stay attributable.

Hours 0-6: one receipt line per attempt

I kept the model call behind a single command so the receipt format stays identical no matter which client I use that week. The script below is close to the version I actually ran, with the generator left as a stand-in you can swap out.

#!/usr/bin/env bash
# attempt.sh <prompt-file> <fixture-dir> <attempt-id>
set -euo pipefail
prompt_file="$1"; fixture_dir="$2"; attempt_id="$3"

workdir="$(mktemp -d)"; trap 'rm -rf "$workdir"' EXIT
mkdir -p receipts

# gen_attempt is my wrapper around the free-model call.
# Replace it with whatever client you use; keep stdout as a plain diff.
gen_attempt "$prompt_file" > "$workdir/candidate.diff"

# The candidate is applied only inside the throwaway box, never in my repo.
rsync -a "$fixture_dir/" "$workdir/repo/" >/dev/null
cd "$workdir/repo"
git init -q . && git add -A && git commit -qm base
git apply --check "$workdir/candidate.diff" && git apply "$workdir/candidate.diff"

I left the client out on purpose, because there is no single blessed way to call a model and I would rather not pretend there is. The receipt is one JSON object per line, since I want to grep it, sort it, and diff two days of runs without opening an editor:

# receipt.py — one line per attempt, appended, never rewritten.
import hashlib, json, re, time, subprocess

def fingerprint(diff: str) -> str:
    lines = [ln.rstrip() for ln in diff.splitlines()]
    lines = [ln for ln in lines if ln and not ln.lstrip().startswith(("#", "//"))]
    return hashlib.sha256(re.sub(r"\s+", " ", "\n".join(lines)).encode()).hexdigest()

started = time.monotonic()
out = subprocess.run(["pytest", "-q", "tests"], capture_output=True, text=True)
collected = re.search(r"collected (\d+) item", out.stdout + out.stderr)
receipt = {
    "attempt": attempt_id,
    "patch_fp": fingerprint(open("candidate.diff").read()),
    "exit_code": out.returncode,
    "collected": int(collected.group(1)) if collected else 0,
    "duration_s": round(time.monotonic() - started, 2),
}
print(json.dumps(receipt))

Hours later I understood why collected deserved its own field. An exit code of 0 does not mean "tests passed," it means "the command I ran was happy," and those are two very different statements.

Hours 6-18: the PASS that meant nothing

My wrapper ran pytest and checked only the exit code, and for a while every candidate came back green. That felt excellent for about an hour. Then I noticed the sandbox had copied my test directory but not my conftest.py, so the suite collected zero tests, and a zero-test run exits 0 with a cheerful summary line.

Would you have caught that in review? I did not, because I was reading the verdict column instead of the receipt underneath it.

The fix was to stop deriving the verdict from the exit code alone, and to give empty runs their own name:

• PASS requires exit_code == 0 and collected > 0.
• EMPTY is its own verdict whenever collected == 0, regardless of exit code.
• FAIL requires exit_code != 0 and collected > 0.
• ERROR covers everything else, including a sandbox that never started.

Once empty runs had a label, several of my early "successes" were reclassified as harness problems rather than candidate problems. That is not a story about a model being wrong; it is a story about me measuring the wrong event.

Hours 18-30: I was hashing the wrong artifact

I tried to deduplicate attempts by hashing the raw response text, which was useless, because every response differed in whitespace and nothing ever looked like a repeat. Hashing prose tells you about formatting, not about the patch in front of you. The fix is the fingerprint above: drop blank and comment lines, collapse whitespace, then hash what remains.

There is a real trade-off here and I want to be honest about it. Normalization can merge two candidates that add different comments but identical logic, which is usually what I want, and occasionally exactly what I do not want when a comment is the only documentation of a workaround. So I keep the raw diff on disk, hash the normalized form, and never delete the original.

Hours 30-42: the sandbox was right and my fixture was wrong

I moved the replay step onto a throwaway server because I wanted a machine whose state I could destroy without thinking about it. Each run got a fresh box, so nothing survived between attempts, no half-applied patch, no cache, no leftover virtualenv. That is the part of this workflow I would keep forever, and it is the part I trust most.

What broke was my own fixture. One test asserted a date string built from the machine's local clock, and the throwaway box ran in UTC while my laptop did not. Three assertions failed for a reason that had nothing to do with the candidate under test. The box was correct; my test had encoded an assumption about where it would run.

The lesson is smaller than it sounds: any fixture that reads ambient state belongs in the receipt as a recorded input, not in the assertion as a constant.

Hours 42-48: the escalation rule

By the end I had a table instead of a feeling, and it is the artifact I would hand to someone starting this tomorrow. It exists mostly to stop me from escalating too early.

Receipt says	What it usually means	What I do
PASS, collected > 0, new fingerprint	candidate survived the fixtures	re-run once, then read the diff myself
PASS, collected == 0	the harness is lying	fix the harness, ignore the patch
FAIL, same assertion, two fingerprints	fixtures describe the bug, candidates are guesses	escalate to a stronger model, or read the code
FAIL, different assertions each run	flaky fixture	fix fixtures before blaming anything
ERROR, no receipt line	infrastructure, not the model	rerun, then investigate the box

The escalation trigger is deliberately narrow. Two different patches failing the same assertion means the failure is about the problem rather than about one candidate, and one failure tells me almost nothing, which is why I stopped treating the first red run as a decision.

What I would repeat, and what I would not

I would repeat the disposable box, the receipt-per-attempt, and the rule that a candidate is only ever a diff. I would not repeat hashing prose, trusting an exit code on its own, or writing a fixture that quietly assumes the clock and the locale of my laptop.

If you want to run this loop, the two pieces I leaned on were MonkeyCode's free model access for the generate step and its free server option for the throwaway replay box. Bring your own fixtures, because those are the part that decides whether any of this is real.

Limitations and who should skip it

Free model access makes a large batch of candidate patches cheap to produce, but cheap candidates are still candidates, and free capacity is a convenience rather than an SLA. I did not measure quotas, latency, or model equivalence, and I would not plan around my own two days as if they were representative of your workload.

Skip this approach if you need pinned model versions for audit, if a generated patch must be attributable to a specific model revision, or if your fixtures cannot reliably fail on purpose. And if nobody on the team will read the diff after it passes, the loop mostly adds receipts to a decision you were never making in the first place.