48-Hour Field Notes: Two Models, One Gate

Two patches came back from the same prompt last Tuesday. One passed the tests, one looked cleaner, and I merged the cleaner one. It failed in review forty minutes later, which is how I learned I had no cheap way to tell "different" from "better." This is the protocol I ran for the next 48 hours, and the part I kept afterward.

The idea is not new: ask two models, compare answers. What made it affordable was running both on free model access instead of metering every call against a card. MonkeyCode is an open-source coding project that offers free model access and a free server option; check its repository for the current limits, because those limits move and I am not going to quote a number that may already be wrong.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Below is the whole gate, and the failures are more interesting than the wins.

What a gate actually needs to decide

An acceptance gate has four outcomes, and only one of them is a merge. Both models pass and produce the same normalized diff: cheap to accept. Both pass and disagree: this is the valuable signal, so log it and read it. One passes: usually a flaky test or a diff that only applies on one branch. Neither passes: the prompt is wrong, not the models.

That fourth case is the one that saved me the most time, because it stopped me from rerolling a prompt that had no chance. The tests were telling me the same thing the models were, one step earlier and for free.

The harness

The script below is stdlib-only, runs both calls concurrently, and writes one JSONL row per item so the run can be resumed and audited. Nothing here stores a key; the model CLI is read from the environment.

# gate.py — dual-model acceptance gate, Python 3.11+, stdlib only
import asyncio, hashlib, json, os, subprocess, tempfile, time
from pathlib import Path

LEDGER = Path("ledger.jsonl")

def norm(patch: str) -> str:
    """Whitespace-insensitive form, so cosmetics don't count as agreement."""
    return "\n".join(l.rstrip() for l in patch.splitlines() if l.strip())

def fp(patch: str) -> str:
    return hashlib.sha256(norm(patch).encode()).hexdigest()[:12]

async def ask(model: str, prompt: str) -> str:
    """Adapter. Point MODEL_CLI at whatever the provider documents."""
    proc = await asyncio.create_subprocess_exec(
        os.environ["MODEL_CLI"], model, "--prompt", prompt,
        stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE,
    )
    out, err = await proc.communicate()
    if proc.returncode != 0:
        raise RuntimeError(f"{model}: {err.decode()[:200]}")
    return out.decode()

def run_tests(patch: str, cmd: list[str], seed: int) -> tuple[bool, str]:
    """Apply into a throwaway copy of tracked files, then run the suite."""
    with tempfile.TemporaryDirectory() as d:
        tar = Path(d, "src.tar")
        subprocess.run(["git", "archive", "HEAD", "-o", tar], check=True)
        subprocess.run(["tar", "-xf", str(tar)], cwd=d, check=True)
        chk = subprocess.run(["git", "apply", "--check", "-"],
                             input=patch.encode(), cwd=d, capture_output=True)
        if chk.returncode != 0:
            return False, "patch-does-not-apply"
        subprocess.run(["git", "apply", "-"], input=patch.encode(), cwd=d, check=True)
        env = {**os.environ, "PYTHONHASHSEED": str(seed)}
        r = subprocess.run(cmd, cwd=d, capture_output=True, env=env)
        return r.returncode == 0, (r.stdout + r.stderr).decode()[-400:]

The driver is deliberately boring. One item at a time, one ledger line at a time, and a flush after every row so a paused server costs you one item instead of a night.

async def gate(item_id: str, prompt: str, tests: list[str], seed: int = 0):
    models = [os.environ["MODEL_A"], os.environ["MODEL_B"]]
    t0 = time.monotonic()
    results = await asyncio.gather(*(ask(m, prompt) for m in models),
                                   return_exceptions=True)
    row = {"id": item_id, "models": models, "seed": seed}
    for m, r in zip(models, results):
        if isinstance(r, Exception):
            row[m] = {"error": str(r)[:200]}
            continue
        ok, log = run_tests(r, tests, seed)
        row[m] = {"fp": fp(r), "tests": ok, "log": log if not ok else ""}
    passed = [m for m in models if row.get(m, {}).get("tests")]
    fps = {row[m]["fp"] for m in passed}
    row["verdict"] = ("no-signal" if not passed
                      else "agree" if len(fps) == 1
                      else "split")
    row["seconds"] = round(time.monotonic() - t0, 1)
    with LEDGER.open("a") as f:
        f.write(json.dumps(row) + "\n")
        f.flush()
    return row["verdict"]

Run it in batches and skip anything already in the ledger, so an interrupted run is a resume and not a restart:

MODEL_CLI=./model-cli MODEL_A="$M1" MODEL_B="$M2" \
  python -c 'import asyncio, json, sys, gate
seen = {json.loads(l)["id"] for l in open("ledger.jsonl")} if gate.LEDGER.exists() else set()
items = [json.loads(l) for l in open(sys.argv[1])]
async def main():
    for it in items:
        if it["id"] in seen: continue
        print(it["id"], await gate.gate(it["id"], it["prompt"], it["tests"]))
asyncio.run(main())' items.jsonl

Then the only analysis that matters:

jq -r '.verdict' ledger.jsonl | sort | uniq -c
jq -r 'select(.verdict=="split") | .id' ledger.jsonl | head -20

The worksheet, not a benchmark

I am not going to hand you reaction-ready percentages from one weekend on one repository. Fill this in for your own codebase; the shape of the numbers is the finding, not my numbers.

Outcome	What it means	What to do
agree	both pass, same normalized diff	accept without reading; sample 1 in 10 to keep yourself honest
split	both pass, different diffs	read both; count how often you prefer one model consistently
one-pass	one applies and passes, one does not	check for flaky tests before blaming the model
no-signal	neither passes	fix the prompt or the test, do not reroll
error	timeout, rate limit, malformed patch	count separately; it is infrastructure, not model quality

Track four columns per batch: total items, share of agree, share of split, and wall-clock seconds per item. If agree is high, the gate is mostly a merge button and you can stop running two models for that task class. If split is high, the gate is an editor, and it earns its cost. If error is above a few percent, you have a transport problem and the gate cannot tell you anything yet.

What broke

The first break was cosmetic. Two diffs were semantically identical but differed in trailing whitespace and import order, so the gate reported split on every item. Normalizing before fingerprinting fixed it, and it taught me that a naive string compare produces disagreement as an artifact of formatting.

The second break was the server, but not in the way I expected. The instance paused mid-batch and the ledger was only on local disk. I lost the batch, not the work, because the rerun replayed prompts that were already answered and paid for them again. The fix is unglamorous: commit ledger.jsonl to a scratch branch after each batch, and make every step a function of the item id. Free servers are fine for this workload precisely because the job is idempotent — but only if you make it idempotent before you need it to be.

The third break was my own trust. A split where one model was right and the other was confidently wrong looked exactly like a split where both were reasonable. The verdict does not measure correctness; it measures disagreement. Anything security-adjacent, migration-shaped, or hard to reverse still goes to a human, and I stopped routing those through the gate at all.

Who should not do this

Skip the gate if your test suite takes more than a couple of minutes, since you will pay that cost twice per item and stop using the tool by Wednesday. Skip it if your tests are flaky, because a flaky suite turns every one-pass into a coin flip and the ledger fills with noise. Skip it for one-off refactors you will read line by line anyway — the reading was the gate.

Also skip it if you need a guarantee. Two models agreeing on a patch is evidence of shared training data as often as it is evidence of correctness, and a correlated error will pass both calls in the same second.

What I would repeat: the ledger format, the git-archive isolation, and the rule that no-signal means fix the prompt rather than reroll it. What I would drop: running two models on trivial formatting tasks, and trusting the verdict for anything I could not revert in one command.

If you want to try it, the script above is the entire artifact. The free model access is what makes a double call a default rather than a budget decision while you are still figuring out which task classes deserve it — start with your test suite and a batch of ten items, and let the ledger tell you whether the second opinion is worth the wall clock.