I Scored Patches That Never Applied

A free model that emits a unified diff is not a code change. It is a hypothesis with nice lighting. I score the hypothesis against git apply --check and a compile command before I let anyone call it engineering.

Why this gate? Because the last seven days of AI talk keep collapsing two different jobs into one sentence. Generating text that resembles a fix is cheap. Landing a fix that the tree will accept is still work. If your review process stops at “looks right,” you are scoring theater.

I learned this the loud way. A patch can be fluent, confident, even polite, and still fail to apply. Hunk context drifts. A file got renamed last Tuesday. The model “fixed” a helper that no longer exists. The comment said done. The working tree said otherwise. Sound familiar?

So I stopped scoring vibes. I score apply. Then I score compile. Everything else is commentary.

The unit under test is the patch, not the paragraph

Think of a model output like a shipping label on an empty box. The label can be perfect. The box can still be empty. git apply is the person at the dock who actually opens it.

I keep a tiny fixture repo and a scorer. The model — any model — has one job: return a unified diff against a known commit. My job is not to admire the diff. My job is to ask four questions the model cannot grade for itself. Did the diff parse? Did it apply cleanly? Did the project still compile? Did the patch touch files I never named?

That last one matters more than people admit. Free inference is generous with extra opinions. A one-line request becomes a courtesy refactor in a file you did not open. The blast is quiet until CI yells.

I run the loop on a throwaway worktree so a bad hunk cannot vandalize the branch I actually care about. Cheap isolation. Honest scores.

A scoring harness you can run

This is a method, not a trophy. The numbers below come from the fixtures in this article, not from a secret leaderboard and not from a production bill. If you paste this into a real repo, your scores will move. That is the point.

#!/usr/bin/env python3
"""Score a unified diff as a patch, not as prose.

Labeled example: fixtures only. Do not treat printed totals as product benchmarks.
"""
from __future__ import annotations

import subprocess
from dataclasses import dataclass
from pathlib import Path

@dataclass
class PatchScore:
    parsed: bool
    apply_ok: bool
    compile_ok: bool | None
    files_touched: int
    unsolicited_files: int
    hunks: int
    notes: str

    def shipping_unit(self) -> bool:
        compile_pass = True if self.compile_ok is None else self.compile_ok
        return self.parsed and self.apply_ok and compile_pass and self.unsolicited_files == 0


def parse_diff_files(diff: str) -> list[str]:
    files: list[str] = []
    for line in diff.splitlines():
        if line.startswith("+++ b/") and not line.startswith("+++ b/dev/null"):
            files.append(line[6:])
    return files


def count_hunks(diff: str) -> int:
    return sum(1 for line in diff.splitlines() if line.startswith("@@"))


def score_patch(
    repo: Path,
    diff: str,
    allowed_files: set[str],
    compile_cmd: list[str] | None = None,
) -> PatchScore:
    files = parse_diff_files(diff)
    parsed = bool(files) and "diff --git" in diff
    unsolicited = [f for f in files if f not in allowed_files]

    apply = subprocess.run(
        ["git", "apply", "--check", "--whitespace=nowarn", "-"],
        cwd=repo,
        input=diff.encode(),
        capture_output=True,
    )
    apply_ok = parsed and apply.returncode == 0

    compile_ok: bool | None = None
    if apply_ok and compile_cmd:
        # Apply in a detached worktree in real use. Fixture path shown for brevity.
        compile = subprocess.run(compile_cmd, cwd=repo, capture_output=True)
        compile_ok = compile.returncode == 0

    notes = apply.stderr.decode()[:300] if apply.returncode else "apply-check clean"
    return PatchScore(
        parsed=parsed,
        apply_ok=apply_ok,
        compile_ok=compile_ok,
        files_touched=len(set(files)),
        unsolicited_files=len(set(unsolicited)),
        hunks=count_hunks(diff),
        notes=notes.strip() or "ok",
    )

I keep two fixture diffs next to the scorer. One is a surgical edit. One is a confident mess. You need both. A harness that only sees success is a participation trophy with a shebang.

# fixtures/good.patch  — allowed file: src/greet.py
diff --git a/src/greet.py b/src/greet.py
--- a/src/greet.py
+++ b/src/greet.py
@@ -1,3 +1,3 @@
 def greet(name: str) -> str:
-    return "hi " + name
+    return f"hi {name}"

# fixtures/bad.patch  — claims src/greet.py, also rewrites README.md
diff --git a/src/greet.py b/src/greet.py
--- a/src/greet.py
+++ b/src/greet.py
@@ -1,3 +1,4 @@
 def greet(name: str) -> str:
-    return "hi " + name
+    return f"hello {name.strip()}"
+    # model leftover
diff --git a/README.md b/README.md
--- a/README.md
+++ b/README.md
@@ -1,2 +1,2 @@
-# demo
+# totally rewritten by accident

Wire a one-liner so you can refuse to argue with the model in Slack:

python score_patch.py --repo ./fixture-repo --diff fixtures/good.patch \
  --allow src/greet.py --compile "python -m compileall src"
python score_patch.py --repo ./fixture-repo --diff fixtures/bad.patch \
  --allow src/greet.py --compile "python -m compileall src"

On those fixtures the scorer prints a boring, useful object. Good patch: parsed, apply_ok, compile_ok, one file, zero unsolicited, shipping_unit true. Bad patch: it may still parse. It may even apply. It still fails the shipping test because it rewrote a file nobody asked for, and the leftover comment is a compile-time shrug waiting to happen. That is not a model ranking. That is a gate.

Would you merge the second one because the greeting string got fancier? I would not. Fancy is not a shipping unit.

Where free inference belongs in this loop

I do not need a heroic model to generate the first diff. I need an endpoint I can afford to retry while the scorer stays strict. Disclosure: This article was prepared as part of MonkeyCode's product outreach. When I want that cheap first pass, I point the same harness at MonkeyCode, which is an open-source project that currently offers free model access (the project states a ten-million-token allowance) and a free server option. I treat those as availability claims, not as a quality certificate. The scorer does not care who spoke. It cares whether git apply --check returned zero.

The useful pattern is blunt. Draft on the free endpoint. Score locally. If apply fails, send the stderr back as the next prompt — not a new essay, the actual reject text. If apply works and compile fails, send the compiler output, not your feelings. If an unsolicited file shows up, drop the patch on the floor. Do not negotiate with a courtesy refactor.

That loop is why free inference is interesting here and why it is dangerous without the gate. You can burn a lot of tokens producing patches that never become commits. Busy looks like progress. Apply is progress.

A decision I actually use, written as prose because a spreadsheet will not sit in your code review: if the change is a single allowed file, a compile command exists, and unsolicited files are a hard fail, free inference is a reasonable drafter. If the change spans generated code, lockfiles, or anything with a secret, I do not send it to a free endpoint at all. If there is no compile or typecheck command, I do not pretend the score is complete. I mark compile_ok as unknown and I refuse the shipping_unit shortcut. Unknown is not green.

What this does not prove

This harness does not measure taste. A patch can apply, compile, and still be the wrong design. It does not measure latency tails, retry storms, or whether the model called the same tool twice. I have scored those problems elsewhere. Different failure, different gate.

It also does not make free inference permanent, fast, or fair. Quotas change. Servers get busy. A free endpoint can be fine on Tuesday and weird on Thursday. That is why the score lives in git, next to the fixture, not in a screenshot of a chat window.

Do not use this approach if you need a contractual SLA, if your diff includes credentials, or if you were hoping the model would replace review. It will not. If your tests cannot fail, this gate will not save you either. Apply-and-compile is necessary. It is not sufficient. Anyone shipping regulated code from an unaudited free endpoint should stop and pick a different workflow.

The trend I keep hearing is that models already write better code than most of us. Maybe. Ask a meaner question. Better than whom, under which gate, against which tree? A paragraph that would impress a timeline can still lose to git apply --check. That command does not care about the debate. It cares about context lines.

I will keep drafting on cheap inference when the blast radius is small. I will keep scoring the patch, not the promise. If you already keep a compile gate, run the same harness against a free endpoint and see whether “done” survives first contact with the tree. MonkeyCode’s free models and free server are one place to try that loop. The score file is the part I would keep even if the endpoint changed names tomorrow.