I Imported the Patch. That's When It Died.

I imported the patch. That's when it died.

ast.parse had already smiled. The chat had already called the work complete. Neither of those is a gate I trust. If your coding-model eval still stops at "looks like Python," you are grading handwriting, not software.

Why does this keep happening? Because a lot of us are still scoring the story of a fix. Fluency is cheap. Import is not. I wanted a number I could defend without recruiting a second model to compliment the first one.

So I built a three-gate harness: parse, import, test. Same fixture. Same patch. Three increasingly rude questions. The code below is the experiment. You can run it without a GPU and without believing a single adjective.

The fixture is small on purpose

I do not need a web app to catch a liar. I need a module that parses, then explodes when Python executes the top level, then fails a real assertion if you get past that. Parse-time green is a vanity metric. Import is the first honest gate.

# shop.py  — broken on purpose
from decimal import Decimal, ROUND_HALF_UP

TAX = None  # populated only if you actually import this file correctly

def require_tax_table():
    if TAX is None:
        raise RuntimeError("tax table was never loaded")
    return TAX

# Module-level landmine. ast.parse will not run this.
require_tax_table()

def add_tax(cents: int) -> int:
    rate = require_tax_table()
    money = Decimal(cents) * (Decimal("1") + rate)
    return int(money.to_integral_value(rounding=ROUND_HALF_UP))

# test_shop.py
from shop import add_tax

def test_add_tax_rounds_half_up():
    assert add_tax(100) == 108
    assert add_tax(1) == 1

See the trap? ast.parse never calls require_tax_table(). A model can rewrite add_tax with gorgeous syntax and still leave the import path haunted. That gap is what I wanted to count. Chat will not count it for you.

The intended fix is boring. Load a Decimal("0.08") tax table before anyone calls add_tax, keep the rounding, do not invent a helper that is not in the file. Boring is the point. I am not grading creativity. I am grading whether the module exists in a Python process.

I ask for a diff. I refuse to read it.

The prompt is hostile on purpose. If the model wants to narrate, it can narrate into a zero.

Return a unified diff for shop.py only.
Do not explain. Do not wrap the diff in markdown.
The file must import and test_add_tax_rounds_half_up must pass.
Tax rate is 8%. Use decimal rounding already in the file.

Then I do not read the completion. I extract. I apply. I score. If that sounds unfriendly, good. An oracle should be unfriendly.

import re, subprocess, sys, tempfile, shutil, ast
from pathlib import Path

DIFF_RE = re.compile(r"(?ms)^--- .*^\+\+\+ .*?^(?=\Z)", re.M)

def extract_unified_diff(text: str) -> str | None:
    if text.lstrip().startswith("--- "):
        return text
    blocks = re.findall(r"```

(?:diff|patch)?\n(.*?)

```", text, re.S)
    for b in blocks:
        if b.lstrip().startswith("--- "):
            return b
    return None  # essays get nothing. no pity parse.

If the completion is a memoir with a fenced snippet in the middle, and that snippet is not a unified diff, the extractor returns None. Score zero at gate zero. No partial credit for confidence. I used to "helpfully" take the first code fence. That is how you score markdown as Python and then convince yourself the model improved.

Applying the patch is equally unromantic:

def apply_diff(src: Path, diff: str) -> Path | None:
    tmp = Path(tempfile.mkdtemp())
    shutil.copytree(src.parent, tmp, dirs_exist_ok=True)
    patch = tmp / "change.diff"
    patch.write_text(diff)
    r = subprocess.run(
        ["patch", "-p1", "--batch", "-d", str(tmp)],
        input=diff, text=True, capture_output=True,
    )
    return tmp if r.returncode == 0 else None

Need patch on the machine. That is a feature. If your eval loop cannot survive a 1970s Unix tool, it is not ready to survive a model that emits @@ headers for files you never asked it to touch.

Three gates, one script

def gate_parse(py: Path) -> bool:
    try:
        ast.parse(py.read_text())
        return True
    except SyntaxError:
        return False

def gate_import(py: Path) -> bool:
    r = subprocess.run(
        [sys.executable, "-c", f"import sys; sys.path.insert(0, {py.parent.as_posix()!r}); import shop"],
        capture_output=True, text=True,
    )
    return r.returncode == 0

def gate_test(root: Path) -> bool:
    r = subprocess.run(
        [sys.executable, "-m", "pytest", "-q", str(root / "test_shop.py")],
        capture_output=True, text=True,
    )
    return r.returncode == 0

def score(completion: str, fixture: Path) -> dict:
    diff = extract_unified_diff(completion)
    if not diff:
        return {"extract": False, "parse": False, "import": False, "test": False}
    root = apply_diff(fixture, diff)
    if root is None:
        return {"extract": True, "parse": False, "import": False, "test": False}
    py = root / "shop.py"
    parsed = gate_parse(py)
    imported = parsed and gate_import(py)
    tested = imported and gate_test(root)
    return {"extract": True, "parse": parsed, "import": imported, "test": tested}

I run it against checked-in completions, not against a vibe:

python score_patch.py fixtures/essay.txt
python score_patch.py fixtures/parse_only.txt
python score_patch.py fixtures/import_ok_test_fail.txt
python score_patch.py fixtures/pass.txt

Those four files are oracles for the scorer. They are not a vendor bake-off. If my grader cannot separate them, I do not get to grade a model yet. The labeled output below is from those fixtures, which you can paste yourself. It is not a claim about any hosted model.

# fixture-driven scorer output (not a product benchmark)
essay                 extract=0 parse=0 import=0 test=0
parse_only            extract=1 parse=1 import=0 test=0
import_ok_test_fail   extract=1 parse=1 import=1 test=0
pass                  extract=1 parse=1 import=1 test=1

parse_only is the one that should make you nervous. It is valid syntax. It still cannot be imported. An LLM judge will often call it solid. ast.parse will call it fine. Python will not.

What the gates actually argue about

Parse-only completions are empty calories. They look like engineering. They are essays wearing parentheses. I used to stop there because it felt quantitative. It was not.

Import failures are more interesting. The model invents load_rates(), or it leaves a name it saw in a comment, or it deletes the landmine and forgets to put a tax table back. Fluency stays high. The process dies before pytest exists. That is not a "almost." That is a format error with extra steps.

Test failures after a clean import are the only failures I treat as the model having engaged the problem. Everything before that is packaging. I blurred those categories for months. That was on me, not on the model.

Want a one-liner while you iterate on the extractor?

python -c "import ast,pathlib; ast.parse(pathlib.Path('shop.py').read_text()); print('parsed')"
python -c "import shop" || echo "imported? no"
python -m pytest -q test_shop.py

Three commands. Three different truths. If your dashboard collapses them into one thumbs-up, your dashboard is lying with better typography.

Where a cheap generation loop helps, and where it breaks

I still need something to emit candidate diffs. Burning a paid endpoint to discover that a pile of samples never import is how you light money on fire. The mechanical grader is the expensive-looking part that is actually free. Generation is the part that tempts you to skip gates because "it read well."

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I used MonkeyCode only as a generation pipe: free model access and a free server option, so the loop can throw diffs at this harness without me standing up inference. The scorer does not call it. Delete the pipe and the gates still work. That is the only honest integration I can defend.

Where that setup performs well: volume. You can park messy completions against parse/import/test until the failure taxonomy stops moving. You are paying in attention, not in GPU mythology. Where it breaks: the free path will not save a weak fixture. If your test is assert True, import becomes theater again. Unstructured completions wrap diffs in sermons; if you relax the extractor, you will grade the sermon. Truncation dies at parse or import, which is good, unless you mark those as "needs review" instead of fail. Do not do that. Fail is a complete sentence.

Limitations, out loud

This harness does not measure taste. It will pass a brutal one-liner that satisfies test_add_tax. It will fail a correct redesign that renames the function. It is hostile to refactors. That is a bias, not a virtue. It also will not catch a wrong tax rate if you never asserted 108. Weak tests make import look like enlightenment.

Do not use this if you are evaluating docs, architecture notes, or UI copy. Do not use it as a hiring rubric. Do not use it to stamp "production ready" on a model because gate three went green on one file. If you need semantic review, add a linter, then a type checker, then stop before you add another LLM as judge. The last time I let a model grade a model, pretty prose still moved the score.

I also do not claim these gates outlive a moving product surface. Free access and a free server are operator-supplied availability notes, not a promise about models, hardware, duration, or quota. If the pipe changes, the scorer should not care. If the scorer cares, you coupled the wrong things.

The ranking that flipped

Once I ignored chat, my personal ranking of "this completion looks good" collapsed. The essay I would have merged in a hurry scored zeros. The ugly diff that never said "sure" imported and passed. That is the whole lesson, and it does not require a leaderboard.

So I am done asking whether AI outgrew our tests. I ask whether our tests ever loaded the code. Most days they did not. Parse is polite. Import is not. I grade the rude one.