How Wide Is the Patch?

A green test run tells you one thing: the assertion you wrote now holds. It says nothing about the eleven lines the agent changed to get there — and those eleven lines are what your reviewer has to read at 5 p.m. on a Friday.

So I stopped scoring agents on pass/fail. Pass/fail is a coin, not a measurement. What I score now is patch width: how much of the diff lands outside the function that actually broke.

Pin the span before you look at the diff

You can't measure width without an origin. The origin is the innermost function containing the line in the traceback. Don't eyeball it — parse it.

#!/usr/bin/env python3
# scope.py <repo> <path> <line>  ->  {"path", "start", "end"}
import ast, json, sys

repo, path, line = sys.argv[1], sys.argv[2], int(sys.argv[3])
full = f"{repo}/{path}"
tree = ast.parse(open(full).read(), filename=full)

best = None
for node in ast.walk(tree):
    if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
        end = getattr(node, "end_lineno", node.lineno)
        if node.lineno <= line <= end:
            if best is None or (end - node.lineno) < (best[1] - best[0]):
                best = (node.lineno, end)

if best is None:
    sys.exit(f"no function spans line {line} in {path}")

print(json.dumps({"path": path, "start": best[0], "end": best[1]}))

Nested functions resolve to the innermost match, which is what you want. A fix that only touches the outer wrapper when the bug is in the closure is a fix in the wrong place, and this makes that visible on the first run.

Then score the diff

The scorer reads a unified diff and asks one question per hunk: how many of your new lines fall inside the span?

#!/usr/bin/env python3
# blast.py <diff-file> <scope.json>
import json, re, sys

HUNK = re.compile(r"^@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@")

def hunks(text):
    out, path = [], None
    for line in text.splitlines():
        if line.startswith("+++ b/"):
            path = line[6:].strip()
        elif line.startswith("@@") and path:
            m = HUNK.match(line)
            if m:
                out.append({"path": path,
                            "new_start": int(m.group(3)),
                            "new_len": int(m.group(4) or 1)})
    return out

def overlap(h, scope):
    if h["path"] != scope["path"]:
        return 0
    a, b = h["new_start"], h["new_start"] + h["new_len"] - 1
    return max(0, min(b, scope["end"]) - max(a, scope["start"]) + 1)

diff, scope = open(sys.argv[1]).read(), json.load(open(sys.argv[2]))
hs = hunks(diff)
added = sum(h["new_len"] for h in hs)
inside = sum(overlap(h, scope) for h in hs)

print(json.dumps({
    "files_touched": len({h["path"] for h in hs}),
    "hunks": len(hs),
    "added_lines": added,
    "in_scope_lines": inside,
    "out_scope_lines": added - inside,
    "scope_ratio": round((added - inside) / added, 3) if added else 0.0,
}))

Run it with -U0 or the hunk lengths include three lines of context each side and your ratio is fiction:

git diff -U0 > patch.diff
python scope.py . pkg/parser.py 57 > scope.json
python blast.py patch.diff scope.json | tee blast.json

A worked example, so the numbers aren't a claim

Hand-build a diff against scope.json of {"path": "pkg/parser.py", "start": 40, "end": 78} and the scorer has something to chew on. Say the patch adds two lines at 58, three lines at 13, and one line to the README. Six added lines, two of them inside the span, two files touched. scope_ratio comes back 0.667.

Read that out loud. Two thirds of the patch never goes near the function that failed. One hunk adds a module-level logger nobody asked for; another edits documentation as part of a bug fix. Both are review surface. Both can collide with someone else's branch. Neither is the fix.

Is a two-line fix wrong because it came with a logger? No. Is it wider than it needed to be? Yes, and width is the thing that costs you a second review cycle.

How to read the ratio

scope_ratio	files_touched	what to do next
≤ 0.25	1	review the diff as-is
0.25 – 0.6	1–2	read every out-of-scope line before re-running CI
> 0.6 or ≥ 3 files	any	treat it as a proposal; re-prompt with the span pinned

Re-prompting with the span pinned is the cheap lever. Paste lines 40–78 of pkg/parser.py into the next attempt and watch the ratio fall. Same model, same task, narrower target — that's a harness change, not a model change, and it's the kind of thing you can prove with two runs and a diff.

This is where MonkeyCode's free model access earns its place in my loop: I want four candidate patches for the same seeded bug, and I want them from something I can burn through without thinking about a meter. The free server option matters for the same reason — the test runner and the scorer live on the same host, so reproducing a run doesn't depend on my laptop's toolchain. Treat the allowance and the server as currently published and check the project's own page before you size a batch on them; allowances move and I'm not going to quote a number that ages badly.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What this doesn't tell you

It doesn't count deletions. A patch that removes eight lines outside the span is worse than one that adds eight, and my script above ignores that entirely — extend hunks to track - lines and you'll have the better metric. It also assumes the fix belongs in one function, which is false whenever a signature or an interface has to move. And it's Python-only until you swap ast for a tree-sitter pass, so TypeScript and Go are out until then.

Skip this if you're working in greenfield code, where "outside the failing function" has no meaning yet. Skip it if your failures don't come with a line number. And skip it if you're going to treat the ratio as a quality score — it measures shape, not correctness. A perfectly narrow patch can still be wrong.

So run it on one seeded bug this week. Not forty. One. If half the patch lands somewhere you didn't ask for, would you have caught that from a green checkmark?