Diff-Only Drafting: Give the Model a Patch Budget and Let CI Audit the Churn

A docs PR arrives with a net change of +412 / −387. Every heading is in a new place. The actual intent was two sentences about a config flag. Reviewers approve it in ninety seconds because reading it properly would take an hour.

That is the shape of the failure. Not a hallucination you can spot, but a review surface so large that nobody reviews it. The model was allowed to rewrite files, so the diff stopped being an argument and became a replacement.

This post describes a narrower contract: the model emits patches only, each patch must stay inside a section you deliberately opened for it, and a small validator rejects anything that exceeds the budget. The human keeps the parts that carry liability.

Why full-file generation defeats review

A rewrite destroys the reviewer's cheapest tool: locality. When a model returns a whole file, you cannot tell a refactor from a regression without diffing it yourself.

• Review cost scales with file size, not intent. A one-line fix inside a regenerated 600-line file costs the same review effort as 600 changed lines.
• git blame collapses. Attribution becomes "the bot, on Tuesday" instead of a person who can be asked why.
• Silent deletions hide in reflow. A dropped caveat looks like a paragraph that got shorter.
• Approval becomes theater. Large diffs get skimmed, and skimmed approvals are the ones that ship wrong flags.

The fix is not a better prompt. It is a smaller unit of change plus a mechanical gate.

The contract, in six rules

1. Model output is a unified diff. No prose wrapper, no full files.
2. Every hunk must fall inside exactly one ledger section.
3. Section marker lines are immutable. A patch that edits a marker is rejected outright.
4. Only sections tiered derived or assembled are writable. authored sections are read-only for the model.
5. Budgets are hard: at most 3 files, 8 hunks per file, 60 changed lines total.
6. Rejection is data. The validator emits JSON, and that report becomes the retry hint — two attempts maximum, then a human takes over.

Step 1 — mark the sections you are willing to defend

Sections live in the doc as HTML comments, so they survive rendering and stay grepable:

<!-- section: install tier: derived owner: @docs-bot -->
## Install

Run the installer, then confirm the binary is on your `PATH`.
<!-- /section -->

<!-- section: compatibility tier: authored owner: @morgan -->
## Compatibility

Supported platforms and the versions we refuse to break are listed here.
<!-- /section -->

derived means the text follows mechanically from code or logs. assembled means the model may draft but may not invent new commitments. authored means a named person owns the sentence and the model does not get to touch it. If you cannot decide the tier for a section, the tier is authored.

Step 2 — the output contract

Paste this as the system prompt, and keep the escape hatch — it prevents invented prose about facts the model was never given.

You may return a unified diff and nothing else.

Rules:
- Every hunk must stay inside exactly one section marker block.
- Never add, remove, or edit a <!-- section: ... --> or <!-- /section --> line.
- Sections tiered "authored" are read-only for you.
- Budget: at most 3 files, 8 hunks per file, 60 changed lines total.
- If a change needs a fact you were not given, do not write prose. Return:
  NEEDS_HUMAN: <one-line question>

The NEEDS_HUMAN path is what makes the workflow honest. Most bad generated docs are not wrong sentences; they are confidently filled gaps. An explicit refusal turns a silent failure into a queued question.

Step 3 — audit the patch before it lands

The validator below needs only the standard library and a git checkout. It maps each hunk onto pre-patch line numbers, resolves the owning section, and checks the tier. Because marker changes are forbidden, section ranges stay aligned and no post-patch reconstruction is needed.

#!/usr/bin/env python3
"""audit_patch.py - gate a model-proposed docs diff against section tiers."""
from __future__ import annotations

import argparse, json, re, subprocess, sys
from dataclasses import dataclass

OPEN = re.compile(
    r"^<!--\s*section:\s*(?P<id>[A-Za-z0-9_./-]+)\s+tier:\s*(?P<tier>[a-z]+)"
    r"\s+owner:\s*(?P<owner>\S+)\s*-->$"
)
CLOSE = re.compile(r"^<!--\s*/section\s*-->$")
HUNK = re.compile(r"^@@ -(?P<old>\d+)(?:,(?P<oc>\d+))? \+(?P<new>\d+)(?:,(?P<nc>\d+))? @@")

MODEL_TIERS = {"derived", "assembled"}
BUDGET = {"files": 3, "hunks_per_file": 8, "churn": 60}


@dataclass
class Section:
    id: str
    tier: str
    owner: str
    start: int  # 0-based index of the opening marker
    end: int    # 0-based index of the closing marker


def parse_sections(text: str) -> list[Section]:
    out, open_at = [], None
    for i, raw in enumerate(text.splitlines()):
        line = raw.strip()
        m = OPEN.match(line)
        if m:
            if open_at is not None:
                raise SystemExit(f"nested section marker at line {i + 1}")
            open_at = Section(m["id"], m["tier"], m["owner"], i, -1)
        elif CLOSE.match(line):
            if open_at is None:
                raise SystemExit(f"stray closing marker at line {i + 1}")
            open_at.end = i
            out.append(open_at)
            open_at = None
    if open_at is not None:
        raise SystemExit(f"unclosed section {open_at.id!r}")
    return out


def parse_patch(patch: str) -> list[dict]:
    files, cur = [], None
    for raw in patch.splitlines():
        if raw.startswith("+++ "):
            path = raw[4:].strip()
            cur = None if path == "/dev/null" else {
                "path": path[2:] if path.startswith("b/") else path, "hunks": []
            }
            if cur:
                files.append(cur)
        elif raw.startswith("@@ "):
            if cur is None:
                raise SystemExit("hunk before file header")
            m = HUNK.match(raw)
            if not m:
                raise SystemExit(f"unparseable hunk header: {raw!r}")
            cur["hunks"].append({"old": int(m["old"]), "old_count": int(m["oc"] or 1), "lines": []})
        elif cur and cur["hunks"] and (raw[:1] in (" ", "+", "-") or raw == ""):
            cur["hunks"][-1]["lines"].append(raw)
    return files


def base_text(repo: str, path: str) -> str | None:
    proc = subprocess.run(["git", "-C", repo, "show", f"HEAD:{path}"],
                          capture_output=True, text=True)
    return proc.stdout if proc.returncode == 0 else None


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("--repo", default=".")
    ap.add_argument("--patch", required=True)
    ap.add_argument("--json")
    args = ap.parse_args()

    with open(args.patch, encoding="utf-8") as fh:
        files = parse_patch(fh.read())

    problems, churn = [], 0
    if not files:
        problems.append("empty patch")
    if len(files) > BUDGET["files"]:
        problems.append(f"{len(files)} files touched > budget {BUDGET['files']}")

    for f in files:
        path = f["path"]
        base = base_text(args.repo, path)
        if base is None:
            problems.append(f"{path}: not present at HEAD; models may not create docs")
            continue
        secs = parse_sections(base)
        if len(f["hunks"]) > BUDGET["hunks_per_file"]:
            problems.append(f"{path}: {len(f['hunks'])} hunks > budget {BUDGET['hunks_per_file']}")

        for n, h in enumerate(f["hunks"], 1):
            churn += sum(1 for line in h["lines"] if line[:1] in "+-")
            if any(OPEN.match(l.strip()) or CLOSE.match(l.strip())
                   for l in h["lines"] if l[:1] in "+-"):
                problems.append(f"{path} hunk {n}: rewrites a section marker")
                continue
            lo = h["old"]
            hi = h["old"] + max(h["old_count"], 1) - 1
            hit = [s for s in secs if not (s.end + 1 < lo or s.start + 1 > hi)]
            if len(hit) != 1:
                problems.append(f"{path} hunk {n}: touches {len(hit)} sections, expected 1")
                continue
            s = hit[0]
            if s.tier not in MODEL_TIERS:
                problems.append(f"{path} hunk {n}: section {s.id!r} is tier {s.tier!r} (owner {s.owner})")

    if churn > BUDGET["churn"]:
        problems.append(f"churn {churn} lines > budget {BUDGET['churn']}")

    report = {"ok": not problems, "churn": churn,
              "files": [f["path"] for f in files], "problems": problems}
    if args.json:
        with open(args.json, "w", encoding="utf-8") as fh:
            json.dump(report, fh, indent=2)
    print(json.dumps(report, indent=2))
    return 0 if not problems else 2


if __name__ == "__main__":
    sys.exit(main())

Wire it into the loop like this:

git checkout -b draft/install-section
python scripts/prompt_docs.py --section install > /tmp/prompt.txt
# call your model endpoint, write the raw completion to model.patch
python scripts/audit_patch.py --patch model.patch --json /tmp/report.json
git apply --check model.patch && git apply model.patch

Exit code 2 plus the JSON report is the signal to either retry once with the problems appended, or hand the section to a human.

Step 4 — make the test plan hermetic

Do not test this against a live model. Test the gate against fixtures, and test the prompt against a stub server that returns canned diffs.

1. Accept case. One derived section, one two-line hunk, expect exit 0.
2. Tier violation. Same patch retargeted at an authored section, expect exit 2 with the section id in problems.
3. Boundary straddle. A hunk whose old range spans a closing marker and the next opening marker, expect touches 2 sections.
4. Marker edit. A patch that renames a tier:, expect rejection before any tier lookup.
5. Budget breach. Nine hunks, or 61 changed lines, expect 2.
6. New file. A +++ b/docs/new.md header, expect the not-present-at-HEAD rejection.

Keep the stub server dumb — it returns a fixture file byte-for-byte from disk. You are testing parsing and policy, not model quality. Fixture diffs also double as documentation of what "allowed" looks like.

Running the drafting step on a free tier

The gate above is cheap; the model call is the part that costs money, which is why I tried the drafting step against MonkeyCode, an open-source project that offers free model access and a free server option — the operator behind this article cites a 10-million-token free allowance and no-cost server, as advertised at the time of writing. Treat those numbers as a snapshot, not a contract: quotas, model availability, and server terms change, so check the project's current page before you plan a pipeline around them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What made it a reasonable fit for this workflow is mundane and worth stating plainly. The drafting step is many small, bounded calls with tight output budgets, not one heroic long-context job. A diff-only contract with a 60-line churn ceiling is exactly the kind of call a free allowance can absorb while you are still deciding whether the gate earns its keep. If your docs cannot leave your network, or your allowance expires, swap the endpoint — nothing in the validator knows which model produced the patch.

Limits, and who should skip this

• The validator checks permission, not truth. It cannot tell you the flag you documented exists. It only tells you the sentence was written in a section you opened.
• Diff reliability varies. Some models emit malformed hunk headers or drift context lines. Expect retries, and keep the retry budget at two.
• Boundary-straddling hunks fail as false positives. That is deliberate: a hunk that crosses sections is unauditable, so the model gets asked to split it.
• git show HEAD:path ignores uncommitted work. Commit or stash first, or the base text is wrong.
• Renames and file creation are out of scope. Docs structure changes are a human change.

Skip this if your docs are certified by a named engineer under a compliance regime, if you have a single author and no review step to protect, or if the sections are so tangled that every hunk spans three of them. In those cases the ledger becomes ceremony, and ceremony gets disabled in month two.

What the human still owns

These never enter the model's lane, regardless of how good the draft looks:

1. Why the section exists and who it is for.
2. Compatibility promises, deprecation dates, and anything a customer could quote in a contract.
3. Security and data-handling statements.
4. The decision to delete a caveat. Deletions are the highest-risk output a model can produce, and the gate can measure their size but not their cost.
5. Final sign-off on authored sections, by the person named in the marker.

The model writes sentences. The repo proves who was allowed to. If you want to try the drafting half on a free allowance before building the gate, MonkeyCode's project page is the place to check current terms — then keep the gate either way.

Comments

[Gautam K]

"The diff stopped being an argument and became a replacement" is a sharp way to put it. Silent deletions in reflow are the scariest case for docs, since a dropped caveat can carry legal weight. A validator rule that flags any removed line containing words like "must", "never" or "not" would catch most of those.