Draft Convergence Is Not Correctness: Stopping Rules for Repeated Model Redrafts of Docs

Repeatedly asking a model to redraft the same documentation section produces a measurable signal, and that signal is churn, not quality. When the edits between consecutive drafts fall to near zero, the model has stopped changing its mind, which is useful to know. It means you can stop spending passes, and it says nothing at all about whether the remaining sentences are true. This article builds a small gate that measures churn across draft iterations and routes the section to one of four actions: one more pass, human-first rewrite, stalled-claims escalation, or oscillation escalation.

Why churn is the only thing you can measure from drafts alone

A single draft gives you text and no reference point, so any judgment about it collapses into opinion. A series of drafts gives you a trajectory, and trajectory is cheap to compute with token-level diffing. If two consecutive drafts are 96 percent identical, the model is not converging on new information; it is restating a fixed position with fresh wording. That position may be right or wrong, and separating those two cases is exactly where a human enters the workflow.

The gate below never claims a section is correct. It only decides whether additional model passes have any expected value, and it blocks acceptance while a claim identifier lacks a named human verifier.

The artifact: a convergence gate for one doc section

The script is deliberately dependency-free, so you can drop it into a repository and run it in CI without a lockfile argument. It reads immutable draft files, computes churn between neighbours, extracts claim identifiers from the latest draft, and prints a machine-readable decision.

#!/usr/bin/env python3
"""convergence.py - decide when to stop redrafting a doc section with a model."""
from __future__ import annotations

import argparse
import difflib
import json
import re
from pathlib import Path

TOKEN_RE = re.compile(r"[a-z0-9_]+|`[^`]+`")
CLAIM_RE = re.compile(r"\[claim:([a-z0-9_.-]+)\]")


def load(path: Path) -> str:
    return path.read_text(encoding="utf-8")


def tokens(text: str) -> list[str]:
    return TOKEN_RE.findall(text.lower())


def claim_ids(text: str) -> set[str]:
    return set(CLAIM_RE.findall(text))


def churn(previous: list[str], current: list[str]) -> float:
    matcher = difflib.SequenceMatcher(a=previous, b=current, autojunk=False)
    return round(1.0 - matcher.ratio(), 4)


def decide(hops: list[float], pending: list[str],
           plateau: float = 0.04, rebound: float = 1.5) -> tuple[str, str]:
    last, prev = hops[-1], hops[-2]
    if last >= max(0.08, prev * rebound):
        return "escalate-oscillation", "churn rebounded; the model is re-deciding settled text"
    if last <= plateau and prev <= plateau:
        if pending:
            return "escalate-stalled", "text stopped moving, but unverified claims still block publication"
        return "accept-verified", "stable text and every claim id has a named verifier"
    if last < prev:
        return "redraft-once", "churn fell again; one more pass is likely to pay for itself"
    return "human-first", "churn flattened above the plateau; more passes buy noise, not accuracy"


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("drafts", type=Path)
    parser.add_argument("--verified", type=Path, default=None)
    parser.add_argument("--plateau", type=float, default=0.04)
    parser.add_argument("--json", action="store_true")
    args = parser.parse_args()

    files = sorted(args.drafts.glob("draft-*.md"))
    if len(files) < 3:
        print("need at least three drafts: draft-01.md, draft-02.md, draft-03.md")
        return 2

    texts = [load(path) for path in files]
    tokens_per_draft = [tokens(text) for text in texts]
    hops = [churn(tokens_per_draft[i - 1], tokens_per_draft[i])
            for i in range(1, len(tokens_per_draft))]

    verified = set()
    if args.verified and args.verified.exists():
        verified = {line.strip() for line in load(args.verified).splitlines()
                    if line.strip() and not line.startswith("#")}
    pending = sorted(claim_ids(texts[-1]) - verified)

    action, reason = decide(hops, pending, args.plateau)
    report = {
        "drafts": [path.name for path in files],
        "tokens": [len(item) for item in tokens_per_draft],
        "churn": hops,
        "unverified_claims": pending,
        "action": action,
        "reason": reason,
    }

    if args.json:
        print(json.dumps(report, indent=2))
    else:
        for name, count, hop in zip(report["drafts"][1:], report["tokens"][1:], hops):
            print(f"{name:<14} {count:>5} tok  churn={hop:.3f}")
        print(f"unverified={pending}")
        print(f"action={action}  ({reason})")
    return 0 if action.startswith("accept") else 1


if __name__ == "__main__":
    raise SystemExit(main())

Step 1: Freeze the prompt and the inputs before you measure anything

Churn is only meaningful if the only thing changing between runs is the model. Keep the instruction text, the source files, and the surrounding outline byte-identical across the whole series, and record sampling settings beside the drafts. If you cannot freeze temperature or seed, write down that the hop series is noisy, because an unrecorded change in sampling is indistinguishable from the model learning something.

Step 2: Store every draft as an immutable file

Name drafts draft-01.md, draft-02.md, and so on, and never overwrite one to save disk space. The gate depends on the full series, and a lost intermediate draft silently turns a real oscillation into a smooth curve. A directory per section keeps the series local, for example .drafts/auth-timeout/.

Step 3: Mark the claims a human must own

Inline identifiers make ownership auditable without a second file format. Add short tags to the sentences a reviewer would need to defend, such as [claim:auth.timeout.default] on the sentence stating the default timeout. Then list resolved identifiers in verified.txt, one per line, and the gate will refuse an acceptance decision while any identifier is missing.

mkdir -p .drafts/auth-timeout
cp draft-0*.md .drafts/auth-timeout/
python3 convergence.py .drafts/auth-timeout --verified verified.txt

The output below comes from a constructed four-draft fixture, not from a production documentation run, so treat the numbers as shape rather than as a benchmark:

draft-02.md    331 tok  churn=0.240
draft-03.md    344 tok  churn=0.035
draft-04.md    349 tok  churn=0.021
unverified=['auth.timeout.default']
action=escalate-stalled  (text stopped moving, but unverified claims still block publication)

Two consecutive hops under the plateau with a pending claim identifier is the most common real outcome, because models settle on phrasing before anyone settles on facts. The correct next step is a human checking the timeout value in the source code, not a fifth draft.

Step 4: Read the decision from the table, not from intuition

Latest hops	Pending claims	Action	What the human does
Both ≤ 0.04	None	accept-verified	Review flow and links only
Both ≤ 0.04	At least one	escalate-stalled	Verify each pending claim in a primary source
Falling, above plateau	Irrelevant	redraft-once	Run exactly one more pass and re-measure
Flat, above plateau	Irrelevant	human-first	Rewrite the weak paragraphs, then restart the series
Rebounded ≥ 0.08 or 1.5× previous	Irrelevant	escalate-oscillation	Rewrite the prompt or the source inputs, not the prose

Exit code 1 for every non-accept action means a CI job blocks the merge without extra wiring, and --json gives the same report to whatever dashboard you already run.

Step 5: Keep a one-line ledger per section

A grep-able log line per section turns the gate into a trend you can review monthly. Record the section path, the number of drafts, the final churn value, and the action, then look for sections that consistently end in escalate-oscillation. Those are usually prompt defects or contradictory source material, and no amount of redrafting will fix them.

2026-10-19 docs/auth/timeout.md drafts=4 final_churn=0.021 action=escalate-stalled claims=1

What this gate cannot tell you, and where a free tier fits the method

Three limits matter. First, churn is noisy on sections under roughly 120 tokens, because a single reworded sentence moves the ratio by several points. Second, low churn can mean the model has memorised a confident but wrong sentence, which is why the claim check exists and why accept-verified still requires a human to have resolved every identifier. Third, the thresholds here are heuristics; calibrate them on ten to twenty frozen sections from your own repository before trusting a number.

The cost driver of this workflow is the number of redraft passes, since calibration alone can mean dozens of section series. MonkeyCode's free model access and its free server option are relevant at exactly that point: the free models make a multi-pass series affordable to calibrate, and the free server option lets the gate run as a scheduled or pre-merge job without a dedicated runner. Confirm the current limits in the project's own documentation before designing a pipeline around them, because I have not measured throughput or quota behaviour here.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Who should not use this approach

Skip it if your workflow is one draft and a careful edit, since a single pass has no trajectory to measure. Skip it for regulated documentation where every sentence needs a named reviewer, because convergence cannot substitute for that review process. Skip it too if your team has no way to identify checkable claims, since the gate would then accept stable text with no ownership behind it.

If you want to test the calibration pass, start with five sections you already know are correct and five you know are wrong, then check whether churn separates them before you trust the table.