Run the Draft Twice: Use Generation Variance to Decide What a Human Must Own

Reviewing a model-drafted documentation page front to back, line by line, is the wrong default. Most lines on such a page are formatting habits that carry no risk, while a small number of lines decide whether a reader breaks production. Run the same generation job twice against frozen inputs, diff the output at block level, and let the variance tell you where human attention actually belongs.

The method is cheap to build and boring to operate. It does not require a claim taxonomy, a policy document, or a reviewer who classifies sentences before reading them. It produces one artifact: a ledger that says which drafted blocks were stable across runs, which wobbled, and which were invented from scratch.

Why variance is a useful ownership signal

A single generation pass gives you text and no information about the model's confidence in any part of it. Every sentence arrives with identical authority, so the reviewer applies an even suspicion budget across the whole page. That budget is misallocated, because a sentence about heading hierarchy and a sentence about retry semantics do not deserve the same scrutiny.

Running the job twice surfaces a signal that a single pass cannot. Blocks that survive byte-for-byte are the model's stable habits, usually phrasing, structure, and formatting. Blocks that change between passes are places where the model has no reliable answer, which is exactly where the page is genuinely undecided. The second pass is not a vote to be averaged; it is a probe.

This inverts the usual order of operations. Instead of asking a human to classify before reading, you measure stability mechanically, then route only the unstable material to a person. The reviewer's judgment is spent where it changes the outcome.

Define the unit before you measure anything

Sentence-level diffs produce noise, because clause order and punctuation wobble for reasons that carry no information. Heading-level diffs are too coarse, since one unstable sentence inside an otherwise stable section disappears into a passing grade. A paragraph block scoped under its nearest heading is the practical unit: small enough to assign to one owner, large enough that a rewrite registers as a single change.

Be explicit that this is triage, not verification. A confidently wrong block can be perfectly stable across runs, and a trivially correct block can wobble because the model rephrased it. Variance tells you where to look first; it never tells you what is true.

A runnable two-pass variance ledger

The harness below segments each pass into heading-scoped blocks, aligns blocks across the two passes with difflib, and classifies every pair as STABLE, WOBBLE, or UNSTABLE. It writes a JSON record for review and, with --gate, exits non-zero when unstable blocks appear in CI.

#!/usr/bin/env python3
"""variance_ledger.py: classify drafted documentation blocks by cross-run stability."""
from __future__ import annotations

import argparse
import hashlib
import json
import pathlib
import re
from difflib import SequenceMatcher

HEADING = re.compile(r"^(#{1,6})\s+(.*)$")


def segment(markdown: str) -> list[dict]:
    """Split Markdown into blocks keyed by the nearest preceding heading."""
    blocks: list[dict] = []
    buf: list[str] = []
    heading = "(preamble)"

    def flush() -> None:
        text = "\n".join(buf).strip()
        if text:
            blocks.append({"heading": heading, "text": text})
        buf.clear()

    for line in markdown.splitlines():
        match = HEADING.match(line)
        if match:
            flush()
            heading = match.group(2).strip()
        else:
            buf.append(line)
    flush()
    return blocks


def similarity(a: str, b: str) -> float:
    return SequenceMatcher(None, a, b).ratio()


def classify(score: float, stable: float, wobble: float) -> str:
    if score >= stable:
        return "STABLE"
    if score >= wobble:
        return "WOBBLE"
    return "UNSTABLE"


def _row(a_index, b_index, heading, score, label, a_text, b_text) -> dict:
    return {
        "a_index": a_index,
        "b_index": b_index,
        "heading": heading,
        "score": round(score, 4),
        "class": label,
        "a_text": a_text,
        "b_text": b_text,
    }


def align(a_blocks: list[dict], b_blocks: list[dict], stable: float, wobble: float) -> list[dict]:
    """Greedy best-match alignment; a heading mismatch caps the score at zero."""
    remaining = list(enumerate(b_blocks))
    rows: list[dict] = []
    for i, ab in enumerate(a_blocks):
        if not remaining:
            rows.append(_row(i, None, ab["heading"], 0.0, "UNSTABLE", ab["text"], ""))
            continue
        best_j, best_score = -1, -1.0
        for j, bb in remaining:
            score = similarity(ab["text"], bb["text"]) if ab["heading"] == bb["heading"] else 0.0
            if score > best_score:
                best_j, best_score = j, score
        remaining = [(j, bb) for j, bb in remaining if j != best_j]
        rows.append(_row(i, best_j, ab["heading"], best_score,
                         classify(best_score, stable, wobble),
                         ab["text"], b_blocks[best_j]["text"]))
    for j, bb in remaining:
        rows.append(_row(None, j, bb["heading"], 0.0, "UNSTABLE", "", bb["text"]))
    return rows


def main() -> int:
    parser = argparse.ArgumentParser(description="Cross-run stability ledger for drafted docs.")
    parser.add_argument("pass_a", type=pathlib.Path)
    parser.add_argument("pass_b", type=pathlib.Path)
    parser.add_argument("--stable", type=float, default=0.97)
    parser.add_argument("--wobble", type=float, default=0.80)
    parser.add_argument("--json-out", type=pathlib.Path, default=pathlib.Path("variance.json"))
    parser.add_argument("--gate", action="store_true")
    args = parser.parse_args()

    text_a = args.pass_a.read_text(encoding="utf-8")
    text_b = args.pass_b.read_text(encoding="utf-8")
    rows = align(segment(text_a), segment(text_b), args.stable, args.wobble)

    record = {
        "inputs": {
            "pass_a": hashlib.sha256(text_a.encode()).hexdigest()[:16],
            "pass_b": hashlib.sha256(text_b.encode()).hexdigest()[:16],
            "stable": args.stable,
            "wobble": args.wobble,
        },
        "counts": {
            label: sum(1 for r in rows if r["class"] == label)
            for label in ("STABLE", "WOBBLE", "UNSTABLE")
        },
        "rows": rows,
    }
    args.json_out.write_text(json.dumps(record, indent=2) + "\n", encoding="utf-8")

    for row in rows:
        print(f'{row["class"]:<8} {row["score"]:.3f}  {row["heading"]}')
    print(json.dumps(record["counts"]))
    if args.gate and record["counts"]["UNSTABLE"]:
        return 1
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Invoke it on the two saved outputs, keeping the JSON next to the pull request that consumes it:

python variance_ledger.py pass_a.md pass_b.md \
  --stable 0.97 --wobble 0.80 \
  --json-out variance.json --gate

The report shape looks like the block below. These values come from a synthetic two-pass fixture built to exercise every branch of the classifier; they illustrate the output format and are not measurements of any model.

STABLE   1.000  Authentication
STABLE   1.000  Error codes
WOBBLE   0.884  Rate limits
UNSTABLE 0.412  Migration order
UNSTABLE 0.000  Deprecated flags
{"STABLE": 14, "WOBBLE": 5, "UNSTABLE": 2}

The workflow, in six steps

1. Freeze the generation inputs. Prompt text, source snapshots, tool version, and any retrieval corpus go into one digest. Both passes must consume the same digest, otherwise your variance measures input drift instead of model uncertainty.

2. Run both passes in the same environment. Different runtime versions, locale settings, or clock-dependent context inject variance that has nothing to do with the model, and it will land in your UNSTABLE bucket.

3. Segment and align. Run the ledger script above and commit the JSON alongside the draft branch. A reviewer six months later cannot reconstruct why a block was escalated without that record.

4. Calibrate the thresholds on your own pages. The defaults of 0.97 and 0.80 are starting points, not validated constants. Tune them until STABLE corresponds to prose you would have accepted without reading closely.

5. Route each class to a different owner. STABLE blocks enter the draft PR as drafted prose. WOBBLE blocks go to the section owner with both variants shown side by side. UNSTABLE blocks get a human-written replacement rather than a third generation pass.

6. Gate on the unstable count, not on total variance. A slowly rising unstable count usually means source facts are drifting under the page, which is a documentation problem before it is a review problem.

Two passes double the generation calls, which is the main practical objection to this workflow. MonkeyCode's free model access removes the marginal cost of the second pass, and its free server option lets both passes run in one environment, so environment drift is not recorded as model variance. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Those two availability facts are operator-supplied; nothing here should be read as a claim about quotas, hardware, model identity, or permanence, and you should verify current terms before depending on them.

Limitations you should accept before adopting this

Stability is not correctness, and the two are easy to confuse once the ledger produces green rows. A block that says retries are idempotent will read identically in both passes whether or not that is true. Keep a separate mechanism for facts that must be owned rather than measured, because variance has no opinion about accuracy.

The alignment logic is deliberately simple and therefore brittle. A pass that renames a heading or reorders whole sections produces zero-score rows that look like instability but are really structural divergence. Normalize whitespace and code fences before comparing, and treat repeated zero scores under one heading as a signal to inspect the prompt rather than the output.

The overhead is real even when generation is free. Someone has to freeze inputs, store both outputs, run the ledger, and read the JSON, and on a two-paragraph page that overhead exceeds the value. This workflow pays off on long reference pages where a single wrong block carries operational cost.

Who should not use this approach

Teams whose single reviewer already reads every line of every page before publish gain nothing here, because the routing step duplicates work they are already doing. Pages that are mostly captured command output and logs have almost no drafted prose to classify, so the ledger will report a comfortable all-STABLE result that means nothing.

Regulated or contractual content should keep a named human author on every claim regardless of stability, since the question is accountability rather than model consistency. Finally, if you cannot pin the generation inputs, skip this method entirely until you can, because unattributable variance is worse than no measurement at all.

The artifact is the argument, so keep variance.json attached to the draft PR and let the recorded scores explain the routing to whoever reviews it next. If you want to reproduce the two-pass ledger end to end without paying for a second generation run, the free model access and free server option are enough to try the workflow on a real page of your own documentation.