Regenerate Docs Without Silent Rewrites: A Fact-Anchored Diff Gate

Most model-written documentation fails for a boring reason: the draft rewrites lines nobody asked it to touch. A reviewer facing a full-file diff cannot separate a corrected timeout value from a reworded security note. The gate below turns that diff into one exit code and a short reason, so a human opens the page only when something material moved. It is roughly sixty lines of Python, and you can run the fixture table at the end before you trust it.

Nothing here depends on a single model vendor; the drafting step is replaceable and the gate is not. MonkeyCode's free model access is enough to run the regeneration step described here, and its free server option is enough to host the scheduled check that runs the gate. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Everything below is written so the workflow still works if you swap the generator for another one tomorrow.

Why a line diff cannot carry this decision

A regenerated page usually mixes three kinds of change inside one diff. Some lines carry a value copied out of code, some lines are narrative the model rephrased, and a few lines are opinions a human wrote on purpose. Line-based review weights all three equally, so review cost grows with file size instead of with risk. Splitting the three classes before a human reads anything is the whole design.

The workflow below assumes you already generate pages from source facts and that your docs live in the same repository as the code they describe. If the second half is false, fix that first, because a gate that runs after publishing is a monitoring tool rather than a gate.

Step 1: anchor the facts the model may copy

Every value that comes from code gets an invisible anchor line directly under the sentence that uses it. The identifier is stable across releases, and the value is copied mechanically rather than paraphrased by the model. One fact per line keeps the parser trivial and the diff readable.

Requests time out after 30 seconds by default; raise it for bulk exports.

<!-- fact: http.client.timeout_seconds = 30 -->

When the source of truth changes, the injection step rewrites only the anchor value. The model may then rewrite the surrounding sentence, but it cannot quietly change the number without the anchor changing with it. That coupling is what makes the next step possible.

Step 2: fence the paragraphs a model must not draft

The second marker is a fenced region for text a human owns: incident notes, deprecation warnings, migration advice. Inside those markers the model may propose text in a comment, but the published paragraph must remain human-authored. The gate treats any edit inside the fence as a hard stop rather than a review comment.

<!-- human:start -->
Do not raise this above 60 seconds in the ingestion worker. The upstream
proxy closes idle connections at that point, and the resulting failure looks
like a data gap rather than a timeout, which cost us a full day to diagnose.
<!-- human:end -->

Being strict here is deliberate. A model that rephrases a debugging note usually preserves the words and loses the constraint, and the loss is invisible until the next person argues with the proxy.

Step 3: classify the draft with exit codes

The script compares a published page against a regenerated draft and returns one of five codes. Each code maps to a different human action, which is the only reason to have codes at all instead of a printed warning.

#!/usr/bin/env python3
# docgate.py: gate a regenerated doc draft against the published version.
import argparse
import difflib
import json
import re
import sys

FACT_RE = re.compile(r'<!--\s*fact:\s*([A-Za-z0-9_.\-]+)\s*=\s*(.*?)\s*-->')
HUMAN_START = '<!-- human:start'
HUMAN_END = '<!-- human:end -->'


def read(path):
    with open(path, encoding='utf-8') as fh:
        return fh.read()


def facts(text):
    found = {}
    for line in text.splitlines():
        match = FACT_RE.search(line)
        if match:
            found[match.group(1)] = match.group(2).strip()
    return found


def human_spans(lines):
    spans, start = [], None
    for idx, line in enumerate(lines):
        stripped = line.strip()
        if stripped.startswith(HUMAN_START):
            start = idx
        elif stripped == HUMAN_END and start is not None:
            spans.append((start, idx))
            start = None
    return spans


def in_spans(idx, spans):
    return any(low <= idx <= high for low, high in spans)


def human_text(text):
    lines = text.splitlines()
    return [lines[i].strip() for low, high in human_spans(lines) for i in range(low, high + 1)]


def prose(text):
    lines = text.splitlines()
    spans = human_spans(lines)
    kept = []
    for idx, line in enumerate(lines):
        stripped = line.strip()
        if not stripped or in_spans(idx, spans) or FACT_RE.search(line):
            continue
        kept.append(stripped)
    return kept


def churn(old_lines, new_lines):
    matcher = difflib.SequenceMatcher(a=old_lines, b=new_lines, autojunk=False)
    total = 0
    for tag, i1, i2, j1, j2 in matcher.get_opcodes():
        if tag != 'equal':
            total += max(i2 - i1, j2 - j1)
    return total


def verdict(old_text, new_text, owners, budget):
    if human_text(old_text) != human_text(new_text):
        return 3, 'blocked: human-owned region edited'

    old_facts, new_facts = facts(old_text), facts(new_text)
    changed = sorted(k for k in old_facts if k in new_facts and old_facts[k] != new_facts[k])
    added = sorted(set(new_facts) - set(old_facts))
    dropped = sorted(set(old_facts) - set(new_facts))

    if dropped:
        return 3, 'blocked: fact removed without a human decision: ' + ', '.join(dropped)
    unowned = sorted(k for k in added + changed if k not in owners)
    if unowned:
        return 3, 'blocked: no named owner for: ' + ', '.join(unowned)

    delta = churn(prose(old_text), prose(new_text))
    ratio = delta / max(1, len(prose(old_text)))

    if changed or added:
        if delta == 0:
            return 2, 'human rewrite required: facts moved, prose did not'
        return 0, 'fact change with prose change: route to owner review'
    if delta == 0:
        return 0, 'no-op: draft matches the published page'
    if ratio > budget:
        return 4, f'churn over budget: {ratio:.2f} > {budget:.2f}'
    return 0, f'prose-only change: churn {ratio:.2f} within budget'


def main(argv=None):
    parser = argparse.ArgumentParser(description='Gate a regenerated doc against the published page.')
    parser.add_argument('published')
    parser.add_argument('draft')
    parser.add_argument('--owners', default='docgate.owners.json')
    parser.add_argument('--budget', type=float, default=0.25)
    parser.add_argument('--json', action='store_true')
    args = parser.parse_args(argv)

    try:
        with open(args.owners, encoding='utf-8') as fh:
            owners = json.load(fh)
    except FileNotFoundError:
        owners = {}

    code, message = verdict(read(args.published), read(args.draft), owners, args.budget)
    if args.json:
        print(json.dumps({'exit': code, 'verdict': message}))
    else:
        print(f'[docgate] {message}')
    return code


if __name__ == '__main__':
    sys.exit(main())

The owners file is a flat map from fact identifier to a handle, and a fact without an entry is a blocked draft rather than an anonymous merge. That single rule is what stops a generator from introducing a new claim nobody has agreed to maintain.

{
  "http.client.timeout_seconds": "@averylin"
}

Step 4: verify the gate against fixtures before wiring it in

Eight tiny file pairs cover the decisions the gate can make, and each pair is small enough to read in a minute. The table states the exit code each fixture is written to produce, so a mismatch is a bug in the script rather than in your judgement.

Fixture	What changed	Expected exit	Who acts
noop	nothing	0	nobody
typo-fix	one prose word	0	nobody
fact-and-prose	anchor 30 to 15 plus sentence	0	named owner reviews
fact-only	anchor changed, prose stale	2	writer rewrites the sentence
human-note-edited	one word inside the fence	3	blocked, model draft discarded
fact-removed	anchor deleted	3	human decides or restores
new-fact-unowned	new anchor, no owner	3	assign an owner first
full-rewrite	every prose line reworded, no fact change	4	regenerate with a narrower prompt

Run the whole set with a loop that prints the code next to the fixture name, and treat any deviation as a failing test rather than a warning.

for case in fixtures/*/; do
  python docgate.py "${case}published.md" "${case}draft.md" \
    --owners "${case}owners.json" --budget 0.25 --json
  status=$?
  echo "$case -> $status"
done

Keep the fixtures in the repository next to the script, because the gate will be edited by someone who did not write it. A gate with no fixtures becomes a gate that returns zero because a regex broke.

Step 5: put the generator behind the gate, not beside it

The regeneration job writes a draft file and exits; it never touches the published page. A second step runs the gate, and only exit code 0 lets the draft replace the previous version. Codes 2, 3 and 4 fail the job and leave the published page exactly as it was, which keeps a bad draft from becoming a bad page while everyone is asleep.

In the loop I run, the drafting step is a model call against MonkeyCode's free model access, invoked from a small script on the free server option. The appeal is not speed; it is that a scheduled job which regenerates every page after every merge is cheap enough to leave running. That frequency is only safe because the gate, not the model, decides what reaches the page.

Limitations and who should not use this

The gate compares text, so a value that formats differently between runs, such as 30 and 30.0, looks like a fact change on every execution. Normalise values in the injection step instead of loosening the comparison in the gate. Fence comparison is also exact, so a whitespace-only edit inside a human region blocks a run until someone restores the original text.

The larger risk is coverage rather than correctness. The gate protects anchors you wrote and nothing else, so an unanchored claim can still be reworded freely by any draft. Prose churn counts changed lines, not changed meaning, and a model can reverse a recommendation inside a single short sentence. I have not benchmarked the review time this saves, and the 0.25 budget is a starting value to tune against your own history, not a measured optimum.

Skip this approach if your documentation is mostly tutorial narrative with no code-derived values, or if your repository has no automated check that can fail. One person maintaining a small docs set who already reads every diff will find the markers pure overhead. Also skip it if nobody is willing to be named as an owner, because the blocked path then becomes a permanently red build.

What to keep if you keep one thing

The fixture table is the cheapest part to copy: eight file pairs and a loop, runnable in an afternoon. It tells you whether your gate is correct before it ever blocks a real pull request, and it survives every change of drafting model you make afterwards. Start there, add anchors to two pages, and only then wire the gate into the merge path.