Stop Reviewing Model-Written Docs Line by Line: Freeze a Fact Bundle and an Ownership Table

Model-written documentation fails in two ways that need two different instruments. A fact failure (a flag that does not exist, a default that changed last Tuesday, a version that was never released) is machine-checkable and should never reach a human reviewer. An ownership failure (a support window nobody agreed to, a security statement nobody verified) cannot be checked at all, because there is nothing to compare it against.

Teams that treat both failure modes as a reading problem end up reviewing diffs line by line and still ship the renamed flag. The pipeline below separates the two: a fact bundle extracted from the code, and a per-block ownership table that decides what a model may draft at all. Both gates run on a free model tier and a free server option, so cost is not the reason to skip either.

Why this lands on documentation first

DEV's front page this week is full of the headcount question around AI (one example), and I have no staffing prediction to add. Documentation is simply the place where the question has a testable answer, because a doc block can be split into the part a machine can be held to and the part only a named person can own. If you cannot draw that line for prose, you cannot draw it for code either, and docs are the cheap place to practise it.

Step 1 - Classify blocks before you generate anything

The table below is the actual policy I would start from. It is not a claim about model capability; a model can produce fluent SLA text, and that is precisely the risk. The classes exist so that no unattributable sentence can be generated in the first place.

Doc block	Class	Model may draft	Human must own
CLI flag table rows	reference	yes, from bundle	nobody (machine-checked)
Quickstart command sequence	procedure	yes, from bundle	a named human who ran it
Migration steps for a breaking change	procedure	yes, from changelog	the support window and dates
Performance or cost statements	commitment	no	owner with a benchmark link
Data retention and security posture	security	no	security owner
Why we chose X over Y	rationale	no	the deciding engineer
Roadmap and SLA	commitment	no	product owner

Two classes are draftable, four are not, and the non-draftable classes are never mixed into a generated file. This keeps the owner: model marker honest: it appears only on blocks where the machine gate can actually fail.

Step 2 - Extract the fact bundle

The bundle is the only permitted source of facts for the drafting pass, and it is regenerated in CI so drift is impossible to hide. The script below pulls everything from argparse and the AST, which covers a surprising share of CLI-referenced documentation.

# tools/docsgen/fact_bundle.py
from __future__ import annotations

import argparse
import ast
import json
import sys
from pathlib import Path


def cli_facts(parser: argparse.ArgumentParser) -> dict[str, dict]:
    facts: dict[str, dict] = {}
    for action in parser._actions:
        if not action.option_strings or action.dest in ('help',):
            continue
        facts[action.option_strings[0]] = {
            'dest': action.dest,
            'default': None if action.default is None else str(action.default),
            'choices': sorted(map(str, action.choices)) if action.choices else None,
            'required': bool(action.required),
            'takes_value': action.nargs != 0,
        }
    return facts


def module_facts(root: Path) -> dict[str, list[str]]:
    out: dict[str, list[str]] = {}
    for path in sorted(root.rglob('*.py')):
        tree = ast.parse(path.read_text(encoding='utf-8'))
        out[str(path.relative_to(root))] = [
            node.name
            for node in tree.body
            if isinstance(node, (ast.FunctionDef, ast.ClassDef))
            and not node.name.startswith('_')
        ]
    return out


def build(parser: argparse.ArgumentParser, package_root: Path) -> dict:
    return {
        'schema': 'docfacts/1',
        'cli': {'prog': parser.prog, 'options': cli_facts(parser)},
        'modules': module_facts(package_root),
    }


if __name__ == '__main__':
    sys.path.insert(0, str(Path.cwd()))
    from mypkg.cli import build_parser  # replace with your entry point

    json.dump(build(build_parser(), Path('mypkg')), sys.stdout, indent=2, sort_keys=True)

mkdir -p docsgen
python tools/docsgen/fact_bundle.py > docsgen/facts.json

Commit docsgen/facts.json and review it like code. A diff that adds an option without a bundle change is now a reviewable event, and a stale bundle litigates itself on the next CI run.

Step 3 - Tag blocks with owner and class

The tag is an HTML comment, so it survives rendering and stays greppable. Each block declares who is accountable before any drafting happens.

<!-- owner: model | class: reference -->
| Flag | Default | Notes |
|---|---|---|
| `--timeout` | 10 | seconds |

<!-- owner: avery.lin | class: commitment -->
Version 2.x receives security fixes for 12 months from release.

Step 4 - Generate with a fact-only prompt

A contract beats a long instruction. The prompt below forbids the model from introducing any fact outside the bundle, and it forbids the four non-draftable classes by name. I ran the drafting pass for a small CLI reference tree on MonkeyCode's free model tier, which is enough to iterate on the prompt without metering anxiety.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

You draft only blocks of class `reference` or `procedure`.
The only permitted facts are in docsgen/facts.json. Quote flags, defaults, choices,
and paths exactly as they appear there. If a needed fact is absent, emit TODO(fact)
and stop that block.
Never write: dates, support windows, benchmarks, cost, security posture,
comparisons with other tools, or the words fast, secure, scales, best.
One `<!-- owner: model | class: ... -->` header per block. No prose outside blocks.

The operator states that the free tier includes a 10M-token allowance; treat that as a starting budget and check the current terms before you plan a full-docs pass, because allowances are the kind of thing that changes. Count tokens yourself per run rather than trusting my number, since the count depends on how much of the bundle you inline.

Step 5 - Gate the output with a checker

The checker turns the policy into exit codes, which is the only way documentation gates survive a busy week. It fails on phantom flags, wrong documented defaults, and any model-owned block in a human-owned class.

# tools/docsgen/check_docs.py
from __future__ import annotations

import json
import re
import sys
from pathlib import Path

FLAG = re.compile(r'(?<![\w-])(--[a-z][a-z0-9-]*)(?:=(\S+))?')
HEADER = re.compile(r'<!--\s*owner:\s*([^\s|]+)\s*\|\s*class:\s*([a-z]+)\s*-->')
HUMAN_OWNED = {'commitment', 'rationale', 'security'}


def check(path: Path, facts: dict) -> list[dict]:
    problems: list[dict] = []
    options = facts['cli']['options']
    owner, klass = None, None
    for lineno, line in enumerate(path.read_text(encoding='utf-8').splitlines(), 1):
        header = HEADER.search(line)
        if header:
            owner, klass = header.group(1), header.group(2)
            if owner == 'model' and klass in HUMAN_OWNED:
                problems.append({'file': path.name, 'line': lineno, 'rule': 'class-not-draftable'})
            continue
        for match in FLAG.finditer(line):
            flag, value = match.group(1), match.group(2)
            if flag not in options:
                problems.append({'file': path.name, 'line': lineno, 'rule': 'unknown-flag', 'flag': flag})
                continue
            default = options[flag]['default']
            if value is not None and default is not None and value != default:
                problems.append({
                    'file': path.name, 'line': lineno, 'rule': 'default-mismatch',
                    'flag': flag, 'documented': value, 'actual': default,
                })
    return problems


if __name__ == '__main__':
    facts = json.loads(Path(sys.argv[1]).read_text(encoding='utf-8'))
    report = [p for doc in sorted(Path('docs').rglob('*.md')) for p in check(doc, facts)]
    print(json.dumps({'schema': 'doccheck/1', 'problems': report}, indent=2))
    sys.exit(1 if report else 0)

python tools/docsgen/check_docs.py docsgen/facts.json

Before trusting either exit code, run this fixture set; a gate you have not seen fail is decoration.

Fixture	Line under test	Expected result
F1 phantom flag	a table row with --dry-runn	error unknown-flag
F2 stale default	--timeout=30 when the bundle says 10	error default-mismatch
F3 exact default	--timeout=10	pass
F4 model-owned commitment	owner: model with class: commitment	error class-not-draftable
F5 human rationale	owner: avery.lin with class: rationale	pass
F6 flag in prose	a sentence mentioning --verbose	pass, flag exists

Step 6 - Run the gates where the drafts are produced

Documentation CI is usually the job nobody wants to add a machine for. MonkeyCode's free server option is a reasonable home for this checker, because the workload is a short Python run over Markdown plus a JSON read, not a build farm. Wiring it as a scheduled job or a pre-receive check keeps docs gates off your application pipeline, which is the only way they stay enabled.

Limitations and who should not use this

The regex checker understands flags and nothing else; it will not catch a wrong deprecation date, a broken link, or a claim in prose that never mentions a flag. The bundle extractor only covers Python argparse and AST-reachable names, so Go flag sets, C getopt, and Terraform variables each need their own extractor before the gate is meaningful. The ownership table also demands a real name per block, which means teams without CODEOWNERS discipline will stall at Step 3 rather than Step 6.

Do not adopt this if your documentation is mostly narrative or marketing copy, if you cannot regenerate the bundle inside CI, or if you want the pipeline to publish on its own; drafts should stay unpublished until a named owner attests. It is also a poor fit if your docs live outside version control, since provenance is the whole mechanism.

Start with the CLI reference table, because that is where the fact bundle pays for itself on the first renamed flag. If you want to try the drafting pass before building the checker, the free tier is enough for a first run over a small docs tree, and the checker is the part worth writing yourself.