Doc Drift Is a Bug: A Claim-Checking Loop for Model-Generated Docs

Generated docs fail most often at the boundary between prose and code, where a flag name or a signature silently disagrees with the source. A free-tier model can draft that prose quickly, but the disagreement is a bug that better prompting alone will not fix. The reliable fix is a verification loop that extracts mechanical claims from the doc and diffs them against the source tree. A human then reviews only the semantic parts, and the loop runs on a free server, which keeps the pipeline cost near zero.

Why model-generated docs drift

Every CLI reference contains two kinds of statements with very different trust requirements. Semantic guidance includes examples, caveats, and trade-offs, and it genuinely needs a human writer who understands the tool. Mechanical claims include flag names, defaults, argument arity, and function signatures, and they are verifiable facts that a script can check in milliseconds. Model-generated docs tend to produce plausible mechanical claims that are subtly wrong, because the model generalizes from similar tools instead of reading your source. That is why a doc drafted by a model needs a claim-checking stage before any human signs it.

The five-stage verification loop

The pipeline has five stages, and only the first one uses a model:

1. Draft — a free-tier model writes a first-pass reference doc from the source tree.
2. Extract — a script pulls mechanical claims such as flags and signatures from the Markdown.
3. Verify — the same script pulls ground truth from the Python AST.
4. Diff — the script compares both sets and prints a drift report.
5. Review — a human owns the semantic sections and the final sign-off.

Stages two through four are a single script, so the verification loop is reproducible in any repository with two commands. The human remains the owner of the final document, but the review surface shrinks from the whole file to a short list of discrepancies.

Stage 1: Draft with a free-tier model

Point a free-tier model at your CLI module and ask for a narrow reference table rather than a full manual. MonkeyCode's free model access is sufficient for this drafting task, because the output is a first pass that the verification loop will check mechanically. Disclosure: This article was prepared as part of MonkeyCode's product outreach. A prompt such as the one below keeps the scope small and still produces useful raw material:

Read src/cli.py and produce a Markdown table of every CLI flag, its default value, and one usage example. List only flags that exist in the source; do not infer flags from similar tools.

Expect the model to invent some flags anyway, because that is what language models do when they fill gaps. The next stage exists precisely to catch those inventions.

Stages 2-4: Extract, compare, report

Save the following script as docdrift.py in the repository root. It has no third-party dependencies and works on Python 3.10 and later.

#!/usr/bin/env python3
"""docdrift.py — diff mechanical claims in Markdown docs against Python source.

Usage:
    python docdrift.py docs/cli.md src/cli.py

Exit code 1 when drift is found, 0 when docs and code agree.
"""

import ast
import re
import sys
from pathlib import Path

FLAG_RE = re.compile(r"`(--?[\w-]+)`")
SIG_RE = re.compile(r"`([a-z_]\w*)\(([^)]*)\)`", re.IGNORECASE)


def doc_claims(md_path: Path) -> tuple[set[str], set[str]]:
    text = md_path.read_text(encoding="utf-8")
    flags = set(FLAG_RE.findall(text))
    signatures = set()
    for name, args in SIG_RE.findall(text):
        normalized = ", ".join(a.strip() for a in args.split(","))
        signatures.add(f"{name}({normalized})")
    return flags, signatures


def source_flags(py_path: Path) -> set[str]:
    tree = ast.parse(py_path.read_text(encoding="utf-8"))
    flags = set()
    for node in ast.walk(tree):
        if not isinstance(node, ast.Call):
            continue
        func = node.func
        if isinstance(func, ast.Attribute) and func.attr in {"add_argument", "option"}:
            for arg in node.args:
                if isinstance(arg, ast.Constant) and str(arg.value).startswith("-"):
                    flags.add(arg.value)
            for kw in node.keywords:
                if kw.arg in {"name", "long", "short"} and isinstance(kw.value, ast.Constant):
                    flags.add(str(kw.value.value))
    return flags


def source_signatures(py_path: Path) -> set[str]:
    tree = ast.parse(py_path.read_text(encoding="utf-8"))
    signatures = set()
    for node in ast.walk(tree):
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and not node.name.startswith("_"):
            args = [a.arg for a in node.args.args + node.args.kwonlyargs]
            signatures.add(f"{node.name}({', '.join(args)})")
    return signatures


def main() -> int:
    if len(sys.argv) != 3:
        print(__doc__)
        return 2
    md_path, py_path = Path(sys.argv[1]), Path(sys.argv[2])
    md_flags, md_sigs = doc_claims(md_path)
    src_flags = source_flags(py_path)
    src_sigs = source_signatures(py_path)

    missing_flags = md_flags - src_flags
    undocumented_flags = src_flags - md_flags
    missing_sigs = md_sigs - src_sigs
    undocumented_sigs = src_sigs - md_sigs

    print("=== Drift report ===")
    print(f"Flags documented but not in code: {sorted(missing_flags) or 'none'}")
    print(f"Flags in code but not documented: {sorted(undocumented_flags) or 'none'}")
    print(f"Signatures documented but not in code: {sorted(missing_sigs) or 'none'}")
    print(f"Signatures in code but not documented: {sorted(undocumented_sigs) or 'none'}")

    drifted = missing_flags or undocumented_flags or missing_sigs or undocumented_sigs
    return 1 if drifted else 0


if __name__ == "__main__":
    raise SystemExit(main())

Run the script against the drafted doc and the CLI module that the doc describes:

python docdrift.py docs/cli.md src/cli.py

A realistic report looks like this:

=== Drift report ===
Flags documented but not in code: {'--force'}
Flags in code but not documented: {'--dry-run', '--timeout'}
Signatures documented but not in code: {'parse(input)'}
Signatures in code but not documented: {'validate(config, strict)'}

Each line maps to a concrete review action. --force is either a hallucination to delete or a removed feature to annotate; --dry-run and --timeout are coverage gaps that need one row in the table; parse(input) suggests a renamed or deleted function; validate(config, strict) needs a documented entry. The human decides which case applies, and the script guarantees that no mechanical claim slips through unnoticed.

Stage 5: Human review owns the semantics

The drift report tells the reviewer exactly where attention is required, and nothing more. A missing flag is a hallucination or a removed feature, while an undocumented flag is a coverage gap that needs one line in the table. The human still owns the semantic sections, because no script can judge whether an example is misleading or whether a warning reflects a real edge case. The verification loop therefore shrinks the review surface from the entire document to a short list of mechanical discrepancies plus the drafted prose.

Running the loop on a free server

The script uses only the standard library, so it runs anywhere Python 3.10 exists, including MonkeyCode's free server option, a GitHub Actions runner, or a pre-commit hook. The job parses two files and prints four lines, which makes it a poor candidate for a dedicated paid runner and a good candidate for the cheapest available execution slot. The exit code lets the pipeline fail the build when drift appears, which turns documentation into a testable artifact instead of a trust exercise. Teams that already generate docs with a model can add this script before the review step and see what the drift report surfaces.

Limitations and who should skip this

The script checks mechanical claims only, and that boundary is deliberate. It cannot tell whether a default value in the doc matches the source, because defaults live in code paths that the AST walk does not evaluate; it cannot judge whether an example is idiomatic; it cannot verify prose claims about behavior, performance, or security. A reviewer who treats the drift report as a full correctness proof will still ship misleading docs. The loop is a sieve for factual drift, not a substitute for a human who understands the tool. Teams with a tiny CLI surface and one maintainer may find the script overhead heavier than the drift it catches, and projects whose docs are mostly conceptual will get little signal from a claim diff. If your documentation is already reviewed line by line by a domain expert, the verification loop adds process without adding insight.

The practical takeaway

Model-generated docs need a mechanical verification stage before they enter human review, and that stage costs almost nothing to build. A free-tier model drafts the prose, a standard-library script checks the claims, and a human owns the semantics and the final sign-off. That division turns documentation from a trust exercise into a testable artifact, and it catches the exact class of bug that generated docs introduce most often.