The Model Drafts the Seams; You Own the Decisions: A Placeholder-Stability Test for Generated Docs

Any sentence in your docs that restates something the repository already knows should never be reviewed — it should be generated. Everything a model writes should be the connective tissue between generated facts, and everything a human signs should be a decision with a name and a date attached. This article describes the split I now use, plus the small test that decides which side a given section falls on, so the answer is mechanical instead of a matter of taste.

The artifact is three files: a facts harvester that walks your package with ast, a drafting step that is only allowed to emit prose and placeholders, and a stability test that drafts each section twice and compares the placeholder sets. If the model reaches for the same facts both times, the section is connective and the model may own it. If the placeholder set wobbles, the section is under-specified and a human has to own it.

Why diff review fails on reference prose

Reviewing a model-written diff teaches you to look for wrong statements, which is the most expensive way to find them and the least reliable. Reference prose drifts mostly by restating a signature, a default, or an exit code that changed three commits ago, and those errors are invisible unless you happen to know the current value. Generators remove that class of error entirely, because there is nothing to restate: the number lives in an include that is rebuilt on every commit.

What generators cannot produce is the part your readers actually need to make a choice. Nobody's AST knows that retries are capped low on purpose because the downstream queue is not idempotent, and no test will ever tell a model which of two options your team rejected. That asymmetry — facts are derivable, consequences are not — is the whole basis for splitting drafting ownership by section rather than by file.

The split: derived, connective, decided

Section class	Example	Who drafts	Gate
derived	parameter tables, defaults, exit codes, error strings	generator only	facts.json --check
connective	"this call returns before the flush completes"	the model	placeholder-stability test
decided	"pick this when your queue is not idempotent"	a named human	owner + decision record + review date

Most docs teams I have seen run two classes and call it a day: generated API reference, hand-written everything else. The middle column is where models earn their keep, and it is also where the placeholder test does the useful work of pushing ambiguous sections back to a human.

Step 1 — Harvest the facts the code already owns

The harvester uses only the standard library, so it runs identically on a laptop, in a container, and on any free-tier server you happen to have. It records public functions, their required and optional parameters, the first docstring line, and module-level upper-case constants.

#!/usr/bin/env python3
"""Harvest facts the code already knows, so prose never restates them."""
import ast, json, pathlib, sys

ROOT = pathlib.Path(__file__).resolve().parents[1]
SRC, OUT = ROOT / "src", ROOT / "docs" / "_generated" / "facts.json"


def harvest(path: pathlib.Path) -> dict:
    module = path.stem
    tree = ast.parse(path.read_text(encoding="utf-8"))
    facts = {}
    for node in tree.body:
        if isinstance(node, ast.Assign):
            for target in node.targets:
                if isinstance(target, ast.Name) and target.id.isupper():
                    try:
                        value = ast.literal_eval(node.value)
                    except ValueError:
                        continue
                    facts[f"const:{module}.{target.id}"] = {
                        "value": value, "source": f"{path.name}:{node.lineno}"}
        elif isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
            if node.name.startswith("_"):
                continue
            args = [a.arg for a in node.args.args]
            n = len(node.args.defaults)
            doc = (ast.get_docstring(node) or "").strip()
            facts[f"fn:{module}.{node.name}"] = {
                "required": args[: len(args) - n],
                "optional": args[len(args) - n:],
                "summary": doc.splitlines()[0] if doc else None,
                "source": f"{path.name}:{node.lineno}"}
    return facts


def main() -> int:
    facts = {}
    for path in sorted(SRC.rglob("*.py")):
        facts.update(harvest(path))
    rendered = json.dumps(facts, indent=2, sort_keys=True) + "\n"
    if "--check" in sys.argv:
        current = OUT.read_text(encoding="utf-8") if OUT.exists() else ""
        if current != rendered:
            print("facts.json is stale; run tools/docs_facts.py", file=sys.stderr)
            return 1
        return 0
    OUT.parent.mkdir(parents=True, exist_ok=True)
    OUT.write_text(rendered, encoding="utf-8")
    print(f"wrote {len(facts)} facts")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Commit facts.json rather than gitignoring it, because the diff is the review artifact when a public signature changes. tools/docs_facts.py --check then fails any pull request that edits code without regenerating the facts file.

Step 2 — Classify each section before anyone drafts it

Classification is one line of front matter, and it is the cheapest part of the whole workflow. Put it in the section file itself so the class travels with the text instead of living in a spreadsheet nobody opens.

---
section: retry-budget
class: decided          # derived | connective | decided
owner: "@your-handle"
decision_record: docs/decisions/0007-retry-budget.md
reviewed: 2026-10-24
---

When in doubt, start a section as decided and let the stability test demote it later. Promotion from decided to connective should require a passing test run, and demotion should be automatic the moment the test fails twice in a row on unchanged inputs.

Step 3 — Give the drafting step a contract it cannot satisfy by improvising

A connective section contains prose and {{fact:KEY}} placeholders, and nothing else. Numerals are banned from prose because every number in documentation is either derivable or a decision, and both of those have a home outside the sentence. Keep the drafting client behind one environment variable so swapping providers is a one-line change rather than a rewrite.

The drafting command below is provider-agnostic on purpose. MonkeyCode's free model access is a reasonable default for this step, since drafting seams is cheap, repetitive work that benefits more from a stable prompt than from a frontier model. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

# Any client that reads the seam file on stdin and writes markdown on stdout works.
export DRAFT_CMD='python tools/draft.py --temperature 0'
printf 'Draft prose only. Use {{fact:KEY}} for every signature, default, or code.\n\n' | cat - docs/retry.seam.md | $DRAFT_CMD

A four-line lint catches the common failure where the model decides a placeholder is unnecessary and inlines the value instead. It is intentionally dumb: if the prose contains a digit, the sentence failed.

NUMERAL = re.compile(r"\b\d+(?:\.\d+)?\b")
if NUMERAL.search(strip_placeholders(text)):
    fail("connective prose must not contain numerals; use a {{fact:...}} placeholder")

Step 4 — Use a placeholder-stability test as the ownership test

This is the part that replaces judgment calls. Draft the same section twice with identical inputs, extract the placeholder keys from each run, and compare the sets; identical sets mean the section has a stable fact skeleton the model can write around.

PLACEHOLDER = re.compile(r"\{\{fact:([^}]+)\}\}")

for section in sorted(pathlib.Path("docs").glob("**/*.seam.md")):
    runs = [set(PLACEHOLDER.findall(draft(section))) for _ in range(2)]
    if runs[0] != runs[1]:
        print(f"UNSTABLE {section}: {sorted(runs[0] ^ runs[1])}")
    for key in runs[0] - set(facts):
        print(f"UNKNOWN  {section}: {key} is not in facts.json")

Two outcomes matter. A symmetric difference means the section is ambiguous about which facts govern it, so it belongs to a human until someone rewrites it more tightly. A key that is absent from facts.json means the model invented a fact handle, which is a hard failure and not a warning.

Step 5 — Make the human gate a signature with a date

decided sections need three things: a named owner, a link to the decision record that explains the tradeoff, and a review date. A pull request that touches one of these sections without refreshing reviewed is rejected by the lint, and an owner who has left the team is visible in the diff rather than discovered a year later.

python tools/docs_lint.py --max-age 90
# FAIL docs/retry-policy.md: class=decided, reviewed 2026-05-02 (177 days), owner @former

This is the only part of the workflow that costs real human time, which is exactly why the other two classes exist: they shrink the set of pages that need a signature.

Step 6 — Wire the three checks into one command

Three checks with three exit codes get skipped; one command gets run. Keep them in make docs so the pre-commit hook, CI, and a human typing locally all do the same thing.

.PHONY: docs
docs:
    python tools/docs_facts.py --check
    python tools/docs_lint.py --max-age 90
    python tools/prose_stability.py

The whole suite is standard-library Python and finishes in seconds on a modest repository. That makes MonkeyCode's free server option a practical place to run it as a scheduled job if you do not want to spend a paid CI runner on a lint that touches no application code. Treat that availability as variable rather than guaranteed, and keep the checks runnable anywhere so a scheduling change never blocks a release.

Limitations and who should skip this

The harvester reads syntax, not meaning: it will happily record a parameter whose name lies, and it cannot tell you whether the docstring is accurate. The stability test is a screening tool, not a proof — two identical drafts reduce the chance of improvisation without eliminating it, and a low-temperature model can be stably wrong. decided sections still depend on someone caring enough to write a decision record, which is the failure mode this workflow cannot fix for you.

Skip it if your documentation is mostly conceptual, if your product is not a code artifact with an importable surface, or if your team will not maintain review dates. For a library with a real API, a CLI, and a handful of configuration constants, the split removes the majority of reference prose from human review and leaves people owning only the pages where judgment is the actual product.

If you try only one piece, make it Step 1: a committed facts.json with a --check flag is a small change that stops a whole category of stale sentences from ever reaching review.