Documentation written by a model is only as trustworthy as the facts it was handed, so extract those facts mechanically before a single sentence is drafted. The workflow below inverts the usual order: a deterministic extractor builds a fact manifest from your CLI help output, environment variables, and self-test exit codes; the model then writes only the connective prose between placeholders it cannot alter. A short lexical gate fails the build whenever model-drafted prose contains a number or identifier that has no manifest entry, which converts an unbounded review problem into a bounded diff review.
1. Why prose-first generation cannot be reviewed
When a model reads your repository and writes a paragraph, every claim in that paragraph arrives without a citation and without a diff boundary. Reviewers then face the worst possible question: is this sentence wrong, or merely unfamiliar to me. Studies of review behavior are not needed to see the asymmetry — verifying ten numbers costs minutes, verifying ten sentences costs an afternoon. Inverting the direction fixes the asymmetry at the source: facts come from tools, prose comes from the model, and the gate only has to check the seam between them.
2. Step one: define a manifest schema with an owner field
Each fact needs four fields, and the fourth one matters more than the other three. id names the fact, kind classifies it, value holds the exact string that may appear in prose, and owner records whether a machine or a named human is accountable for it. The owner field is what stops the manifest from quietly absorbing opinions — a flag name has a machine owner, a deprecation promise never will.
{
"facts": [
{"id": "flag:--timeout", "kind": "flag", "value": "--timeout", "source": "mytool --help", "owner": "machine"},
{"id": "env:MYTOOL_TOKEN", "kind": "env", "value": "MYTOOL_TOKEN", "source": "src/ (grep)", "owner": "machine"},
{"id": "exit:2", "kind": "exit_code", "value": "2", "source": "mytool.selftest", "owner": "machine"}
]
}
3. Step two: extract with code, not with the model
Keep extraction boring and reproducible, because a wrong fact propagates into every paragraph that cites it. The script below captures --help output, greps for environment variable names, and asks a self-test subcommand to emit exit codes as JSON; adapt the three extraction functions to your own tool. Run it in CI on the same commit that builds the docs, and commit the manifest so reviewers can diff facts separately from prose.
# tools/extract_facts.py (illustrative; adapt paths and entry points)
import json, re, subprocess, sys
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
facts = []
def add(kind, value, source, owner="machine"):
facts.append({"id": f"{kind}:{value}", "kind": kind, "value": value,
"source": source, "owner": owner})
def run(cmd):
return subprocess.run(cmd, capture_output=True, text=True, check=True).stdout
help_text = run([sys.executable, "-m", "mytool", "--help"])
for flag in sorted(set(re.findall(r"(?<![\w-])--[a-z][a-z0-9-]+", help_text))):
add("flag", flag, "mytool --help")
src_text = "\n".join(p.read_text() for p in (ROOT / "src").rglob("*.py"))
for name in sorted(set(re.findall(r"environ(?:\.get)?\[?[\"']([A-Z][A-Z0-9_]+)[\"']", src_text))):
add("env", name, "src/ (regex)")
codes = json.loads(run([sys.executable, "-m", "mytool.selftest", "--emit-exit-codes"]))
for code in codes:
add("exit_code", str(code), "mytool.selftest")
out = ROOT / "docs" / "_facts.json"
out.write_text(json.dumps({"facts": facts}, indent=2, sort_keys=True))
print(f"wrote {len(facts)} facts to {out}")
4. Step three: render placeholders, then let the model fill prose slots
Generate the skeleton yourself so the model never types a fact value at all. Your template holds {{fact:flag:--timeout}} placeholders plus marked drafting regions, and the renderer substitutes values after the model returns, which makes drift structurally impossible for extracted facts. Marked regions keep the boundary visible in review: <!-- ai:start --> through <!-- ai:end --> marks prose a model wrote, and TODO(human): marks slots only a person may fill. This is the point where hosted tooling helps rather than decorates the method — free model access is enough for drafting connective paragraphs, and the free server option is enough to run render-plus-check outside a laptop. Both availabilities are operator-supplied, so confirm the current terms yourself before depending on them.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The drafting contract you give the model should be short enough to enforce mechanically. Ask it to rewrite only text inside marked regions, forbid new numerals and new backticked identifiers, and require it to leave TODO(human): markers untouched rather than guessing. A model that follows those three rules produces prose you can review at the sentence level in minutes.
5. Step four: gate the seam with a lexical check
The checker does not try to understand meaning, and that is a feature rather than a compromise. It extracts every number and every backticked token from model-drafted regions, then fails if any token lacks a manifest entry or an explicit allowlist entry. Lexical checks are cheap, deterministic, and impossible to argue with in a review thread.
# tools/check_grounding.py (illustrative)
import json, re, sys
from pathlib import Path
facts = json.loads(Path("docs/_facts.json").read_text())["facts"]
known = {f["value"] for f in facts}
allow_path = Path("docs/_allow.json")
allow = set(json.loads(allow_path.read_text())) if allow_path.exists() else set()
text = Path(sys.argv[1]).read_text()
regions = re.findall(r"<!-- ai:start -->(.*?)<!-- ai:end -->", text, re.S)
if not regions:
print("no model-drafted regions found")
raise SystemExit(0)
prose = "\n".join(regions)
tokens = set(re.findall(r"(?<![\w.])\d+(?:\.\d+)*(?![\w])", prose))
tokens |= set(re.findall(r"`([^`]+)`", prose))
bad = [t for t in sorted(tokens) if t not in known and t not in allow]
for t in bad:
print("ungrounded token:", t)
raise SystemExit(1 if bad else 0)
Run it as python tools/extract_facts.py && python tools/check_grounding.py docs/README.generated.md and treat a non-zero exit as a hard build failure. When a token is legitimately new, the fix is to add a fact with a source, not to widen the allowlist silently; review the allowlist diff monthly or it becomes the place honesty goes to die.
6. Step five: decide what a human must own
The gate protects extracted facts but says nothing about judgment, so assign ownership explicitly and in writing. Anything that promises behavior, implies a timeline, or describes risk belongs to a named person regardless of how confident the draft sounds. The table below is the policy I would start from, not a universal standard.
| Content type | Model may draft | Human must own |
|---|---|---|
| Flag and env var descriptions | Yes, from manifest entries | Sign-off on wording that implies defaults |
| Exit code tables | Yes, mechanical rendering | Meaning of new codes |
| "Why this exists" rationale | Bullet scaffolding only | Final rationale and tradeoffs |
| Migration and deprecation dates | No | Named maintainer |
| Security and permissions guidance | No | Named maintainer plus reviewer |
| Support and SLA statements | No | Owner of the commitment |
7. Limitations and who should skip this
Extraction bugs propagate silently, and the gate cannot tell a stale fact from a fresh one, so the manifest is only as good as the command that produced it. The gate is also lexical: it catches invented numbers and identifiers, not invented causality, hedging, or emphasis. A paragraph can pass with every token grounded and still mislead, which is why marked regions plus human-owned slots remain mandatory rather than optional.
Skip this approach when your documentation has no machine-extractable source, such as design essays or incident narratives. Skip it too if you cannot run a build step on the same commit as the docs, because a gate that runs late is a gate that runs never. Teams of one maintaining a small README will spend more time on the manifest than the docs are worth; the payoff begins when several tools, flags, and exit codes share one document that many people edit.
If you adopt only one piece, adopt the checker first and run it against your existing docs to see which numbers currently have no source at all.
Top comments (0)