The Model May Draft the Prose. The Repo Must Confirm the Facts.

A doc said the retry limit defaults to 5. The code said 3. The env var in the doc had been renamed two releases earlier, so anyone who copy-pasted the snippet got a silent no-op instead of an error.

Nothing about that draft looked wrong. The sentences were clean, the structure was better than what we had, and the reviewer approved it in four minutes because reading prose is easy and checking facts is work.

That is the actual failure mode with AI-drafted documentation. Not hallucinated paragraphs — unverified specifics wearing good grammar.

The last week of DEV's front page leaned hard into AI fatigue and vibe-coding regret. Fair. But the useful response isn't refusing the tool; it's narrowing what the tool is allowed to produce and making the rest mechanically checkable.

Split the doc before you generate it

Before any prompt runs, decide two things per section: is this prose the model may draft, and are the facts in it something a machine can confirm against the repo?

Sections where the answer is "no" on the second question are human-owned, always.

Section	Model drafts?	Evidence source	Why
Overview / what it does	Yes	Repo README + module docstrings	Low risk, easy to correct
Local setup steps	Yes	scripts/, Makefile, CI config	Commands can be executed
Config reference	Yes	Code that reads the env vars	Names and defaults are greppable
API surface summary	Yes, with limits	Route definitions	Paths and methods are greppable
Security model	No	Human security owner	Wrong text is a liability
Billing, quotas, limits	No	Product owner, dated source	Numbers change without code
Deprecation / sunset dates	No	Release manager	A guess here breaks customers
SLA and support tiers	No	Contracts	Not derivable from code

The rule that survives contact with reality: if the fact lives outside the repository, the model does not get to state it. No amount of retrieval fixes a source of truth you don't control.

Artifact: a claim checker that fails the build

Here is the piece I keep coming back to. A scope file declares which headings may be drafted, and a small standard-library script extracts every hard claim — env vars, CLI flags, repo paths, version strings — and requires each one to appear somewhere in the repo.

The code below is a template, and the output further down is illustrative. It runs on the Python standard library plus git grep; nothing else.

docs/ai-scope.json:

{
  "draftable": ["Overview", "Local setup", "Configuration reference", "API surface"],
  "human_owned": ["Security model", "Billing and quotas", "Deprecation", "SLA"],
  "fail_on_missing": ["env_var", "flag", "path"],
  "warn_on_missing": ["version"]
}

tools/claim_check.py:

#!/usr/bin/env python3
"""Fail a model-drafted doc that asserts repo facts the repo cannot confirm."""
import json, re, subprocess, sys
from pathlib import Path

REPO = Path(__file__).resolve().parents[1]
SCOPE = json.loads((REPO / "docs/ai-scope.json").read_text())

PATTERNS = {
    "env_var": re.compile(r"`([A-Z][A-Z0-9]*(?:_[A-Z0-9]+)+)`"),
    "flag":    re.compile(r"`(--[a-z][a-z0-9-]+)`"),
    "path":    re.compile(r"`((?:src|docs|scripts|config)/[\w./-]+)`"),
    "version": re.compile(r"\bv?(\d+\.\d+\.\d+)\b"),
}

def grep(needle: str):
    out = subprocess.run(["git", "grep", "-n", "-F", "--", needle],
                         cwd=REPO, capture_output=True, text=True)
    return [l for l in out.stdout.splitlines() if l.strip()]

def sections(text: str):
    """Yield (heading, body) for each H2."""
    current, buf = None, []
    for line in text.splitlines():
        if line.startswith("## "):
            if current is not None:
                yield current, "\n".join(buf)
            current, buf = line[3:].strip(), []
        elif current is not None:
            buf.append(line)
    if current is not None:
        yield current, "\n".join(buf)

def main(path: str) -> int:
    draft = Path(path).read_text()
    failures, warnings = [], []

    for heading, body in sections(draft):
        if any(h.lower() in heading.lower() for h in SCOPE["human_owned"]):
            failures.append(f"human-owned section present: {heading!r}")
            continue
        if not any(h.lower() in heading.lower() for h in SCOPE["draftable"]):
            failures.append(f"heading outside the draftable scope: {heading!r}")
            continue
        for kind, rx in PATTERNS.items():
            for claim in sorted(set(rx.findall(body))):
                hits = grep(claim)
                if hits:
                    print(f"OK   {kind:8} {claim:32} {hits[0].split(':')[0]}")
                elif kind in SCOPE["fail_on_missing"]:
                    failures.append(f"unverified {kind}: {claim} (section {heading!r})")
                else:
                    warnings.append(f"unverified {kind}: {claim} (section {heading!r})")

    for w in warnings:
        print(f"WARN {w}")
    for f in failures:
        print(f"FAIL {f}")
    return 1 if failures else 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1]))

Run it on the draft before a human ever reads it:

python3 tools/claim_check.py docs/drafts/configuration.md; echo "exit=$?"

Illustrative output for the retry-limit draft that started this post:

OK   env_var  LOG_LEVEL                        src/config.py
OVER env_var  RETRY_MAX_ATTEMPTS               src/retry.py
OK   flag     --dry-run                        src/cli.py
OK   path     scripts/seed.py                  scripts/seed.py
WARN version  2.4.1                            (no match)
FAIL unverified env_var: RETRY_MAX_ATTEMPTS (section 'Configuration reference')
FAIL human-owned section present: 'Billing and quotas'
exit=1

OVER is an overlay your checker will need anyway: the string exists, but not in the section the draft claims. Catching the rename is the whole point — the doc and the code disagree, and the checker refuses to let that pass.

Why a grep and not a model

Because the verdict has to be boring. A second model grading the first model's facts gives you two opinions and no evidence. git grep gives you a file, a line, and an exit code — something a reviewer can check in five seconds and a pipeline can gate on.

Use the model for the part it is genuinely good at: turning the diff, the config read, and the module docstrings into readable prose. Use the repo for the part prose cannot be trusted with.

Wiring it into a real loop

1. Generate into docs/drafts/ only. Never generate directly into the published tree.
2. Run claim_check.py; fix the draft or fix the docs, never the rule.
3. Human review starts at the FAIL/WARN list, so attention lands on facts instead of adjectives.
4. Promote to docs/ on merge; keep the draft path in the PR body for traceability.

Step 2 is where a scheduled job earns its keep. A nightly run over every draft plus every recently changed config module catches drift before a reader does — and this is where free model access and a free server option become practical rather than theoretical: the generation step is a batch job you don't want to pay per-seat for, and the verification step is cheap enough to run on a small always-on box.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I'm describing the workflow I'd use; the project advertises free model access and a free server option, and those terms are set by its operators, so check the current ones before you depend on them.

Limitations, stated plainly

• Presence is not truth. A grep hit proves the string exists, not that the sentence about it is correct. Defaults, ordering, and conditional behavior still need a human.
• Monorepos produce noise. Vendored code and generated files will "confirm" claims that have nothing to do with your service. Keep an ignore list and review it monthly.
• External claims are out of scope. If the doc describes a third-party API, no amount of repo grepping helps. Those sections belong in the human-owned list.
• Scope files rot too. A new heading will fail the run, which is correct — but it means someone must update ai-scope.json deliberately.
• Free tiers are unstable by nature. Batch jobs should degrade to "draft only, no generation" rather than blocking a release when a quota or server is unavailable.

Who should not use this

Skip it if your docs are mostly prose about why rather than how — architecture rationale, decision records, onboarding narrative. There are few hard claims to check and the overhead won't pay back.

Also skip it if your source of truth is a wiki outside the repo, or if nobody can own the scope file. An unmaintained gate becomes a skipped gate, and a skipped gate is worse than none because it looks like assurance.

If you do have a config surface, a CLI, and a repo you can grep, the split is worth the afternoon: let the model draft the setup steps and the config table, let claim_check.py argue with it, and keep the security model and the pricing page in human hands where they belong.