Generated Docs Should Compile From a Fact Index

Generated documentation fails when a model invents endpoints, status codes, or time bounds that no test can refute. Human review still misses those inventions because a fluent paragraph looks finished long before its identifiers are checked. The durable fix is to compile a fact index from source artifacts, then forbid any drafted claim that cannot cite that index. Models may still write connective prose, but tests own every path, code, version, and numeric bound.

This article treats documentation as a compile target rather than a prompt result. Derived pages may be drafted from indexed facts. Interpretive pages stay human-owned, and hybrid pages allow prose only around locked identifiers. The workflow below is labeled as a reproducible method, not as a report of production traffic or vendor benchmarks.

The failure mode review does not catch

Most doc reviews score tone, structure, and completeness, then assume identifiers were copied from the product. A model can still emit a plausible /v2/export path, a 204 success code, or a 99.95% availability sentence that never existed in source. Those tokens survive because they look like documentation rather than unverified input. A reviewer who is also shipping the feature will often confirm the story and skip the literals.

Cheap generation makes that miss more expensive over time. Regenerating a guide can rewrite yesterday's correct path into today's fluent error, and git blame will point at a bot commit instead of a decision. The useful control is not a stricter prompt. It is a machine-readable fact index that both the drafter and the linter must share.

What a model may draft versus what tests must own

Split every documentation claim into two classes before any draft job starts. Indexed facts are identifiers and quantities that already exist in OpenAPI files, CLI parsers, feature flags, or release tags. Judgment prose is causal language, audience framing, migration advice, and any sentence that cannot be extracted from those artifacts. Models may draft judgment prose only where the file policy allows it, and only after the fact index is passed in as the sole numeric vocabulary.

Tests own the indexed class completely. If a drafted sentence contains a path, method, status code, version, timeout, quota, or rate limit, that token must appear in the index or the build fails. Humans own the judgment class completely. Security promises, support commitments, pricing, and breaking-change intent never enter the generator, even when the surrounding guide is otherwise eligible for drafting.

Artifact: facts.json plus a claim linter

Keep the index small and boring. The following schema is enough to catch the usual invented tokens in HTTP product docs, and it can be extended later without changing the linter contract.

{
  "paths": ["/v1/widgets", "/v1/widgets/{id}"],
  "methods": ["GET", "POST", "DELETE"],
  "status_codes": [200, 201, 404, 409, 429],
  "versions": ["1.4.2"],
  "timeouts_ms": [2500],
  "rate_limits": ["100/min"],
  "forbidden_in_drafts": ["SLA", "uptime", "guarantee", "we promise"]
}

Pair it with a file policy so the generator never receives human-owned paths. Derived files may be rewritten from the index. Hybrid files may receive prose around fenced fact blocks. Human files are excluded from generation and still pass through the linter if they contradict the index.

# docs-policy.yml
rules:
  - glob: "docs/reference/**/*.md"
    mode: derived
  - glob: "docs/guides/**/*.md"
    mode: hybrid
  - glob: "docs/policy/**/*.md"
    mode: human

The linter below is a complete, local check. It extracts HTTP-looking paths, status codes, vX.Y.Z versions, millisecond timeouts, and slash-based rate limits, then compares them with facts.json. Forbidden phrases fail only in non-human files, which keeps policy pages free to state commitments that models must not invent.

#!/usr/bin/env python3
"""claim_lint.py — fail generated docs that invent identifiers."""
from __future__ import annotations

import json, re, sys
from pathlib import Path

PATH_RE = re.compile(r"(?<![A-Za-z])(/v\d+/[A-Za-z0-9_{}/-]+)")
STATUS_RE = re.compile(r"\b([1-5]\d{2})\b")
VERSION_RE = re.compile(r"\b(\d+\.\d+\.\d+)\b")
TIMEOUT_RE = re.compile(r"\b(\d+)\s*ms\b", re.I)
RATE_RE = re.compile(r"\b(\d+\s*/\s*(?:min|sec|hour))\b", re.I)

HTTPISH = {200, 201, 202, 204, 301, 302, 304, 400, 401, 403, 404, 409, 412, 415, 429, 500, 502, 503}

def load_facts(path: Path) -> dict:
    data = json.loads(path.read_text())
    data["status_codes"] = set(data.get("status_codes", []))
    data["timeouts_ms"] = set(data.get("timeouts_ms", []))
    return data

def lint_text(text: str, facts: dict, mode: str) -> list[str]:
    errors = []
    for token in PATH_RE.findall(text):
        if token not in facts["paths"]:
            errors.append(f"unknown path {token}")
    for token in STATUS_RE.findall(text):
        code = int(token)
        if code in HTTPISH and code not in facts["status_codes"]:
            errors.append(f"unknown status {code}")
    for token in VERSION_RE.findall(text):
        if token not in facts["versions"]:
            errors.append(f"unknown version {token}")
    for token in TIMEOUT_RE.findall(text):
        if int(token) not in facts["timeouts_ms"]:
            errors.append(f"unknown timeout {token}ms")
    for token in RATE_RE.findall(text):
        normalized = token.replace(" ", "")
        allowed = {r.replace(" ", "") for r in facts["rate_limits"]}
        if normalized not in allowed:
            errors.append(f"unknown rate limit {token}")
    if mode != "human":
        lowered = text.lower()
        for phrase in facts.get("forbidden_in_drafts", []):
            if phrase.lower() in lowered:
                errors.append(f"forbidden draft phrase {phrase!r}")
    return errors

Numbered workflow

1. Extract the index from source, never from last week's docs. Point a small extractor at OpenAPI, Cobra/Click flag tables, or generated JSON Schema. If the product has no schema yet, maintain facts.json by hand beside the release tag and fail CI when the tag moves without an index diff. Do not seed the index from existing Markdown, because that would freeze prior inventions as truth.

2. Classify each docs glob before the draft job is scheduled. Derived reference pages can be rebuilt whenever the index changes. Hybrid guides may receive a model draft only for paragraphs outside fenced fact blocks. Human policy pages are not sent to any model, and the generator must list them as skipped in its log.

3. Hand the model the index as vocabulary, not as optional context. The prompt should state that every path, status, version, timeout, and rate token must be copied verbatim from the JSON. It should also state that missing facts stay missing: the model may write "not documented in this index" rather than guess. Label this prompt as a template until a team has run it against its own repository.
   

You draft Markdown for the file named below.
Use facts.json as the only source of paths, methods, status codes,
versions, timeouts, and rate limits. Do not invent identifiers.
Do not write SLA, uptime, pricing, or support commitments.
If a reader question needs a fact that is absent, say it is absent.
File: docs/guides/retry.md
Mode: hybrid

1. Run claim lint on the working tree, including human files. Human pages should not be rewritten by the job, but they can still drift away from shipped behavior. A policy page that still advertises /v1/widgets after a rename should fail for the same reason a generated guide would fail. Keep the linter deterministic so a red build points at a token, not at a style opinion.

2. Snapshot derived pages, and review only hybrid prose. After a green lint, write a snapshot of docs/reference/** and fail when the snapshot changes without an index change. Hybrid files still need a person to accept tone, order, and omissions. The person is not asked to re-audit every status code, because that work already happened in step 4.

3. Record the skipped set next to the pull request. A short generation-report.md listing derived, hybrid, and skipped-human paths makes the ownership split visible in review. If a path is missing from the report, treat the job as incomplete rather than merging silent gaps.

Running the draft job without mixing it into laptops

Local extraction and linting should stay in the repository so reviewers can run them without extra accounts. The draft step is the part that benefits from remote execution, because it is slow, repetitive, and easy to rerun when the index changes. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can host that draft step while facts.json, docs-policy.yml, and claim_lint.py remain the merge gate in CI.

Do not treat remote drafting as authority over identifiers. The server returns Markdown; the linter decides whether the Markdown is allowed to merge. If the remote job is unavailable, derived pages can still be regenerated with a dumb template that prints tables from facts.json, and hybrid pages simply wait. That fallback is a feature, because it proves the index—not the model—is the source of record.

A minimal CI shape looks like the following unexecuted sketch. Adapt the installer and Python version to the repository; the important part is order: extract, draft, lint, snapshot.

# .github/workflows/docs-fact-index.yml
name: docs-fact-index
on: [pull_request]
jobs:
  lint-docs:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python tools/extract_facts.py --in openapi.yaml --out facts.json
      - run: python tools/claim_lint.py --facts facts.json --policy docs-policy.yml
      - run: python tools/snapshot_reference.py --check docs/reference

Decision table for new pages

Page kind	Example	Generator	Owner of identifiers	Owner of wording
Derived reference	Parameter tables, status catalogs	Allowed	Fact index + linter	Template or model
Hybrid guide	Retry tutorial, pagination walkthrough	Allowed with fenced facts	Fact index + linter	Human after draft
Human policy	Security, support, pricing, breaking change	Denied	Human + linter	Human only
Ephemeral ops	Incident banners, status-page copy	Denied	On-call human	On-call human

Use the table when a new file is added, not after a draft already exists. Reclassification after generation creates a cleanup diff that reviewers will rubber-stamp. Classification first keeps the skipped set honest.

Limitations

The linter does not understand meaning. It will accept a sentence that uses a real 404 in the wrong story, and it will reject a newly shipped path until the extractor runs. Regex tokenizers are English- and HTTP-centric, so gRPC names, GraphQL operations, and locale-specific number formats need extra extractors. Stale OpenAPI remains a supply-chain problem: a green docs build can still describe an unpublished spec if the spec file itself is wrong. Snapshot noise will grow if derived pages include timestamps or unsorted maps, so render those pages with stable ordering. None of these limits is a reason to skip the index; they are reasons not to confuse a token check with a full editorial review.

Who should not use this approach

Do not apply this workflow to legal terms, privacy notices, or contractual SLAs, because those pages are not compile outputs. Do not apply it to marketing pages whose claims are intentionally non-extractable, and do not apply it while the product still lacks any machine-readable surface. Teams that cannot name an owner for facts.json will watch the index rot, then blame the linter for blocking legitimate docs. Teams that want a model to invent onboarding narrative without a schema should keep that experiment off the default branch rather than weakening the gate.

Close

The practical conclusion is narrow. Models can draft sentences; they cannot be the source of paths, codes, versions, or bounds. Compile those tokens from product artifacts, lint every Markdown file against the index, and keep human-owned pages out of the generator. If a free model tier is already in the drafting path, run this linter on the output before the documentation pull request opens.