Stop Generated Reference Pages From Publishing Unsourced Rate Limits

Generated API docs fail in production when a model inserts a rate limit, timeout, or retry rule that the OpenAPI file never recorded. A longer prompt does not solve that class of error, because empty descriptions still invite fluent policy language. The working control is a build gate that extracts policy-shaped claims, requires a cited source path for each claim, and leaves unsourced policy text unpublished. Human writers then own every remaining commitment against a real product decision, not against a plausible completion.

Why empty fields become invented policy

When a reference generator sees a blank description, it completes the sentence with operations language that sounds finished. Brownfield OpenAPI files often omit latency, quota, and retention notes, so the model supplies round numbers that resemble a real platform. Reviewers miss those inventions because the nearby parameter table still matches schema types, required flags, and example payloads. The mismatch appears later, when support quotes the page and engineering cannot defend the number in an incident.

Policy claims are a distinct failure mode from wrong types or stale examples. A mistyped integer is easy to catch against the schema, while a sentence about retries is not a schema object. Generated troubleshooting text is especially risky, because it mixes status-code maps that the spec does own with recovery advice that the spec never stated. Treat those two layers as different publication rights, or the model will blend them in one paragraph.

What the model may draft, and what a human must own

Use a decision table before any model run, and keep it next to the OpenAPI file rather than inside a chat prompt. The table below is a proposal for public reference pages, not a record of a production audit.

Claim class	Source that authorizes a model draft	Human-owned remainder
Parameter name, type, required	paths.*.parameters or component schemas	Why the field exists in the product
Enum members and default literals	enum, default, and locked fixtures	Promise that the default will not change
HTTP status keys and response schemas	responses entries with $ref targets	Retry, paging, and on-call recovery steps
Example request bodies	Schema-valid fixtures hashed in CI	Narrative walkthroughs that imply a workflow
Rate limits, quotas, burst behavior	Versioned config or a published SLA file	Any “we guarantee” or “never throttle” wording
Timeouts, idempotency, ordering	Explicit vendor extensions or tests	Durability, exactly-once, and support commitments
Retention, deletion, and PII handling	Legal or security source owned by humans	All customer-facing privacy promises

The rule is mechanical: if the source file does not contain the number or the modal verb, the model may not write that clause. Restating a field type is recoverable from the spec after a bad draft. Inventing a 429 window is not recoverable, because readers treat it as a contract.

Artifact: a banned-inference catalog and a claim scanner

Keep a YAML catalog of policy topics that a model must not infer. The catalog is data, not prompt flavor, and CI should fail closed when a generated page mentions a topic without a source map.

# docs/policy_catalog.yaml
version: 1
banned_inferences:
  - id: rate_limit
    patterns:
      - "rate limit"
      - "requests per"
      - "throttl"
      - "429"
    required_source_kinds: ["config", "sla"]
  - id: timeout
    patterns:
      - "timeout"
      - "deadline"
      - "within \\d+ (ms|s|seconds|minutes)"
    required_source_kinds: ["config", "test"]
  - id: retry
    patterns:
      - "retr"
      - "backoff"
      - "idempoten"
    required_source_kinds: ["test", "runbook"]
  - id: retention
    patterns:
      - "retain"
      - "retention"
      - "delete after"
      - "PII"
    required_source_kinds: ["legal", "security"]
model_may_draft:
  - schema_restatement
  - status_key_table
  - fixture_example
human_must_own:
  - unsourced_policy_sentence
  - support_escalation
  - compatibility_promise

Pair that catalog with a source map that only humans or a checked generator may write. Each published policy sentence needs a source_path, a kind, and a short excerpt that actually appears in the source.

# docs/generated/payments_errors.source.yaml
claims:
  - topic: rate_limit
    markdown_anchor: "rate-limits"
    source_path: config/rate_limits.yaml
    kind: config
    excerpt: "POST /v1/payments: 120 requests per minute"
  - topic: timeout
    markdown_anchor: "request-timeouts"
    source_path: tests/http_timeout_test.go
    kind: test
    excerpt: "client timeout is 15s for POST /v1/payments"

If the generated markdown mentions rate limits and that map omits rate_limit, the build fails. If the excerpt is not a substring of the source file, the build also fails. The model may propose a map, but the gate treats an unverified excerpt as missing evidence.

Numbered workflow

1. Freeze the OpenAPI document and any config files that encode quotas or timeouts, then record their content hashes in the docs job. Generation without those hashes cannot prove that a later number still matches the product.
2. Label each target markdown section with a draft class: schema_restatement, status_key_table, fixture_example, or human_only. A human_only section must stay empty during the model pass.
3. Run the model only over sections that the table marks as restatements or tables, and require the output to include the source map stub shown above. Discard any draft that fills human_only headings.
4. Extract policy-shaped claims from the generated markdown with the catalog patterns, including numbers that sit beside time units or “per minute” language. Do not trust the model to list its own claims.
5. Join extracted claims to the source map, then verify each excerpt against the frozen source file. Fail on missing topics, wrong kinds, or excerpts that do not match.
6. Leave a blank human stub for every unmatched claim, and block publish until a person writes the policy sentence or deletes the topic from the page.

The order matters because a model that writes first and cites later will invent the citation. Extraction after generation is the audit. Mapping before generation is only a hint.

Scanner code the docs job can run

The following script is a proposal you can save as tools/scan_doc_policy.py and run in CI. It reads generated markdown, the catalog, and the source map, then exits nonzero on unsourced policy language.

#!/usr/bin/env python3
"""Fail when generated docs state policy the source map cannot prove."""
from __future__ import annotations

import re
import sys
from pathlib import Path

import yaml

ANCHOR_RE = re.compile(r"^#{2,3}\\s+(.+)$", re.M)


def load_yaml(path: Path):
    return yaml.safe_load(path.read_text(encoding="utf-8"))


def excerpt_in_source(source_path: Path, excerpt: str) -> bool:
    if not source_path.is_file():
        return False
    text = source_path.read_text(encoding="utf-8")
    return excerpt.strip() in text


def find_claims(markdown: str, catalog: dict) -> list[dict]:
    found = []
    for topic in catalog["banned_inferences"]:
        for raw in topic["patterns"]:
            pattern = re.compile(raw, re.I)
            for match in pattern.finditer(markdown):
                found.append(
                    {
                        "id": topic["id"],
                        "span": match.group(0),
                        "kinds": topic["required_source_kinds"],
                    }
                )
    return found


def main(md_path: str, catalog_path: str, map_path: str) -> int:
    markdown = Path(md_path).read_text(encoding="utf-8")
    catalog = load_yaml(Path(catalog_path))
    source_map = load_yaml(Path(map_path)) or {"claims": []}
    mapped = {row["topic"]: row for row in source_map.get("claims", [])}
    errors = []
    for claim in find_claims(markdown, catalog):
        row = mapped.get(claim["id"])
        if row is None:
            errors.append(f"unsourced {claim['id']}: {claim['span']!r}")
            continue
        if row["kind"] not in claim["kinds"]:
            errors.append(
                f"kind {row['kind']!r} cannot cover {claim['id']}"
            )
            continue
        if not excerpt_in_source(Path(row["source_path"]), row["excerpt"]):
            errors.append(
                f"excerpt not found in {row['source_path']} for {claim['id']}"
            )
    for line in errors:
        print(line, file=sys.stderr)
    return 1 if errors else 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1], sys.argv[2], sys.argv[3]))

A docs job can call it after generation and before any publish step.

python tools/scan_doc_policy.py \
  docs/generated/payments_errors.md \
  docs/policy_catalog.yaml \
  docs/generated/payments_errors.source.yaml

A failing run should look like the following, which is a fixture for the scanner rather than a live production log.

unsourced rate_limit: '120 requests per minute'
kind 'runbook' cannot cover timeout
excerpt not found in tests/http_timeout_test.go for timeout

Those three lines map to three different ownership failures. The first is a model invention. The second is a source of the wrong kind, such as an internal note used as a public SLA. The third is a fabricated citation, which is worse than a missing citation because it looks reviewed.

A small OpenAPI slice that should stay numeric-free in prose

Consider a payments error fragment that lists status codes and schemas only. The model may restate those keys into a table. It may not add retry or quota sentences unless config or tests already contain them.

paths:
  /v1/payments:
    post:
      responses:
        "201":
          description: ""
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Payment"
        "429":
          description: ""
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Error"

A model that sees empty description fields will often write that clients should wait sixty seconds, or that the route allows one hundred requests per minute. Neither figure exists in the fragment, so both sentences are policy inventions. The scanner must fail that page even if the 201 and 429 keys are correctly listed. Correct status keys do not license invented recovery advice.

Where a hosted generator fits without owning the gate

Some teams want the restatement pass and the scanner to run on a shared box so local laptops are not the only publishers. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option, which can host the generator plus scan_doc_policy.py for a small public API. The server is useful only as a place to freeze sources, run the model on labeled sections, and block publish when the catalog finds an unsourced claim. Free model access does not decide which sentences are contracts, and it should not be asked to grade its own citations.

Limitations

Pattern catalogs miss paraphrases, including “briefly pause before sending again” as a retry rule with no banned token. They also over-match, so a historical note about HTTP 429 in another product can fail a page that never states your quota. Excerpt checks are substring tests, not semantic proofs, and a source file can contain a number in a comment that is not an external promise. Multilingual docs need per-language pattern lists, or the English catalog will ignore policy written in another locale. The workflow also assumes OpenAPI, config, and tests are the system of record; wiki sentences cannot authorize a model draft under this gate.

This approach does not measure factual accuracy beyond the excerpt test. A config file can be wrong, and the scanner will still pass a matching sentence. Humans still own the decision to publish a quota that engineering might change next week. The gate only stops the model from being the author of that decision.

Who should not use this approach

Do not use this scanner as a substitute for legal review on privacy or retention pages, because a regex cannot certify a compliance claim. Do not apply it to conceptual guides whose purpose is narrative, where banned words like “retry” appear as teaching examples rather than product promises. Do not run it against internal runbooks if those runbooks are the human-owned source for public docs, or you will block the very text that should fill the source map. Teams without a frozen OpenAPI file and hashed config will spend the job fighting missing paths instead of invented policy.

Skip the model fill entirely when the page is a compatibility promise, an SLA, or an on-call procedure. Those sections start blank, stay blank through generation, and are written by the people who can be paged when the sentence is wrong. The scanner then becomes a fence around empty headings rather than a grader of model prose.

Operators who already keep OpenAPI and quota config in CI can run the catalog and scanner on a free server before they widen model-written reference sections. The useful outcome is a page that restates the spec, leaves policy blank, and fails the build when a draft invents a limit nobody stored.