No Source, No Number: A Provenance Gate for AI-Drafted API Docs

A generated Quick Start page told integrators to wait 30 seconds before retrying a 429. The edge gateway used a five-second token bucket. Integrators followed the page, not the gateway. They collected a longer 429 streak, opened a Sev-2, and pasted the docs URL into the ticket.

The API had not changed that week. The docs had. A model restated the OpenAPI paths correctly, then filled the gaps with plausible intervals. Plausible is not an SLA.

That pattern is a docs bug class, not a model-quality complaint. Schema restatement is cheap to generate. Numeric and temporal claims are not restatements. They are promises.

The sentence that should have failed CI

Here is a compressed version of the bad paragraph. It looks finished. It even uses the right status code.

If the API returns `429 Too Many Requests`, wait 30 seconds and retry
the same idempotency key. The gateway guarantees a fresh token after
that interval.

Three claims sit in those two sentences:

1. The wait interval is 30 seconds.
2. The same idempotency key is the recovery method.
3. The gateway guarantees a fresh token on that clock.

OpenAPI can confirm that 429 exists. It rarely confirms the bucket size. It almost never confirms a guarantee. A draft that invents the number will still read like a reference page.

The fix is not "write better prompts." The fix is a source map. Every numeric or temporal sentence must point at a file a human already owns. If the pointer is missing, CI cuts the sentence.

Two classes of sentences

Split the page before anyone asks a model to write it.

Restatable. These sentences restate structure that already exists in machine-readable form. A model may draft them. A checker still needs a pointer, but the pointer may be the spec.

• Path, method, and operationId
• Required vs optional fields
• Enum members listed in the schema
• Status codes declared on the operation
• Example JSON that validates against the schema

Must-cite. These sentences assert behavior in time, quantity, or commitment. A model may phrase them only after a human-owned claims file already contains the value. If the value is absent, the sentence does not ship.

• Timeouts, retries, backoff, and jitter
• Rate limits, quotas, burst sizes, and token buckets
• p95 / p99 language, SLAs, and "usually" intervals
• Deprecation and sunset dates
• Idempotency semantics that are not in the spec
• Any use of guarantee, always, never, or we promise

The test is mechanical. If a sentence still makes sense after you delete every digit and every time unit, it might be restatable. If the sentence collapses, it was a promise.

A source-map convention

Keep the map in the Markdown, next to the sentence, not in a side spreadsheet. HTML comments survive most static-site pipelines and are easy to grep.

<!-- src: openapi.yaml#/paths/~1widgets/get/responses/429 -->
A `429 Too Many Requests` response means the client exceeded the
declared rate limit for this operation.

<!-- src: ops-claims.yaml#retry_after_seconds -->
Retry after `5` seconds. Send the original `Idempotency-Key`.

<!-- human-owned:start -->
Support will not extend a burst window during an incident.
<!-- human-owned:end -->

Rules for the convention:

• One src comment owns the following paragraph, not the whole file.
• Allowed files are an allowlist: openapi.yaml and ops-claims.yaml in this workflow.
• Keys after # must exist in the claims file when the source is ops-claims.yaml.
• human-owned blocks are opaque to the model. Regeneration must not rewrite them.
• Bare URLs, Slack messages, and "as discussed" are not sources.

This is a proposed convention, not a standard. Label it that way in the repo so later editors do not treat the comments as prose.

The human-owned claims file

Put numbers where on-call already looks. Do not hide them inside a prompt.

# ops-claims.yaml
# Human-owned. Models may read. Models may not edit.
claims:
  retry_after_seconds: 5
  rate_limit_per_minute: 120
  burst_size: 20
  request_timeout_ms: 2500
  idempotency_key_ttl_seconds: 86400
  deprecation:
    widgets_v1_sunset: "2026-12-01"
forbidden_in_draft:
  - guarantee
  - guaranteed
  - we promise
  - SLA
  - five nines

The file is small on purpose. If a number cannot fit here, it does not belong in a getting-started page. Product, SRE, and docs should change this file in a normal pull request. The model is a reader of the merge, not an author of the values.

Decision table

Sentence type	Model may draft wording?	Required src	Human must own the value
Field description copied from schema	Yes	openapi.yaml	No
Example body that validates	Yes	openapi.yaml	Review shape
HTTP status listed on the operation	Yes	openapi.yaml	No
Retry interval, timeout, burst, quota	Wording only	ops-claims.yaml#key	Yes
Sunset or deprecation date	Wording only	ops-claims.yaml#key	Yes
Idempotency TTL	Wording only	ops-claims.yaml#key	Yes
"We guarantee…" / SLA / five nines	No	n/a	Yes, inside human-owned
Unsourced digit or time unit	No	fail closed	Yes

If a cell says "fail closed," the checker exits non-zero. There is no warning mode for numbers. Warnings become documentation debt.

The checker

The script below is a proposed CI gate. It is not a benchmark and it has not been tuned against a private corpus. Run it on the Markdown you are about to publish.

#!/usr/bin/env python3
"""Fail when drafted API docs assert numbers without a source pointer."""

from __future__ import annotations

import pathlib
import re
import sys

import yaml

ROOT = pathlib.Path(__file__).resolve().parents[1]
DOCS = ROOT / "docs"
CLAIMS_PATH = ROOT / "ops-claims.yaml"
ALLOWED_FILES = {"openapi.yaml", "openapi.yml", "ops-claims.yaml"}

SRC_RE = re.compile(r"<!--\s*src:\s*([^\s#]+)(?:#([^\s]+))?\s*-->")
OWN_START = "<!-- human-owned:start -->"
OWN_END = "<!-- human-owned:end -->"
NUMBERISH = re.compile(
    r"(\b\d+(\.\d+)?\s*(ms|s|sec|secs|seconds|m|min|minutes|h|hours|days)\b)"
    r"|(\b(p95|p99|sla)\b)"
    r"|(\b(retry|timeout|rate[- ]?limit|quota|burst|idempoten)\w*\b)",
    re.I,
)
GUARANTEE = re.compile(r"\b(guarantee[ds]?|we promise|five nines)\b", re.I)


def load_claims() -> dict:
    data = yaml.safe_load(CLAIMS_PATH.read_text()) or {}
    return data.get("claims") or {}


def claim_exists(claims: dict, pointer: str) -> bool:
    cur: object = claims
    for part in pointer.split("."):
        if not isinstance(cur, dict) or part not in cur:
            return False
        cur = cur[part]
    return True


def paragraphs(text: str) -> list[tuple[str, bool]]:
    owned = False
    chunks: list[tuple[str, bool]] = []
    buf: list[str] = []

    def flush() -> None:
        if buf:
            chunks.append(("\n".join(buf).strip(), owned))
            buf.clear()

    for line in text.splitlines():
        if line.strip() == OWN_START:
            flush()
            owned = True
            continue
        if line.strip() == OWN_END:
            flush()
            owned = False
            continue
        if line.strip() == "":
            flush()
            continue
        buf.append(line)
    flush()
    return chunks


def check_file(path: pathlib.Path, claims: dict) -> list[str]:
    errors: list[str] = []
    pending_src: tuple[str, str | None] | None = None
    text = path.read_text()

    # Walk line-level only to bind src comments to the next paragraph.
    owned = False
    para_lines: list[str] = []
    start_line = 1

    def bind_and_reset(end_line: int) -> None:
        nonlocal pending_src, para_lines, start_line
        body = "\n".join(para_lines).strip()
        para_lines.clear()
        if not body:
            pending_src = None
            return
        if owned:
            pending_src = None
            start_line = end_line
            return
        if GUARANTEE.search(body):
            errors.append(
                f"{path}:{start_line}: guarantee language outside human-owned block"
            )
        if NUMBERISH.search(body):
            if pending_src is None:
                errors.append(
                    f"{path}:{start_line}: numeric/temporal claim has no src comment"
                )
            else:
                file_name, pointer = pending_src
                if file_name not in ALLOWED_FILES:
                    errors.append(
                        f"{path}:{start_line}: src file {file_name} is not allowlisted"
                    )
                elif file_name.startswith("ops-claims") and (
                    not pointer or not claim_exists(claims, pointer)
                ):
                    errors.append(
                        f"{path}:{start_line}: ops-claims pointer {pointer!r} missing"
                    )
        pending_src = None
        start_line = end_line

    for i, line in enumerate(text.splitlines(), start=1):
        if line.strip() == OWN_START:
            bind_and_reset(i)
            owned = True
            continue
        if line.strip() == OWN_END:
            bind_and_reset(i)
            owned = False
            continue
        src = SRC_RE.fullmatch(line.strip())
        if src:
            bind_and_reset(i)
            pending_src = (src.group(1), src.group(2))
            start_line = i
            continue
        if line.strip() == "":
            bind_and_reset(i)
            continue
        if not para_lines:
            start_line = i
        para_lines.append(line)
    bind_and_reset(len(text.splitlines()) + 1)
    return errors


def main() -> int:
    if not CLAIMS_PATH.exists():
        print(f"missing {CLAIMS_PATH}", file=sys.stderr)
        return 2
    claims = load_claims()
    errors: list[str] = []
    for path in sorted(DOCS.rglob("*.md")):
        errors.extend(check_file(path, claims))
    for err in errors:
        print(err, file=sys.stderr)
    return 1 if errors else 0


if __name__ == "__main__":
    raise SystemExit(main())

Install the one runtime dependency the script needs, then run it the same way CI will.

pip install pyyaml
python tools/provenance_check.py
echo $?

A non-zero exit is the product. Do not wrap it in a bot comment that reviewers can ignore. Docs PRs that only regenerate Markdown should still fail this gate.

Minimal CI hook

# .github/workflows/docs-provenance.yml
name: docs-provenance
on:
  pull_request:
    paths:
      - "docs/**"
      - "ops-claims.yaml"
      - "openapi.yaml"
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install pyyaml
      - run: python tools/provenance_check.py

Path filters matter. If a claims value changes and the Markdown still cites the old number, that is a different bug. Add a second check later that the cited key's value actually appears in the paragraph. The script above does not do that yet. Call that gap out in the PR template so nobody assumes it does.

What the model is allowed to draft

Feed the model three inputs only:

1. The OpenAPI file, or a single operation slice of it.
2. ops-claims.yaml, read-only.
3. A page skeleton with empty restatable sections and already-filled human-owned blocks.

A proposed prompt, not a magic spell:

Draft the restatable sections of this API page.
Copy field names, types, and status codes from OpenAPI.
For any retry, timeout, rate limit, date, or idempotency TTL,
insert a paragraph only if ops-claims.yaml has the key.
Prefix that paragraph with <!-- src: ops-claims.yaml#key -->.
Do not invent digits. Do not write guarantee language.
Do not edit text between human-owned:start and human-owned:end.

The output is still a draft. A human accepts or deletes each must-cite paragraph. Regeneration may overwrite restatable sections. It must not overwrite human-owned blocks or ops-claims.yaml.

If you need a machine to run that draft pass, keep the machine off the claims file. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can host the draft pass against those three inputs. The provenance checker does not depend on that host. Remove the draft step and the gate still works.

A worked fail, then a pass

Failing page, docs/widgets.md:

# Widgets

Create a widget, then poll for completion.

On `429`, wait 30 seconds and retry. The platform guarantees a fresh
token after that wait.

Expected checker output:

docs/widgets.md:5: numeric/temporal claim has no src comment
docs/widgets.md:5: guarantee language outside human-owned block

Passing page:

# Widgets

<!-- src: openapi.yaml#/paths/~1widgets/post -->
`POST /widgets` accepts a JSON body and returns `201` or `429`.

<!-- src: ops-claims.yaml#retry_after_seconds -->
On `429`, wait `5` seconds and resend the same `Idempotency-Key`.

<!-- src: ops-claims.yaml#idempotency_key_ttl_seconds -->
Keys remain valid for `86400` seconds.

<!-- human-owned:start -->
Incident response will not raise burst size for a single tenant.
<!-- human-owned:end -->

The digits in the passing page are copies. That is the point. The model did not choose 5 or 86400. The claims file did.

Limitations

The gate is narrow. Treat these as known holes, not future slogans.

• OpenAPI pointers are not resolved. A comment may cite a missing path and still pass if the source file name is allowlisted.
• The checker does not confirm that the paragraph's digit equals the YAML value. A stale 30 can sit next to a retry_after_seconds: 5 pointer.
• Time units in words without digits ("shortly", "eventually", "a moment") slip through NUMBERISH.
• Tables, fenced code, and HTML <table> cells are not parsed as first-class claims.
• Multi-paragraph lists after one src comment will over-bind. Keep claims in short paragraphs.
• The script does not detect screenshots, mermaid timings, or Postman collections.
• Forbidden words inside human-owned blocks are not scanned. That is intentional. It is also an abuse hatch.

Add value equality next if your corpus is small. Add OpenAPI pointer resolution if your spec is the only schema. Do not add a sentiment classifier. The bug is unsourced numbers, not tone.

Who should not use this

Skip the workflow if any of the following is true.

• The API has no OpenAPI file and no other machine-readable contract. There is nothing restatable to draft.
• Rate limits are tenant-specific and must not appear in public Markdown at all. Put them in an authenticated console, not a source map.
• Legal or compliance text is generated. This gate is not a contract review.
• The team cannot protect ops-claims.yaml from model writes. If the model can edit the claims, the map is theater.
• Docs are a single giant HTML export with no paragraph structure. The comment convention will not survive.

For those cases, freeze the page. A frozen wrong number is still wrong, but it is attributable.

Ship the gate first

Start with one page that already caused a ticket. Add ops-claims.yaml with only the numbers on that page. Wrap the promises in human-owned blocks. Run the checker on a branch that still contains the invented interval. Confirm it fails. Then replace the interval with a cited copy of the claims file. Confirm it passes.

The model is optional after that. The source map is not. A restated schema can be regenerated. A number without a source should not leave the branch.