Bound Generated API Examples to Reviewed Fixture Hashes

Generated API docs remain trustworthy when every example payload is copied from a reviewed fixture file and hashed in CI. A model may draft connective prose around those fixtures, but it must not invent status codes, timings, or production guarantees. The human owner still writes deprecation calendars, retry policy, and any sentence that would survive a customer lawsuit. This split keeps regeneration cheap without letting unreviewed numbers or dates leak into the published documentation page.

Why example drift is a documentation defect

API readers paste examples into clients, load tests, and incident tickets, so a wrong payload is an operational defect. Models that reconstruct JSON from parametric memory routinely change field names, enum spellings, and nested nullability. OpenAPI can list parameter names and types, yet it cannot prove that a narrative example still matches the last reviewed fixture. The workflow below treats example JSON as a signed artifact and treats model prose as a disposable wrapper.

Wrong examples spread faster than wrong adjectives because they are executable. A renamed field in a fenced block will fail a customer integration before any reviewer notices tone. A status code that never appears in the fixture will be copied into retry logic and on-call runbooks. Documentation CI should therefore fail on byte drift in examples, not on whether the surrounding English sounds confident.

What the model may draft versus what a human must own

The useful boundary is not a split between entire AI pages and entire human pages. The useful boundary is bytes that already exist in reviewed files versus commitments that do not exist there. Models may copy fixture JSON, restate operation identifiers, and write short setup sentences that only mention fields already present. Humans must own caveats that describe production behavior, legal exposure, or any time-bounded promise.

The table below is the working contract for this article. The generator may fill outputs that trace to reviewed sources and must stop otherwise.

Doc fragment	Canonical source	Model action	Human owner
Request example JSON	fixtures/<op>.request.json	Copy verbatim	Fixture reviewer
Response example JSON	fixtures/<op>.response.json	Copy verbatim	Fixture reviewer
Path, verb, operationId	openapi.yaml	Copy identifiers	Spec author
Parameter names and types	openapi.yaml	Tabulate from schema	Spec author
Security scheme names	components.securitySchemes	Name the schemes only	Spec author
Token lifetime or clock skew	Not in fixtures	Refuse	Security owner
Retry, backoff, idempotency	Not in fixtures	Refuse	SRE owner
Deprecation or sunset date	Not in fixtures	Refuse	Product owner
Production base URL	Human caveat file	Refuse	Platform owner
PII and logging warnings	Human caveat file	Refuse	Privacy owner

This table is a proposal for teams that already keep OpenAPI documents and checked-in fixtures. It is not a measurement of model quality, latency, cost, or long-term availability. If a cell has no reviewed source, the generator must emit a stub heading and refuse to invent a body. Empty confidence is cheaper to repair than a published number nobody will admit writing.

Proposed repository layout

Keep generated pages and owned caveats in different paths so reviewers can read two clean diffs. Fixture files stay pretty-printed with stable key order so hashes move only when bytes move. The ownership map is itself a reviewed artifact, not a prompt the model is allowed to edit.

docs/
  generated/
    create-charge.md
    get-charge.md
  caveats/
    create-charge.md
    get-charge.md
fixtures/
  create-charge.request.json
  create-charge.response.json
  get-charge.request.json
  get-charge.response.json
openapi.yaml
doc-ownership.yaml
fixtures.sha256.json
scripts/
  hash_fixtures.py
  generate_examples.py
  verify_doc_examples.py
  mock_roundtrip.py

# Proposed ownership map. Reviewers edit this file; generators only read it.
operations:
  createCharge:
    openapi_operation_id: createCharge
    fixtures:
      request: fixtures/create-charge.request.json
      response: fixtures/create-charge.response.json
    generated_page: docs/generated/create-charge.md
    human_caveats: docs/caveats/create-charge.md
    allow_model_prose: true
    forbidden_in_model_prose:
      - SLA
      - guaranteed
      - production
      - forever
      - GDPR
      - retry
      - backoff
      - milliseconds

Numbered workflow

Step 1: Freeze fixtures before any prose job starts

Store each request and response body as pretty-printed JSON with two-space indent and sorted keys. Compute a SHA-256 digest for every fixture and commit those digests in a companion manifest file. Reviewers should approve fixture changes in a pull request that contains no generated Markdown. A prose job that cannot load a matching digest must fail closed instead of synthesizing a sample.

# scripts/hash_fixtures.py — proposed local check, not a published benchmark.
from pathlib import Path
import hashlib
import json
import sys


def digest(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def main() -> int:
    root = Path("fixtures")
    manifest = {path.as_posix(): digest(path) for path in sorted(root.glob("*.json"))}
    Path("fixtures.sha256.json").write_text(json.dumps(manifest, indent=2) + "\n")
    print(f"wrote {len(manifest)} fixture digests")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

python3 scripts/hash_fixtures.py
git add fixtures/*.json fixtures.sha256.json doc-ownership.yaml

Treat fixture review as contract review. Ask whether the body is still valid against the current schema, whether identifiers are fake, and whether timestamps are frozen. Do not mix that discussion with whether the generated introduction reads smoothly.

Step 2: Generate only the example enclosure

The generator reads OpenAPI for headings and reads fixtures for fenced JSON blocks. It may request two or three setup sentences that mention field names already present in those fixtures. It must paste fixture bytes unchanged and tag each fence with the digest from the manifest. Human caveat files are appended after a horizontal rule and must never enter the model as editable context.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can run the prose job and the mock verifier when a team does not want a separate generation host. Ownership rules stay in git: fixtures remain the only legal example bytes, and caveat files stay out of the model prompt.

# scripts/generate_examples.py — proposed generator. Labelled unexecuted.
from pathlib import Path
import json
import yaml

OWN = yaml.safe_load(Path("doc-ownership.yaml").read_text())
DIGESTS = json.loads(Path("fixtures.sha256.json").read_text())

TEMPLATE = """# {title}

{setup}

Request example (digest `{req_digest}`):

json
{request}

Response example (digest `{resp_digest}`):

json
{response}

---

{caveats}
"""

def copy_fixture(relpath: str) -> str:
    path = Path(relpath)
    if DIGESTS[path.as_posix()] != __import__("hashlib").sha256(path.read_bytes()).hexdigest():
        raise SystemExit(f"stale digest: {path}")
    return path.read_text()

# Draft setup sentences only from fixture keys. Do not pass caveat files inbound.

python

The prompt boundary matters more than the template string. If caveat text is visible to the model, the next regeneration will rewrite promises in a style that looks consistent and is operationally wrong. Keep the model context limited to operationId, path, method, fixture JSON, and the denylist.

Step 3: Verify every fenced JSON block against the manifest

After generation, a verifier walks the Markdown, extracts fenced json blocks, and canonicalizes them. Each block must hash-match a fixture listed for that operation in doc-ownership.yaml. Extra fences, missing fences, or pretty-print drift should fail the build with the operation identifier in the error. This check is mechanical and does not grade the quality of the surrounding English.

# scripts/verify_doc_examples.py — proposed CI gate.
import hashlib
import json
import re
from pathlib import Path
import yaml

FENCE = re.compile(r"```

json\n(.*?)\n

```", re.S)

def sha(text: str) -> str:
    return hashlib.sha256(text.encode()).hexdigest()

own = yaml.safe_load(Path("doc-ownership.yaml").read_text())
errors = []
for op, row in own["operations"].items():
    page = Path(row["generated_page"]).read_text()
    fences = FENCE.findall(page)
    expected = [
        Path(row["fixtures"]["request"]).read_text(),
        Path(row["fixtures"]["response"]).read_text(),
    ]
    if len(fences) != 2:
        errors.append(f"{op}: expected 2 json fences, found {len(fences)}")
        continue
    for got, want in zip(fences, expected):
        if sha(got) != sha(want):
            errors.append(f"{op}: example fence does not match fixture bytes")

if errors:
    raise SystemExit("\n".join(errors))
print("example fences match reviewed fixtures")

python3 scripts/verify_doc_examples.py

Canonicalization policy should be explicit. Either fixtures and fences are byte-identical, or both sides are parsed as JSON and re-serialized with sorted keys. Mixing those policies produces noisy failures that teams start ignoring. Pick one rule, document it beside the ownership map, and keep it in the verifier.

Step 4: Round-trip fixtures through a local mock

Copying JSON is necessary and still insufficient when the mock server rejects the same body. A small mock can load the OpenAPI document and assert that each request fixture is accepted and each response fixture validates. Failures here mean the fixture review was incomplete, not that the prose generator should rewrite samples. Keep the mock in the same job that verifies hashes so example pages cannot ship against a broken contract.

# scripts/mock_roundtrip.py — proposed check. Requires a local OpenAPI mock you already trust.
from pathlib import Path
import json
import yaml

# Pseudocode: plug in the mock client your team already uses.
# Do not treat this snippet as a benchmark or as a hosted SLA.
own = yaml.safe_load(Path("doc-ownership.yaml").read_text())

for op, row in own["operations"].items():
    request = json.loads(Path(row["fixtures"]["request"]).read_text())
    response = json.loads(Path(row["fixtures"]["response"]).read_text())
    operation_id = row["openapi_operation_id"]
    print(f"validate {operation_id}: request keys={sorted(request)} response keys={sorted(response)}")
    # client.submit(operation_id, request)
    # client.assert_schema(operation_id, response)

If the mock and the published environment diverge, record that gap in the human caveat file rather than in generated prose. Generated pages should describe the fixture the mock accepted. Caveat pages should describe environments, auth bootstrapping, and any behavior the mock cannot prove.

Step 5: Reject forbidden tokens in generated prose

Generated Markdown should not contain commitment vocabulary that the fixture cannot support. A denylist will not catch every overclaim, but it catches common leaks such as SLA, guaranteed, production, forever, retry, backoff, and GDPR. Caveat files may use those words because humans own them and reviewers read them. If the denylist fires, regenerate prose rather than patching the caveat file to match the model.

# Continuation inside verify_doc_examples.py
FORBIDDEN_DEFAULT = {"sla", "guaranteed", "production", "forever", "gdpr", "retry", "backoff"}

for op, row in own["operations"].items():
    page = Path(row["generated_page"]).read_text().lower()
    caveats = Path(row["human_caveats"]).read_text()
    body = page.split("\n---\n", 1)[0]  # generated enclosure only
    banned = set(row.get("forbidden_in_model_prose", [])) | FORBIDDEN_DEFAULT
    hits = [token for token in banned if token.lower() in body]
    if hits:
        errors.append(f"{op}: generated prose contains {hits}")
    if caveats.strip() == "":
        errors.append(f"{op}: human caveat file is empty")

Digit policy is stricter than word policy. Any number in generated prose should already appear in the fixture JSON or in the OpenAPI fragment copied onto the page. Timeouts, quotas, and calendar dates fail that test unless a human wrote them under docs/caveats/.

Step 6: Review the two-file diff and merge in that order

First merge fixture and OpenAPI changes when the contract actually moved. Second merge caveat edits when product, security, or SRE owners changed a promise. Third merge generated pages only after hash verification and mock round-trip have passed. Reversing that order invites a model draft to define behavior that no owner has accepted.

A practical review checklist stays short:

1. Confirm fixtures.sha256.json changed only when fixture bytes changed.
2. Confirm docs/caveats/* diffs name an owner in the pull request body.
3. Confirm docs/generated/* diffs contain no JSON edits except fixture copies.
4. Confirm CI ran hash_fixtures.py, verify_doc_examples.py, and mock_roundtrip.py.

Limitations

This workflow does not make generated prose legally reviewed, accessible, or localized. Hash equality proves example bytes match git, not that the API still behaves that way in a live region. The denylist is a coarse filter and will both miss clever overclaims and block innocent technical words. Teams without a maintained OpenAPI document and without fixture reviewers will automate empty confidence.

Availability notes in this article are not capacity, uptime, retention, or benchmark claims. Re-run the scripts on the target repository before adopting the ownership map. If fixture review is slower than prose generation, that is the point of the gate, not a defect in the generator.

Who should not use this approach

Skip this pipeline if marketing pages need invented happy-path stories rather than reviewed payloads. Skip it if legal or compliance text must be generated because no human owner will sign the caveat files. Skip it if examples include live secrets, customer data, or unreproducible timestamps that cannot be frozen. Skip it if the product has no OpenAPI snapshot and no mock that can validate fixtures.

Public API teams with checked-in fixtures, a spec gate, and distinct security or SRE owners benefit first. Internal tools with two consumers can start with one operation and one caveat file. The method scales by adding rows to the ownership map, not by widening what the model is allowed to invent.