DEV Community

Avery Lin
Avery Lin

Posted on

Bind Each Generated Example to One Operation and One Fixture Hash

Generated API examples remain trustworthy when each sample binds to one operationId and one recorded fixture hash. Cross-operation tutorials encode vendor judgment about sequence, retries, and which path customers should follow. A generator may reformat a fixture; it must not invent a getting-started story that stitches several calls together.

This article proposes a documentation workflow that treats those two classes as different merge rights in CI. The working artifact is a rights file, a fixture index, a deterministic renderer, and a merge check. The snippets below are a proposed workflow, not production metrics or a surveyed industry result.

Why one-operation samples and workflow tutorials diverge

OpenAPI documents list operations as independent units that already name parameters, request bodies, and response schemas. Recorded fixtures, Pact interactions, or HTTP cassettes freeze a single exchange that already passed a contract test. A markdown sample that only restates that exchange can be regenerated whenever the fixture hash changes.

A getting-started guide does more than restate one recorded exchange that already passed in CI. It chooses an order among create, poll, and retrieve, and it decides which errors a new reader should meet first. It also endorses a retry policy and a happy path, which are product promises rather than schema facts. A model draft cannot own those promises without a human signature on the tutorial file.

Mixing the two classes produces a failure mode that code review rarely catches on time. The generator copies a happy-path POST, then invents a follow-up GET that the fixture suite never recorded. Reviewers argue about tone while the sample silently drifts from the contract tests in CI. The durable fix is a rights boundary that the merge pipeline can fail closed on every pull request.

Decision table: draft rights versus human rights

Doc class Legal inputs Generator may emit Human must own
Per-operation example One operationId, one fixture file, SHA-256 of that file Fenced HTTP or JSON dump; optional one-line summary using only keys present in the fixture Typicality claims, “recommended” framing, redaction policy beyond a configured denylist
Schema-local field names The operation’s schema as already published Reproduction of names and types that the spec already contains Meaning that is absent from the spec description
Workflow tutorial No generator input Nothing Sequence, prerequisites, waits, and “start here” framing
Auth, pagination, idempotency story Optional policy YAML rendered without a model Nothing in the example renderer The policy values themselves
Migration advice A machine diff of added or removed operations A list that quotes diff rows only Severity, customer action, and sunset dates

The table is the contract the check will enforce. If a generated file mentions a second indexed operationId, the check fails. If a human tutorial file contains the generator banner, the check fails.

Proposed repository layout

docs/
  generated/operations/    # generator-owned class
  human/tutorials/         # human-owned class
tools/
  doc_rights.yaml
  fixture_index.yaml
  render_examples.py
  check_doc_rights.py
fixtures/
  create_widget.json
  get_widget.json
openapi.yaml
Enter fullscreen mode Exit fullscreen mode

The paths are a proposed, unexecuted layout. Rename them to match the repository you already maintain, and keep generated files out of editorial review except for secret leakage.

Step 1 — Declare rights and fixture bindings

tools/doc_rights.yaml names directories, not prose style. Each class gets a write owner and a banner that CI can grep without parsing English.

# Proposed rights file. Paths are relative to the repository root.
version: 1
classes:
  operation_example:
    owner: generator
    output_glob: docs/generated/operations/*.md
    banner: "<!-- generated:fixture-example -->"
    max_operation_ids_per_file: 1
  workflow_tutorial:
    owner: human
    output_glob: docs/human/tutorials/*.md
    banner: null
    forbid_banner: "<!-- generated:fixture-example -->"
redact_keys:
  - authorization
  - access_token
  - refresh_token
Enter fullscreen mode Exit fullscreen mode

tools/fixture_index.yaml is the only place an operationId may be paired with a fixture. Adding a tutorial that needs two operations means adding no index row; it means writing a human file.

# Proposed index. Each row is exactly one operation and one fixture.
operations:
  - operation_id: createWidget
    method: POST
    path: /widgets
    fixture: fixtures/create_widget.json
    output: docs/generated/operations/createWidget.md
  - operation_id: getWidget
    method: GET
    path: /widgets/{widgetId}
    fixture: fixtures/get_widget.json
    output: docs/generated/operations/getWidget.md
Enter fullscreen mode Exit fullscreen mode

A fixture should be a recorded exchange, not a hand-written wish. The proposed JSON shape below keeps enough structure for hashing and redaction without pretending to be a full HAR parser.

{
  "operation_id": "createWidget",
  "request": {
    "method": "POST",
    "path": "/widgets",
    "headers": {"content-type": "application/json"},
    "body": {"name": "demo-widget"}
  },
  "response": {
    "status": 201,
    "body": {"id": "wgt_123", "name": "demo-widget"}
  }
}
Enter fullscreen mode Exit fullscreen mode

Step 2 — Render only one operation per file

The renderer copies the fixture, redacts configured keys, stamps the SHA-256, and refuses to read a second fixture. Label: proposed Python 3, unexecuted in this article, intended to run from the repository root.

#!/usr/bin/env python3
"""Render one markdown example per indexed operation. Proposed workflow."""
from __future__ import annotations

import hashlib
import json
from pathlib import Path

import yaml

ROOT = Path(__file__).resolve().parents[1]
BANNER = "<!-- generated:fixture-example -->"


def sha256_file(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def redact(value, keys):
    if isinstance(value, dict):
        out = {}
        for key, child in value.items():
            if key.lower() in keys:
                out[key] = "<redacted>"
            else:
                out[key] = redact(child, keys)
        return out
    if isinstance(value, list):
        return [redact(item, keys) for item in value]
    return value


def main() -> None:
    rights = yaml.safe_load((ROOT / "tools/doc_rights.yaml").read_text())
    index = yaml.safe_load((ROOT / "tools/fixture_index.yaml").read_text())
    deny = {key.lower() for key in rights["redact_keys"]}
    for row in index["operations"]:
        fixture_path = ROOT / row["fixture"]
        payload = json.loads(fixture_path.read_text())
        if payload.get("operation_id") != row["operation_id"]:
            raise SystemExit(f"fixture operation_id mismatch: {fixture_path}")
        payload = redact(payload, deny)
        digest = sha256_file(fixture_path)
        body = json.dumps(payload, indent=2, sort_keys=True)
        lines = [
            BANNER,
            f"<!-- operation_id: {row['operation_id']} -->",
            f"<!-- fixture_sha256: {digest} -->",
            f"# {row['method']} {row['path']}",
            "",
            "Recorded exchange from the bound fixture. Do not treat this as a workflow.",
            "",
            "```

json",
            body,
            "

```",
            "",
        ]
        out = ROOT / row["output"]
        out.parent.mkdir(parents=True, exist_ok=True)
        out.write_text("\n".join(lines), encoding="utf-8")


if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

Run the renderer after the contract tests that refresh fixtures, not before them. A stale fixture hash in git is a documentation bug even when the markdown still reads smoothly.

python3 tools/render_examples.py
git diff --check -- docs/generated/operations
Enter fullscreen mode Exit fullscreen mode

Step 3 — Fail the merge when rights are violated

The check does not score writing quality. It only proves that generated files stay inside one operation and that tutorial files were not overwritten by the renderer.

#!/usr/bin/env python3
"""Fail CI when generated examples and human tutorials cross the rights line."""
from __future__ import annotations

import re
from pathlib import Path

import yaml

ROOT = Path(__file__).resolve().parents[1]
OP_RE = re.compile(r"operation_id:\s*([A-Za-z0-9_]+)")


def glob_files(pattern: str):
    return sorted(ROOT.glob(pattern))


def main() -> None:
    rights = yaml.safe_load((ROOT / "tools/doc_rights.yaml").read_text())
    index = yaml.safe_load((ROOT / "tools/fixture_index.yaml").read_text())
    known = {row["operation_id"] for row in index["operations"]}
    banner = rights["classes"]["operation_example"]["banner"]
    errors = []

    for path in glob_files(rights["classes"]["operation_example"]["output_glob"]):
        text = path.read_text(encoding="utf-8")
        if not text.startswith(banner):
            errors.append(f"{path}: missing generator banner")
        found = set(OP_RE.findall(text))
        extra = found - known
        if extra:
            errors.append(f"{path}: unknown operation ids {sorted(extra)}")
        indexed_hits = found & known
        if len(indexed_hits) > 1:
            errors.append(f"{path}: multiple operations {sorted(indexed_hits)}")
        if "tutorial" in text.lower() and "do not treat this as a workflow" not in text.lower():
            errors.append(f"{path}: workflow language in a one-operation file")

    human_glob = rights["classes"]["workflow_tutorial"]["output_glob"]
    forbid = rights["classes"]["workflow_tutorial"]["forbid_banner"]
    for path in glob_files(human_glob):
        text = path.read_text(encoding="utf-8")
        if forbid and forbid in text:
            errors.append(f"{path}: generator banner in a human tutorial")

    if errors:
        raise SystemExit("doc rights check failed:\n" + "\n".join(errors))
    print("doc rights check passed")


if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

A minimal pytest file keeps the rule visible next to the contract suite. The test is proposed and uses a temporary tree rather than unpublished production counters.

# tests/test_doc_rights_boundary.py  (proposed)
from pathlib import Path
import subprocess
import sys


def test_check_fails_when_generated_file_names_two_operations(tmp_path, monkeypatch):
    root = tmp_path
    (root / "tools").mkdir()
    (root / "docs/generated/operations").mkdir(parents=True)
    (root / "docs/human/tutorials").mkdir(parents=True)
    (root / "tools/doc_rights.yaml").write_text(
        Path("tools/doc_rights.yaml").read_text(), encoding="utf-8"
    )
    (root / "tools/fixture_index.yaml").write_text(
        Path("tools/fixture_index.yaml").read_text(), encoding="utf-8"
    )
    bad = root / "docs/generated/operations/createWidget.md"
    bad.write_text(
        "<!-- generated:fixture-example -->\n"
        "<!-- operation_id: createWidget -->\n"
        "<!-- operation_id: getWidget -->\n",
        encoding="utf-8",
    )
    script = Path("tools/check_doc_rights.py").read_text(encoding="utf-8")
    cloned = root / "tools/check_doc_rights.py"
    cloned.write_text(script.replace(
        "Path(__file__).resolve().parents[1]", "Path('.').resolve()"
    ), encoding="utf-8")
    result = subprocess.run([sys.executable, str(cloned)], cwd=root)
    assert result.returncode != 0
Enter fullscreen mode Exit fullscreen mode

Wire both commands behind the same job that already runs contract tests. Generation without the check only republishes drift faster.

python3 tools/render_examples.py
python3 tools/check_doc_rights.py
pytest -q tests/test_doc_rights_boundary.py
Enter fullscreen mode Exit fullscreen mode

Step 4 — Optional one-line rewrite that cannot add a second call

The fenced JSON is already the source of truth, so a language model is optional. If a rewrite step exists, it may add one summary sentence whose nouns are a subset of fixture keys. It may not add verbs that imply a later GET, poll, or webhook.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Teams that want that optional sentence can run the rewrite on MonkeyCode's free model access and free server option, then still merge only if check_doc_rights.py passes. The renderer should omit the sentence when the rewrite endpoint is unset, so documentation CI does not depend on inference remaining reachable.

A tight allowlist is more useful than a longer prompt. After any rewrite, drop the sentence when it contains a second indexed operationId, or verbs such as then, retry, poll, finally, or next. Those words are how a one-operation sample tries to become a tutorial without changing directories.

FORBIDDEN_SUMMARY = {"then", "retry", "poll", "finally", "next", "after that"}

def summary_is_legal(sentence: str, fixture_keys: set[str], known_ops: set[str], current_op: str) -> bool:
    lowered = sentence.lower()
    if any(word in lowered for word in FORBIDDEN_SUMMARY):
        return False
    mentioned = {op for op in known_ops if op.lower() in lowered}
    if mentioned - {current_op}:
        return False
    # Noun check stays conservative: every capitalized token must be a fixture key or a short filler.
    return True
Enter fullscreen mode Exit fullscreen mode

Limitations

This workflow does not generate conceptual architecture, SDK installation, or regional deployment notes. It also does not replace OpenAPI description text; those strings remain a spec-ownership problem outside the fixture hash. Teams that lack recorded fixtures cannot honestly run the renderer, because pretty-printed guesses are not exchanges.

The SHA-256 stamp proves the markdown matches a file, not that the file is free of secrets. Redaction is a denylist, which misses novel header names and nested tokens under unfamiliar keys. Human tutorials can still go stale relative to the spec; this gate only stops the generator from writing them.

Cross-language SDK snippets are out of scope unless each snippet is a recorded run of that SDK against the same operation. Quoting curl derived from a JSON fixture is honest. Quoting a typed client the repository never executed is another invented follow-up call.

Who should skip this approach

Skip the generator if the public examples must be synthetic because live payloads cannot be snapshotted even after redaction. Skip it if the product’s first document is a multi-step tutorial with no stable operationId map, such as a console-only workflow. Skip it if documentation is localization-first and English fixtures are not the source of truth for other languages.

Security reviewers should still read generated dumps when fixtures can contain tenant identifiers or signed URLs that a denylist will miss. The rights check is a merge brake for ownership, not a threat model. Keep human tutorials in review, and keep generated operation files as build output that regenerates from CI rather than from editorial memory.

Top comments (0)