DEV Community

Avery Lin
Avery Lin

Posted on

Treat Timeout, Retry, and Calendar Sentences as Merge Blockers in Generated Docs

Generated API documentation should copy the wire surface that contract tests already prove, then halt. Timeout budgets, retry counts, idempotency promises, and support calendars are client contracts, not observations a model may invent. A sentence classifier plus a merge gate can reject those claims unless a named human owner stamps the file. The sections below define the classes, the gate, and a testable implementation that runs before publication.

Why generated docs keep inventing wait budgets

Agent-style documentation pipelines often continue past the schema and start coaching the client. They emit retry counts, wait intervals, or support end dates because those phrases sound complete to a language model. Completeness is not evidence, and those numbers exist only when a test or a support owner recorded them. After a draft merges, later agents copy the invented budget into runbooks, SDKs, and public status copy.

The failure mode shows up in review comments even when production metrics are not on hand. Reviewers debate numbers that never appeared in OpenAPI, while the actual path list receives almost no scrutiny. That inversion wastes review time and creates a contract the on-call rotation never accepted. The gate below inverts the default so wire facts may be drafted, while client-contract language blocks merge until a human owns it.

Two surfaces and four sentence classes

Keep a strict split between the observed wire surface and the obligation surface at all times. The model may draft the first surface only from cited machine-readable sources. The second surface stays human-authored, even when the needed prose is only one or two sentences. Mixed sentences inherit the stricter class, which keeps smuggled budgets from hiding beside a status code.

Class Typical language Allowed source Model draft Merge rule
WIRE path, method, status, field type OpenAPI, JSON Schema, named test yes require citation
EXAMPLE request or response body checked-in fixture file yes cite fixture path
CONTRACT timeout, retry, SLA, calendar policy owner, incident record no owner stamp required
POLICY recommend, should, must for clients API governance note no owner stamp required

A sentence such as POST /v1/refunds returns 409 and clients should retry after 30s is CONTRACT, not WIRE. Split that sentence before generation rather than hoping a reviewer will notice the smuggled wait budget during an otherwise routine docs review.

Workflow: inventory, quarantine, classify, then stamp

Apply the following numbered steps on every documentation change that a model is allowed to touch. Do not skip the quarantine directory, because mixing draft files with owned files is how invented calendars reach the default branch.

  1. Collect restatement sources into a manifest that lists OpenAPI documents, schema files, and test identifiers only. Refuse blog posts, chat transcripts, and earlier generated markdown as sources, since those files already contain unowned contracts.
  2. Prompt the model to emit WIRE and EXAMPLE sentences that cite source or test for every paragraph it writes. Instruct it to omit timeout, retry, calendar, availability, and recommendation language with no exceptions listed.
  3. Write the draft under docs/_generated/ and never into docs/contracts/, because the generated tree is disposable. Treat the contracts tree as human-owned source that is reviewed with the same care as production code.
  4. Run the classifier across both trees and fail the build on two conditions that reviewers should not negotiate. CONTRACT or POLICY language under _generated/ is always an error, and the same language under contracts/ is an error without YAML owner metadata.
  5. Have a human write obligation copy in docs/contracts/ with owner, reviewed_at, and policy_ref front matter before review starts. Numbers in that file must match the cited policy document, including units, and must not be copied from the model draft.
  6. Merge only when unit tests pass, the classifier exits zero, and the diff does not relocate CONTRACT sentences back into _generated/. Record the classifier exit code in CI so a green docs job cannot hide a contract violation.

Negative prompt instructions in step two are not sufficient control on their own. Models routinely ignore omission rules, which is why the classifier, not the prompt, is the merge authority for this workflow.

Worked failure: one status line plus a smuggled retry

Consider an OpenAPI fragment that only declares POST /v1/refunds with a 409 response and no timeout extension. A model draft that remains legal would read: POST /v1/refunds returns 409 Conflict (source: openapi.yaml#/paths/~1v1~1refunds). That sentence is WIRE because it names a path, a status, and a citation that a reviewer can open.

The illegal continuation usually arrives in the same paragraph, still wearing a citation. POST /v1/refunds returns 409 Conflict; wait 30 seconds and retry once (source: openapi.yaml#/paths/~1v1~1refunds). remains CONTRACT because of wait 30 seconds and retry. The citation does not redeem the budget, since the OpenAPI file never stated either number. CI must fail that file while it remains under docs/_generated/.

The human-owned replacement lives in a different tree and carries identity metadata that a roster can resolve:

---
owner: api-oncall
reviewed_at: 2026-09-10
policy_ref: policies/refunds-retry.md
---

Refund creates return 409 on duplicate `Idempotency-Key` values.
Retry policy, including any wait interval, is defined only in `policies/refunds-retry.md`.
Enter fullscreen mode Exit fullscreen mode

The second sentence still contains retry language, so the classifier labels it CONTRACT and then accepts it because owner and policy_ref are present. That is the intended split: the model never chooses the interval, and the human points at the policy that does.

Artifact: classifier, tests, and a local command

The Python module below is a proposal you can drop into CI without changing your documentation generator. It fails closed on contract verbs even when a surrounding citation looks complete, which is stricter than a style linter.

#!/usr/bin/env python3
"""Classify markdown sentences and fail unowned client-contract language."""

from __future__ import annotations

import re
import sys
from pathlib import Path

import yaml

CONTRACT_VERBS = re.compile(
    r"\b(timeout|timed out|retry|retries|backoff|idempotent|"
    r"sla|slo|availability|uptime|supported until|deprecat|"
    r"breaking change|wait\s+\d+|at least \d+|up to \d+)\b",
    re.I,
)
POLICY_VERBS = re.compile(
    r"\b(recommend|should|must|do not retry|clients ought)\b",
    re.I,
)
CITATION = re.compile(r"\((?:source|test|schema):\s*[^)]+\)", re.I)
FRONT_MATTER = re.compile(r"^---\n(.*?)\n---\n", re.S)


def sentences(text: str) -> list[str]:
    body = FRONT_MATTER.sub("", text)
    return [s.strip() for s in re.split(r"(?<=[.!?])\s+", body) if s.strip()]


def classify(sentence: str) -> str:
    if CONTRACT_VERBS.search(sentence):
        return "CONTRACT"
    if POLICY_VERBS.search(sentence):
        return "POLICY"
    return "WIRE"


def owner_meta(text: str) -> dict:
    match = FRONT_MATTER.match(text)
    if not match:
        return {}
    data = yaml.safe_load(match.group(1)) or {}
    return data if isinstance(data, dict) else {}


def check_file(path: Path) -> list[str]:
    text = path.read_text(encoding="utf-8")
    meta = owner_meta(text)
    generated = "/_generated/" in path.as_posix()
    errors: list[str] = []
    for sentence in sentences(text):
        kind = classify(sentence)
        if kind == "WIRE":
            if generated and not CITATION.search(sentence):
                errors.append(f"{path}: WIRE lacks citation: {sentence}")
            continue
        if generated:
            errors.append(f"{path}: {kind} language in generated tree: {sentence}")
            continue
        if not meta.get("owner") or not meta.get("policy_ref"):
            errors.append(f"{path}: {kind} without owner/policy_ref: {sentence}")
    return errors


def main(argv: list[str]) -> int:
    roots = [Path(a) for a in argv[1:]] or [Path("docs")]
    errors: list[str] = []
    for root in roots:
        for path in root.rglob("*.md"):
            errors.extend(check_file(path))
    for item in errors:
        print(item)
    print(f"contract_gate errors={len(errors)}")
    return 1 if errors else 0


if __name__ == "__main__":
    raise SystemExit(main(sys.argv))
Enter fullscreen mode Exit fullscreen mode

Pair the module with tests so the lexicon cannot shrink during a drive-by cleanup. These cases encode merge policy for client-contract language rather than any vendor quota or hardware claim.

# test_classify_docs.py
from classify_docs import classify


def test_wire_status_is_wire():
    s = "POST /v1/refunds returns 409 Conflict (source: openapi.yaml#/paths)."
    assert classify(s) == "WIRE"


def test_retry_budget_is_contract():
    s = "Clients should retry the refund after 30 seconds."
    assert classify(s) == "CONTRACT"


def test_support_calendar_is_contract():
    s = "Version 1 is supported until 2027-01-01."
    assert classify(s) == "CONTRACT"


def test_recommendation_is_policy():
    s = "We recommend callers cache the location header."
    assert classify(s) == "POLICY"
Enter fullscreen mode Exit fullscreen mode

Run both commands before every model draft so the gate is proven on a cold tree. Optional Make wiring keeps the same two commands as the only docs-gate entry point.

python3 -m pip install pyyaml pytest
python3 -m pytest test_classify_docs.py -q
python3 classify_docs.py docs
Enter fullscreen mode Exit fullscreen mode
.PHONY: docs-gate
docs-gate:
    python3 -m pytest test_classify_docs.py -q
    python3 classify_docs.py docs
Enter fullscreen mode Exit fullscreen mode

Generated files that mention retry up to 3 times must fail even when the remaining page only restates paths and field types. That failure is the product of the quarantine tree, not a prompt-engineering miss.

Where a drafting host belongs in this loop

The restatement pass needs some model and some machine, but neither one should author CONTRACT sentences for your clients. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode currently offers free model access and a free server option that can host the inventory-and-draft loop against docs/_generated/ while this classifier remains the merge authority. Use that option only for WIRE and EXAMPLE restatement; keep timeout numbers, retry ceilings, and support calendars inside the human-owned tree. If a hosted draft inserts a wait budget, the gate still fails, and that failure is the correct result.

Limitations and who should skip this gate

Lexical classification is not semantic understanding, and that gap is acceptable for merge control. The regex will flag do not retry inside a human contract file, which is correct because the sentence still needs an owner. It will miss a vague paragraph that implies a deadline without digits, so reviewers must still read POLICY-adjacent prose. Multi-sentence examples can hide wait inside fenced code, and teams that paste SDK comments from docs should extend the scanner to inspect those fences.

This workflow also does not prove that a cited test actually asserts the named status code. Keep contract tests as a separate CI job that executes the suite rather than grepping markdown citations. Do not use this approach as a substitute for legal review of public SLAs, and do not apply it to incident reports where timestamps are evidence rather than promises. Teams that publish only narrative guides with no OpenAPI or schema surface will get little value, because the WIRE class would have nothing to cite. If the documentation corpus is itself the policy record, skip generation entirely and write the contracts by hand.

Pre-merge checklist

Before accepting a generated documentation pull request, confirm four facts directly in the diff. Every generated sentence cites a schema path or a test identifier that exists on the branch. No generated file contains timeout, retry, calendar, or availability language of any kind. Every CONTRACT file lists owner and policy_ref values that a human can ping in the current roster. Numeric claims in CONTRACT files match the policy document, including units, rather than matching the model draft.

That checklist is deliberately uneventful on a clean change. Uneventful merge rules are how invented wait budgets stop becoming customer-visible contracts.

Top comments (0)