Treat Commitment Verbs as Build Failures in Generated Docs

Generated API documentation fails in review when models invent support windows, deprecation dates, or rollback guarantees that nobody signed. A model can restate a function signature from a frozen extract without inventing product policy or calendar language. The practical fix is not a longer system prompt; it is a sentence-class gate that fails the build on unowned commitment verbs. The sections below define three classes, a file layout, a classifier, and a CI command that treats chat history as non-source.

Why prompt quality is the wrong control

Prompt instructions such as “do not hallucinate” do not survive the next template change, because they are not checked against output grammar. Reviewers then argue about tone while compatibility language slips into a generated README and becomes implied contract. A cheaper control is to classify each sentence before merge and to reject generated files that speak in guarantees. Chat transcripts remain useful debugging aids, but they are not valid inputs to the documentation compiler.

Recent discussion of agentic workflows often celebrates models that assume missing details and continue generating. Documentation generation needs the opposite default, because published pages can be quoted as if they were contracts. Assumption-heavy drafting remains useful for private sketches, yet it is harmful for public API pages that customers will cite.

Three sentence classes

Every sentence in a public docs tree falls into one of three classes for this workflow. The classes are about what the sentence does, not about how polished the prose sounds in review. Mixing them in one generated file is what turns a restatement into an accidental contract.

1. SURFACE restates extractable structure: names, types, required fields, HTTP methods, and error codes present in schema or tests.
2. PROCEDURE restates a sequence that a test or script already performs; each step must map to a command or assertion in tree.
3. COMMITMENT states duration, support, compatibility, security posture, refund, SLA, or any promise about future behavior.

SURFACE sentences are the only class a model may draft from extracts. PROCEDURE sentences may be drafted only when each step cites a test identifier that still passes. COMMITMENT sentences are human-owned and live in a separate file that the generator cannot write.

A three-file layout that encodes ownership

Keep one topic in three files so git blame and CI can see the class boundary without reading prose. Generated output never shares a path with policy language, which keeps diffs reviewable. The extract sits beside the docs so the compiler can prove it read a file, not a chat log.

docs/
  payments/
    surface.md        # generated; SURFACE sentences only
    procedure.md      # generated from tests, or handwritten
    commitment.md     # human only; generator cannot open for write
extracts/
  payments.facts.json
tests/
  test_doc_sentence_class.py
tools/
  extract_slots.py
  doc_class_gate.py

The generator receives payments.facts.json and may emit surface.md. It never receives commitment.md as an output path. Pull requests that touch commitment.md require a human reviewer who owns support policy; the gate does not try to author that file.

Workflow

1. Extract slot values from OpenAPI, JSON Schema, or public signatures into JSON that records source paths and content hashes.
2. Freeze that extract in git so two runs on the same commit produce the same fact list and the same hashes.
3. Classify the target filename: surface.md allows SURFACE; procedure.md allows PROCEDURE with citations; commitment.md is skipped for generation.
4. Draft SURFACE sentences with a model only from the extract, using slot-filling templates rather than a blank prompt.
5. Run the sentence-class gate on every surface.md and procedure.md file under docs/.
6. Fail CI if a generated file contains COMMITMENT patterns, RFC 2119 verbs, or a PROCEDURE step that lacks a test citation.
7. Merge COMMITMENT edits only from humans, and delete dated promises when the calendar date in that file has passed.

Artifact: extract slots, then lint sentences

The scripts below are labeled proposals you can run locally. They are conservative on purpose: unmatched modal language fails closed rather than passing as SURFACE. They do not claim production accuracy, latency, or coverage numbers.

# tools/extract_slots.py
from __future__ import annotations

import hashlib
import json
import sys
from pathlib import Path


def sha256(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()[:16]


def from_openapi(spec: dict, source: str) -> dict:
    paths = spec.get("paths") or {}
    endpoints = []
    for path, ops in paths.items():
        for method, op in (ops or {}).items():
            if method not in {"get", "post", "put", "patch", "delete"}:
                continue
            params = op.get("parameters") or []
            required = [p["name"] for p in params if p.get("required")]
            optional = [p["name"] for p in params if not p.get("required")]
            codes = sorted((op.get("responses") or {}).keys())
            endpoints.append(
                {
                    "method": method.upper(),
                    "path": path,
                    "required_fields": required,
                    "optional_fields": optional,
                    "error_codes": codes,
                    "operation_id": op.get("operationId"),
                }
            )
    raw = json.dumps(endpoints, sort_keys=True)
    return {"source": source, "source_hash": sha256(raw), "endpoints": endpoints}


def main() -> int:
    src = Path(sys.argv[1])
    spec = json.loads(src.read_text(encoding="utf-8"))
    extract = from_openapi(spec, source=str(src))
    Path(sys.argv[2]).write_text(json.dumps(extract, indent=2) + "\n", encoding="utf-8")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

# tools/doc_class_gate.py
from __future__ import annotations

import argparse
import re
import sys
from pathlib import Path

COMMITMENT = re.compile(
    r"\b("
    r"guarantee|warrant|promise|SLA|uptime|"
    r"supported until|support window|backward compatible|"
    r"no breaking changes|will always|will never|"
    r"we commit|we ensure|production-ready forever"
    r")\b",
    re.IGNORECASE,
)
PROCEDURE_CITE = re.compile(r"\[test:[A-Za-z0-9_.:-]+\]")
FUTURE_DATE = re.compile(r"\b20\d{2}-\d{2}-\d{2}\b")
RFC2119 = re.compile(r"\b(MUST|SHALL|SHOULD|MAY)\b")
TODO_EXTRACT = re.compile(r"TODO\(extract\)")


def sentences(text: str) -> list[str]:
    parts = re.split(r"(?<=[.!?])\s+", text.strip())
    return [p for p in parts if p]


def classify_file(path: Path) -> list[str]:
    errors: list[str] = []
    text = path.read_text(encoding="utf-8")
    kind = path.name
    if TODO_EXTRACT.search(text) and kind != "commitment.md":
        errors.append(f"{path}: unresolved TODO(extract) remains in generated output")
    for i, sent in enumerate(sentences(text), start=1):
        if COMMITMENT.search(sent) or FUTURE_DATE.search(sent):
            if kind != "commitment.md":
                errors.append(
                    f"{path}:{i}: commitment language in generated file: {sent!r}"
                )
        if kind == "procedure.md" and not PROCEDURE_CITE.search(sent):
            if re.match(r"\s*\d+\.", sent) or "run " in sent.lower():
                errors.append(
                    f"{path}:{i}: procedure step missing [test:...] citation: {sent!r}"
                )
        if kind == "surface.md" and RFC2119.search(sent):
            errors.append(
                f"{path}:{i}: RFC 2119 verb belongs in commitment.md: {sent!r}"
            )
    return errors


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--generated", action="append", required=True)
    args = parser.parse_args(argv)
    errors: list[str] = []
    for root in args.generated:
        for path in Path(root).rglob("*.md"):
            if path.name == "commitment.md":
                continue
            errors.extend(classify_file(path))
    for err in errors:
        print(err, file=sys.stderr)
    return 1 if errors else 0


if __name__ == "__main__":
    raise SystemExit(main())

Tests that pin the gate

These tests lock the classifier to examples, not to a claimed production error rate. Run them on the same commit as the extract so a green gate cannot hide a stale facts.json.

# tests/test_doc_sentence_class.py
from pathlib import Path

import tools.doc_class_gate as gate


def write(tmp_path: Path, name: str, body: str) -> Path:
    p = tmp_path / name
    p.write_text(body, encoding="utf-8")
    return p


def test_surface_restatement_passes(tmp_path):
    p = write(tmp_path, "surface.md", "POST /charges requires amount_cents as an integer.")
    assert gate.classify_file(p) == []


def test_surface_rejects_support_window(tmp_path):
    p = write(tmp_path, "surface.md", "This endpoint is supported until 2027-01-01.")
    errors = gate.classify_file(p)
    assert errors and "commitment" in errors[0]


def test_surface_rejects_rfc2119(tmp_path):
    p = write(tmp_path, "surface.md", "Clients MUST retry on 429.")
    errors = gate.classify_file(p)
    assert errors and "RFC 2119" in errors[0]


def test_procedure_requires_test_citation(tmp_path):
    p = write(tmp_path, "procedure.md", "1. Run curl against /charges with a zero amount.")
    errors = gate.classify_file(p)
    assert errors and "citation" in errors[0]


def test_procedure_with_citation_passes(tmp_path):
    p = write(
        tmp_path,
        "procedure.md",
        "1. Send a zero amount and expect 422 [test:test_zero_amount_returns_422].",
    )
    assert gate.classify_file(p) == []


def test_todo_extract_fails_closed(tmp_path):
    p = write(tmp_path, "surface.md", "Error codes: TODO(extract).")
    errors = gate.classify_file(p)
    assert errors and "TODO(extract)" in errors[0]

python tools/extract_slots.py openapi.json extracts/payments.facts.json
python -m pytest tests/test_doc_sentence_class.py -q
python -m tools.doc_class_gate --generated docs

Decision table for draftability

Sentence fragment	Class	Allowed file	Generator may draft
POST /charges requires amount_cents as integer	SURFACE	surface.md	yes, from schema
Error codes present in the spec: 400, 401, 422	SURFACE	surface.md	yes, from responses map
Send zero amount, expect 422 [test:...]	PROCEDURE	procedure.md	yes, if that test still passes
Clients MUST retry on 429	COMMITMENT	commitment.md	no
Supported until 2027-01-01	COMMITMENT	commitment.md	no
No breaking changes in 2026	COMMITMENT	commitment.md	no
Production-ready forever	COMMITMENT	commitment.md	no

The table is the policy. If a sentence does not fit a row, it does not belong in a generated file. Reviewers should not invent a fourth class called “helpful context” to smuggle calendar language back into surface.md.

Slot-filling instead of a blank prompt

When a model is used at all, constrain it to templates bound to extract keys. The following prompt is a labeled proposal, not a measured quality benchmark, and it should fail closed on missing keys.

Fill only this template from facts.json. Do not add verbs of commitment.
Endpoint {method} {path} requires {required_fields}.
Optional fields: {optional_fields}.
Error codes present in the spec: {error_codes}.
Output markdown for surface.md only.
If a fact is missing, emit TODO(extract) and stop. Do not guess codes.

The generator should refuse to invent error codes that are absent from the extract. Missing facts become TODO(extract) comments, which the gate fails if any remain in surface.md. That is a cheaper halt condition than asking the model to “use judgment,” because judgment is the definition of COMMITMENT.

Where free model access and a free server fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Restating SURFACE slots is a bounded transform over a JSON extract, which is a reasonable use of MonkeyCode's free model access. The extractor, classifier, and pytest suite can run on MonkeyCode's free server option so the gate is not bound to a single laptop. Neither the model nor the server should receive commitment.md as a writable target, and this article does not claim model names, quotas, hardware, duration, or permanence.

If the extract is empty, skip generation entirely. An empty extract plus a helpful model is how support windows appear in a README that no one intended to sign.

Limitations

Regex classification is not a parser of English law or of product policy. Idioms such as “this will always print a newline” in a language tutorial can fail the gate even when no commercial promise exists. RFC 2119 pages that are themselves the commitment document should live under commitment.md or a tree the gate ignores. Multilingual docs need per-language pattern lists; the sample above is English-only and will miss promises written without the listed verbs.

The gate does not prove that SURFACE sentences are true. Truth still depends on the extract being compiled from the same commit as the code. A stale facts.json will produce fluent, class-clean, and wrong documentation. Pair this lint with a hash check on the extract, which is a separate control from sentence class and is not implemented as a score.

Teams that publish marketing pages, narrative changelogs, or legal terms of service should not pretend this regex is counsel. Those documents are COMMITMENT by nature and need human authors from the first sentence. The classifier can only keep generated files from sounding like those documents by accident.

Who should not use this approach

Skip the workflow if the repository has no schema, tests, or other extractable sources. A model drafting from tribal knowledge will emit COMMITMENT language because that is what helpful prose usually sounds like. Skip it for internal design docs where speculation is the point of the file. Skip it if the docs are a single handwritten book with no generated files; adding a generator just to lint it creates ceremony without a class boundary.

Do not use the gate as a substitute for a compatibility policy. Failing CI on “supported until” does not create a support window. A human still has to write the date, own the calendar, and delete the sentence when the date passes. The workflow is for public reference surfaces that must not accidentally contract. It is not a general writing assistant, and it will feel hostile to exploratory drafting. Keep sketches out of docs/ so the gate only sees files that will be published.

If a docs compiler already exists in the repo, running this classifier beside it is a small next step; keep commitment files on a human review path.