Bind Error Catalogs to Exception Classes Before a Model Drafts Them

Generated error documentation fails when a model invents status codes, retry windows, or apology language that no source file can support. The reliable split is mechanical: a model may restate exception types, HTTP mappings, and message templates that a scanner already extracted. A human must own severity, customer impact, deprecation of error codes, and any sentence that implies a support commitment. The rest of this article specifies a classifier, an extractor, and a validator that keep those lanes from mixing.

Why invented error codes are a documentation defect

Reference pages that list errors are not decorative prose. Integrators copy those codes into retry loops, alerting rules, and customer-facing runbooks, then treat the page as a contract. A generated paragraph that adds RATE_LIMIT_SOFT because it “sounds plausible” creates a second, unofficial API that support cannot honor. The failure mode is not bad grammar. The failure mode is an undeclared symbol that never existed in the exception hierarchy.

Cheap generation makes that failure cheaper to ship. When a heading can be filled in seconds, empty troubleshooting sections stop looking empty and start looking unfinished. Teams then ask a model to complete the catalog from the surrounding page instead of from the raise sites. The page becomes fluent, internally consistent, and factually unmoored from the binary that actually throws.

The corrective rule is narrow and testable. Every published error row must cite an exception class, a status mapping, or a frozen human claim. Rows that cannot cite one of those three origins stay as stubs. Models never receive a prompt that asks them to guess the missing code.

A field-level ownership matrix

Treat each error as a record with typed fields, not as a free-form section. The table below is a proposed contract for a generated catalog. It is not a claim about any production corpus.

Field	Origin that authorizes a draft	Owner if origin is missing
error_id	Exception class name or explicit code constant	Human freeze file only
http_status	Mapping next to the raise site or router	Leave blank
message_template	Literal in raise / constructor	Leave blank
when_it_occurs	Restatement of the raise predicate	Human if no predicate
retryable	Test named test_retry_* or freeze file	Human
severity	Freeze file only	Human
customer_action	Freeze file only	Human
support_promise	Freeze file only	Human
sunset_or_rename	Freeze file only	Human

The first four fields are observational when the scanner can prove them. The last five fields are obligations even when the model could write fluent guesses. Mixing those groups in one prompt is how catalogs drift. Keep them in separate files so a draft job cannot rewrite a promise by accident.

Proposed workflow

The following sequence is a proposed pipeline. It has not been executed against a public product in this article, and the commands are labeled examples.

1. Freeze human-owned claims in version control before any draft job runs.
2. Extract exception classes, status mappings, and message literals from source.
3. Join freeze rows to extracted rows by error_id; never invent an identifier to complete a join.
4. Send only observational fields to a restatement model, with the freeze file excluded from the prompt.
5. Validate the returned Markdown against the extracted set and the freeze hashes.
6. Fail the docs build if a published cell has origin inferred or if a freeze row changed wording.

Step 1: Keep obligations in a freeze file

Store promises where a model cannot complete them by adjacency. A small YAML file is enough for a first catalog and remains reviewable in pull requests.

# error_claims.freeze.yaml — human-owned; models must not receive this file
errors:
  QuotaExceeded:
    retryable: false
    severity: error
    customer_action: "Reduce request volume or request a quota increase."
    support_promise: "Quota denials are not auto-retried by the platform."
    sunset_or_rename: null
  StaleCursor:
    retryable: true
    severity: warning
    customer_action: "Restart the listing call with a fresh cursor."
    support_promise: null
    sunset_or_rename: null

Hash the freeze file in CI so silent edits are visible. The hash is a change detector, not a quality score. Reviewers still decide whether a new promise is acceptable.

# example: pin the freeze file before generation
sha256sum error_claims.freeze.yaml > error_claims.freeze.sha256
git add error_claims.freeze.yaml error_claims.freeze.sha256

Step 2: Extract only what the compiler already knows

The extractor below is proposed Python. It walks a tree of .py files, records class names that end with Error or subclass a local ApiError, and captures a nearby HTTP status if one is assigned on the class. Dynamically constructed exceptions will be missed; that miss is a feature, because missed rows stay human-owned instead of becoming invented codes.

# extract_exceptions.py — proposed scanner, not a measured production run
from __future__ import annotations

import ast
from pathlib import Path
from typing import Any


class ExceptionCollector(ast.NodeVisitor):
    def __init__(self) -> None:
        self.rows: list[dict[str, Any]] = []

    def visit_ClassDef(self, node: ast.ClassDef) -> None:
        bases = [ast.unparse(b) for b in node.bases]
        looks_like_error = node.name.endswith("Error") or any(
            "Error" in base or "Exception" in base for base in bases
        )
        if not looks_like_error:
            self.generic_visit(node)
            return

        status = None
        template = None
        for stmt in node.body:
            if not isinstance(stmt, ast.Assign):
                continue
            for target in stmt.targets:
                if not isinstance(target, ast.Name):
                    continue
                if target.id == "http_status" and isinstance(stmt.value, ast.Constant):
                    status = stmt.value.value
                if target.id == "message_template" and isinstance(stmt.value, ast.Constant):
                    template = stmt.value.value
        self.rows.append(
            {
                "error_id": node.name,
                "bases": bases,
                "http_status": status,
                "message_template": template,
                "origin": "source_class",
            }
        )
        self.generic_visit(node)


def extract(root: Path) -> list[dict[str, Any]]:
    collector = ExceptionCollector()
    for path in root.rglob("*.py"):
        tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path))
        collector.visit(tree)
    return collector.rows

Run the scanner as a docs prerequisite, not as a post-processing linter after publication. If the catalog is generated first and scanned later, invented identifiers already have a chance to land in search indexes.

# example commands
python extract_exceptions.py > extracted_errors.json
python join_freeze.py extracted_errors.json error_claims.freeze.yaml > catalog.input.json

Step 3: Draft only restatable fields

The prompt surface should be a JSON record with observational keys, plus an explicit deny list. Do not paste the freeze file “for context.” Adjacent obligation text is how models absorb promises and rephrase them as if they were observations.

A restatement job needs a model endpoint and a place to run. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option that can host this restatement step without expanding the model's authority over freeze fields. Those two availability claims are the only product facts used here; this article does not assign model names, quotas, hardware, duration, or benchmark numbers.

# build_draft_prompt.py — proposed prompt builder
OBSERVATIONAL_KEYS = ("error_id", "http_status", "message_template", "bases")

PROMPT_RULES = """
You restate extracted exception metadata as a Markdown table.
You may write a one-sentence 'when_it_occurs' clause only if bases or the
message_template already imply a predicate.
You must not invent error_id values, HTTP statuses, retry policy, severity,
customer actions, support promises, or sunset dates.
If a field is null, output an empty cell, not a guess.
"""

def records_for_model(joined_rows: list[dict]) -> list[dict]:
    payload = []
    for row in joined_rows:
        payload.append({k: row.get(k) for k in OBSERVATIONAL_KEYS})
    return payload

If the free server option is used as the job host, keep the freeze file off that host's prompt directory. Network isolation is optional; prompt isolation is not. A model that cannot see a promise cannot accidentally rewrite it.

Step 4: Validate published cells against origins

The validator is the actual product of this workflow. Generation without a reject path is editing, not documentation control. The checks below are deliberately boring: unknown identifiers, filled blanks, and mutated freeze hashes.

# validate_error_docs.py — proposed gate
import json
import re
from pathlib import Path

ID_RE = re.compile(r"\|\s*([A-Za-z_][A-Za-z0-9_]*)\s*\|")


def validate(markdown: str, extracted_ids: set[str], freeze_text: str, freeze_hash: str) -> list[str]:
    errors: list[str] = []
    published_ids = set(ID_RE.findall(markdown))
    invented = published_ids - extracted_ids
    if invented:
        errors.append(f"undeclared error_id values: {sorted(invented)}")
    if "TODO_HUMAN" not in markdown and freeze_text and "support_promise" in markdown:
        # freeze-backed sections must keep the stub marker when a claim is null
        pass
    if freeze_hash not in Path("error_claims.freeze.sha256").read_text():
        errors.append("freeze hash is stale; re-review human-owned claims")
    forbidden = ("we will always", "guaranteed", "SLA", "99.9", "immediately retry")
    lower = markdown.lower()
    for token in forbidden:
        if token.lower() in lower:
            errors.append(f"obligation language appeared outside the freeze file: {token}")
    return errors

Forbidden-token lists are incomplete by nature. They exist to catch the most common leaked promises, not to replace legal review. Pair them with the identifier check, which is the stronger constraint because it does not depend on English phrasing.

What the model may draft, restated as rules

Write the ownership split as rules a reviewer can apply without reading this article again.

1. A model may draft a table row only when error_id exists in extracted_errors.json.
2. A model may fill http_status and message_template only when the scanner stored non-null values.
3. A model may write when_it_occurs as a restatement of those values, not as operational advice.
4. A human must write retryable, severity, customer_action, support_promise, and any sunset note.
5. A human must write any paragraph that names a customer, a refund, a timeline, or a workaround that is not in tests.
6. CI must reject invented identifiers even when the surrounding prose is accurate.

These rules scale to other reference surfaces. Parameter tables, webhook event names, and CLI flag lists fail in the same way when a model completes a set from English instead of from declarations. Error catalogs are simply the place where an invented token causes the most operational damage.

Limitations

Static extraction under-counts errors raised through helpers, string-built class names, or responses assembled in a gateway outside the scanned tree. Teams that wrap every failure in a generic ApiError(code=request.headers[...]) will extract almost nothing useful. In that design the catalog is not a documentation problem first; it is an API-surface problem, and the freeze file will become the entire source of truth.

The validator cannot prove that a restated when_it_occurs sentence matches production traffic. It can only prove that the identifier and status were not invented. Semantic drift inside an allowed cell still needs a reviewer. Multilingual catalogs need one freeze file per locale, because translating a support promise is still owning that promise.

The workflow also assumes error identifiers are stable public tokens. If a codebase uses anonymous HTTP 400 bodies with free-form strings, binding a catalog to class names will overstate precision. Do not publish class names as customer-facing codes unless those names are already part of the API.

Who should not use this approach

Do not use this pipeline for marketing comparison pages, security attestations, pricing notes, or incident reports. Those documents are obligation-heavy and have almost no extractable exception surface. Do not use it as a substitute for an on-call taxonomy when the product still returns unstructured error blobs. Do not point the restatement job at a heading that asks for “complete troubleshooting coverage,” because that instruction invites inference by design.

Teams without code review on the freeze file should not automate publication. A generated catalog with an unreviewed promise file is still a generated contract. The scanner only protects observational fields. It does not make human claims true.

Closing

Error docs stay honest when generation is restricted to identifiers the compiler already emitted, and when every retry or support sentence lives in a hashed freeze file. If you adapt the join-and-validate steps, keep obligation rows out of the prompt entirely rather than asking a model to leave them blank. Blank instructions are weaker than absent context, and absent context is the cheaper control.