Generate Error Catalogs Without Erasing On-Call Advice

The Friday regen looked successful. Every 4xx in payments/errors.py showed up on the public API page, each with a status code and a sample JSON body.

Saturday morning, support replayed a capture call because the new page treated 409 as a safe retry. It was not. The second capture settled a second time. This is a composite of a failure that appears whenever generated reference is treated as complete documentation.

Happy-path fields are cheap to regenerate. Remediation is not. The workflow below splits an error catalog into extractable facts a model may draft, and operational advice a human must own in a freeze file that CI can test.

The two halves of an error catalog

An error page that only lists codes is a dictionary. On-call needs a second half: retry class, blast radius, customer-visible wording, and whether a replay mutates money or data.

Those two halves do not share a lifecycle.

• Regenerable: error code, HTTP status, identifier in source, short mechanical description, example payload shape.
• Not regenerable: idempotency, retry class, paging rule, data-loss flag, compensation, language a customer will read.

If both halves live in one Markdown file, the next doc regen will keep the enum and drop the sentence that mattered. That is not a model quality problem. It is an ownership problem.

Ownership matrix

Use this table as the contract. If a field has no machine-readable source, it does not belong in the draft lane.

Field	Source of truth	Regen allowed	Owner
code	enum / constant in repo	yes	extractor
http_status	same source	yes	extractor
source_ref	file:line or symbol	yes	extractor
mechanical_summary	comment or identifier	yes	model draft
example_payload	schema or handler	yes	model draft
idempotent	freeze file only	no	human
retry_class	freeze file only	no	human
page_on	freeze file only	no	human
customer_wording	freeze file only	no	human
data_loss	freeze file only	no	human

retry_class is not a prose flourish. It is one of none, same_payload, backoff, or manual_only. If the model is allowed to invent that value from a handler name, it will guess backoff for anything that looks like a conflict.

Freeze file format

Keep generated output and human remediation in different paths. Proposed layout:

docs/
  generated/errors.catalog.json
  frozen/remediation.yml
  public/errors.md

remediation.yml is source, same as code. Proposed schema:

# docs/frozen/remediation.yml
# Human-owned. Regenerators must not write this file.
PAYMENT_CAPTURE_CONFLICT:
  idempotent: false
  retry_class: manual_only
  page_on: payments-oncall
  data_loss: duplicate_settlement
  customer_wording: >-
    Do not retry capture. Open a ticket with the payment intent id.
  last_reviewed: 2026-09-12
  reviewed_by: payments-api
INSUFFICIENT_FUNDS:
  idempotent: true
  retry_class: none
  page_on: null
  data_loss: none
  customer_wording: >-
    The method was declined. Ask the customer for another method.
  last_reviewed: 2026-09-10
  reviewed_by: payments-api

Missing keys are failures, not omissions the model can fill. A blank retry_class is how the Saturday capture happens again.

Extract from source, not from last week's Markdown

The draft lane should read the error enum, not the previous HTML. Proposed extractor against a small Python source of truth:

# payments/errors.py
from dataclasses import dataclass

@dataclass(frozen=True)
class ApiError:
    code: str
    http_status: int
    summary: str

PAYMENT_CAPTURE_CONFLICT = ApiError(
    "PAYMENT_CAPTURE_CONFLICT",
    409,
    "Capture rejected because the intent is not capturable in this state.",
)
INSUFFICIENT_FUNDS = ApiError(
    "INSUFFICIENT_FUNDS",
    402,
    "The processor declined the method for insufficient funds.",
)

CATALOG = (PAYMENT_CAPTURE_CONFLICT, INSUFFICIENT_FUNDS)

# tools/extract_errors.py — proposed, unexecuted in this article
import importlib.util
import json
from pathlib import Path

def load_catalog(path: str):
    spec = importlib.util.spec_from_file_location("errors", path)
    mod = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(mod)
    rows = []
    for item in mod.CATALOG:
        rows.append({
            "code": item.code,
            "http_status": item.http_status,
            "mechanical_summary": item.summary,
            "source_ref": path,
        })
    return rows

def main():
    rows = load_catalog("payments/errors.py")
    out = Path("docs/generated/errors.catalog.json")
    out.parent.mkdir(parents=True, exist_ok=True)
    out.write_text(json.dumps(rows, indent=2) + "\n")
    print(f"wrote {len(rows)} errors to {out}")

if __name__ == "__main__":
    main()

Run it as a boring command, not as a chat instruction:

python tools/extract_errors.py
python tools/check_remediation.py

Extraction can be wrong if the enum is incomplete. It cannot silently drop a freeze-file paragraph. That is the point.

What the model may draft

After extraction, the model may write only regenerable fields. Label the following as a proposed prompt, not a measured eval.

You receive a JSON array of API errors with code, http_status,
mechanical_summary, and source_ref.

For each item, draft:
- example_payload: a JSON object with "error" and "code" keys
- mechanical_summary_rewritten: one sentence, no retry advice

Do not invent idempotency, retry policy, paging, compensation,
or customer-facing promises. If those appear in the summary,
strip them. Return JSON only.

The output lands in docs/generated/errors.catalog.json. It never lands in docs/frozen/remediation.yml.

A drafting job needs somewhere to run that is disposable. Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you already have MonkeyCode's free model access and free server option, that pair is enough to run this draft lane as a batch job. It does not replace the freeze file or the checker. The product is optional; the split is not.

Merge is a program, not a paste

Public Markdown is a view over two files. Proposed merge:

# tools/render_errors.py — proposed
import json
from pathlib import Path
import yaml

def render():
    catalog = json.loads(Path("docs/generated/errors.catalog.json").read_text())
    frozen = yaml.safe_load(Path("docs/frozen/remediation.yml").read_text()) or {}
    lines = ["# Payments error catalog", ""]
    for row in catalog:
        code = row["code"]
        rem = frozen[code]
        lines += [
            f"## `{code}`",
            "",
            f"- HTTP status: `{row['http_status']}`",
            f"- Retry class: `{rem['retry_class']}`",
            f"- Idempotent: `{rem['idempotent']}`",
            "",
            rem["customer_wording"],
            "",
            "Example:",
            "",
            "```

json",
            json.dumps(row.get("example_payload", {"error": True, "code": code}), indent=2),
            "

```",
            "",
        ]
    Path("docs/public/errors.md").write_text("\n".join(lines))

if __name__ == "__main__":
    render()

If merge cannot find a freeze record, it must raise. Filling retry_class with backoff to keep the build green recreates the capture incident in CI clothing.

CI gate: the only test that matters

The test is not “does the page look complete.” The test is “would a full regen still print the irreversible warning.”

# tools/check_remediation.py — proposed
import json
import sys
from pathlib import Path
import yaml

REQUIRED = (
    "idempotent",
    "retry_class",
    "page_on",
    "data_loss",
    "customer_wording",
    "last_reviewed",
    "reviewed_by",
)
RETRY = {"none", "same_payload", "backoff", "manual_only"}

def main() -> int:
    catalog = json.loads(Path("docs/generated/errors.catalog.json").read_text())
    frozen_path = Path("docs/frozen/remediation.yml")
    frozen = yaml.safe_load(frozen_path.read_text()) or {}
    codes = [row["code"] for row in catalog]
    errors = []

    extra = sorted(set(frozen) - set(codes))
    missing = sorted(set(codes) - set(frozen))
    if extra:
        errors.append(f"freeze file has unknown codes: {extra}")
    if missing:
        errors.append(f"freeze file missing codes: {missing}")

    for code in codes:
        rec = frozen.get(code) or {}
        for key in REQUIRED:
            if key not in rec or rec[key] in ("", []):
                errors.append(f"{code}.{key} is empty")
        if rec.get("retry_class") not in RETRY:
            errors.append(f"{code}.retry_class is not in {sorted(RETRY)}")
        if rec.get("idempotent") is False and rec.get("retry_class") != "manual_only":
            errors.append(f"{code}: non-idempotent errors must be manual_only")

    # Guard against a generator writing the freeze path.
    text = frozen_path.read_text()
    if "AUTO-GENERATED" in text or "Drafted by" in text:
        errors.append("freeze file looks generated; refuse merge")

    for e in errors:
        print(e, file=sys.stderr)
    return 1 if errors else 0

if __name__ == "__main__":
    raise SystemExit(main())

Wire it so docs PRs cannot skip the gate:

# .github/workflows/error-catalog.yml — proposed
name: error-catalog
on:
  pull_request:
    paths:
      - "payments/errors.py"
      - "docs/**"
      - "tools/**"
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install pyyaml
      - run: python tools/extract_errors.py
      - run: python tools/check_remediation.py

A useful extra assertion: if payments/errors.py changes and docs/frozen/remediation.yml does not, require a human to either add a key or record why the new code is not customer-visible. Silence is not review.

What this does not prove

The checker proves the remediation record still exists and is well-typed. It does not prove the advice is correct. manual_only can be wrong. page_on can point at a dead rotation. last_reviewed can be stale while still present.

Treat review dates as a separate policy. A 90-day age check is easy to add. It is still a policy, not evidence that on-call agrees.

Other limits:

• If errors are raised as ad-hoc strings, extraction is theater. Put them in an enum first.
• Customer wording in regulated payments, health, or identity flows still needs a reviewer who owns that risk. A freeze file is not counsel.
• Localized public docs need a freeze record per locale, or a reviewed translation step. Do not let a draft job translate Do not retry capture unsupervised.
• Example payloads can leak internal field names. Keep examples on the public schema, not on the log line from production.

Who should not use this

Skip this split if you have no machine-readable error list. A model inventing both the catalog and the retry policy is how you get confident, wrong runbooks.

Skip it if the document is the legal contract. Terms of service, data-processing language, and uptime commitments are not error catalogs.

Skip it if the team will not fail CI on an empty retry_class. A freeze file that reviewers “mean to fill in later” is a blog post in your repo.

Recap

Regenerate codes, status, and example shapes from source. Store retry class, idempotency, paging, data-loss, and customer wording in a file no generator may write. Merge them in code. Fail the build when a new error has no freeze record, or when a non-idempotent error is marked safe to retry.

The question to ask on every docs regen is narrow. After a clean generate, is the do-not-retry line still there?