Join Generated Error Indexes to an Owned Overlay, or Fail the Docs Build

Error catalogs rot when humans rewrite extractable fields and models invent severity they cannot observe. The durable split compiles codes, symbols, and message templates from source, then requires a signed owned file. That owned file must carry severity, customer-facing copy, and the operator action assigned to each code. A documentation build that cannot join those two files should fail in CI before merge.

This article proposes a small compiler, an overlay schema, and a join-gate test you can copy. The examples below are labeled proposals and remain unexecuted against any live production catalog. Copy them into a repository and adapt the walker, rather than treating them as measured incident reductions. Keep judgment fields out of any model prompt that could page a human or reach a customer.

Why error indexes drift

Most stale error docs fail in one of three mechanical ways, and none of them require a quality debate. First, a code ships in telemetry while the public list still omits it, so support searches the wrong string. Second, a human rewrites a template in Markdown, so the docs no longer match the binary that emitted the log. Third, a model assigns high severity because a template contains timeout, which pages people for brownouts that are not incidents.

Those failure modes share a cause: extractable facts and judgment fields were stored in the same editable document. The compiler-plus-overlay split makes the first two failures a diff, and the third failure a missing required key. Reviewers then spend time on runbook quality instead of reconciling duplicate lists by hand every release. None of this claims a reduction in incident minutes; it only claims a merge-time proof that every shipped code has an owner-reviewed overlay row.

Ownership split

Source already knows the stable identifiers that support and telemetry will search during an incident. A compiler should walk enums or constant maps and emit only extractable facts. Those facts include the code, the symbol, the in-source message template, and the defining path. They do not include whether the event is SEV-1, whether a customer email is allowed, or how an on-call engineer remediates the failure.

Judgment fields do not live in comments that a model can polish into policy after the fact. Severity, paging, customer copy, runbook steps, and PII class change company behavior under incident pressure. A model may propose related codes from call sites, and a reviewer can accept or reject that proposal in the overlay. The publish rule is still that a missing overlay key fails the build, even when a generated description looks fluent.

Field	Extract from source	Model may draft	Human must own	Fail the build when
code, symbol, template, path	yes	no	no	row missing from HEAD enums
related_codes	maybe call sites	propose only	accept or reject	published without overlay ack
technical_notes	comments	yes, labeled draft	optional edit	published unlabeled as draft
severity, pages_oncall	no	no	yes	empty or model-authored
customer_copy	no	no	yes	empty, or equal to the template
runbook_steps	no	no	yes	empty on paging codes
pii_class	no	no	yes	empty when the template has placeholders

Artifact layout

The following layout is a proposal for a Python service that already declares errors in one module. Adapt the walker if codes live in protobuf, OpenAPI, or more than one language. Do not paste secrets, tokens, or production hostnames into the overlay or into any draft prompt.

docs/errors/
  generated/index.json      # compiler output; never hand-edit
  owned/overlay.json        # human-owned judgment fields
scripts/
  compile_error_index.py
  join_error_docs.py
tests/
  test_error_doc_join.py
app/
  errors.py

Six publish steps

Step 1 — Keep the catalog of record in imported source

Keep the catalog of record in code that production already imports and that tests already instantiate. A parallel Markdown list will drift within a week, because nobody diffs it against telemetry. Literal codes and templates keep the compiler honest, since interpolated strings from variables cannot be proven at compile time. The module below is a labeled example, not an extracted production catalog.

# app/errors.py
from enum import Enum
from typing import NamedTuple


class ErrorDef(NamedTuple):
    code: str
    template: str


class AppError(Enum):
    AUTH_TOKEN_EXPIRED = ErrorDef(
        "AUTH_TOKEN_EXPIRED",
        "token expired for user_id={user_id}",
    )
    PAYMENT_GATEWAY_TIMEOUT = ErrorDef(
        "PAYMENT_GATEWAY_TIMEOUT",
        "gateway timeout on charge_id={charge_id}",
    )
    QUOTA_EXCEEDED = ErrorDef(
        "QUOTA_EXCEEDED",
        "quota exceeded for tenant={tenant_id} key={key}",
    )
    INTERNAL_INVARIANT = ErrorDef(
        "INTERNAL_INVARIANT",
        "invariant failed in {component}",
    )

Step 2 — Compile extractable facts on every documentation job

The compiler must refuse to honor hand edits in generated/index.json after a reviewer “fixes wording.” Rerun it in CI with --check and fail when the working tree differs from a clean compile. Walking the AST keeps templates tied to literals, which blocks a silent swap to a runtime-formatted string. The script is a proposal; point SOURCE at the module your service actually imports.

#!/usr/bin/env python3
"""Compile extractable error facts. Labeled proposal; unexecuted against your tree."""
from __future__ import annotations

import ast
import json
import sys
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]
SOURCE = ROOT / "app" / "errors.py"
OUT = ROOT / "docs" / "errors" / "generated" / "index.json"


class ErrorIndexError(RuntimeError):
    pass


def _const_str(node: ast.AST) -> str | None:
    if isinstance(node, ast.Constant) and isinstance(node.value, str):
        return node.value
    return None


def compile_index(source: Path, root: Path) -> dict:
    if not source.is_file():
        raise ErrorIndexError(f"missing source: {source}")
    tree = ast.parse(source.read_text(encoding="utf-8"), filename=str(source))
    rows: list[dict] = []
    seen: set[str] = set()
    for class_node in tree.body:
        if not isinstance(class_node, ast.ClassDef):
            continue
        for stmt in class_node.body:
            if not isinstance(stmt, ast.Assign) or len(stmt.targets) != 1:
                continue
            target = stmt.targets[0]
            if not isinstance(target, ast.Name):
                continue
            value = stmt.value
            if not isinstance(value, ast.Call):
                continue
            func = value.func
            if not isinstance(func, ast.Name) or func.id != "ErrorDef":
                continue
            if len(value.args) < 2:
                raise ErrorIndexError(f"{target.id} is not ErrorDef(code, template)")
            code = _const_str(value.args[0])
            template = _const_str(value.args[1])
            if code is None or template is None:
                raise ErrorIndexError(f"{target.id} uses a non-literal code or template")
            if code in seen:
                raise ErrorIndexError(f"duplicate code {code}")
            seen.add(code)
            rows.append(
                {
                    "code": code,
                    "defining_class": class_node.name,
                    "lineno": stmt.lineno,
                    "source_path": str(source.relative_to(root)),
                    "symbol": target.id,
                    "template": template,
                }
            )
    if not rows:
        raise ErrorIndexError("no ErrorDef rows found")
    rows.sort(key=lambda row: row["code"])
    return {
        "row_count": len(rows),
        "rows": rows,
        "source_path": str(source.relative_to(root)),
    }


def render(payload: dict) -> str:
    return json.dumps(payload, indent=2, sort_keys=True) + "\n"


def main() -> int:
    try:
        payload = compile_index(SOURCE, ROOT)
    except ErrorIndexError as exc:
        print(f"compile_error_index: {exc}", file=sys.stderr)
        return 1
    serialized = render(payload)
    if "--check" in sys.argv:
        if not OUT.is_file() or OUT.read_text(encoding="utf-8") != serialized:
            print("compile_error_index: generated index is stale", file=sys.stderr)
            return 1
        return 0
    OUT.parent.mkdir(parents=True, exist_ok=True)
    OUT.write_text(serialized, encoding="utf-8")
    print(f"wrote {OUT} ({payload['row_count']} rows)")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Step 3 — Require an overlay row for every compiled code

Store judgment fields beside the generated index, not inside it, so a regenerate cannot clobber on-call policy. JSON keeps the join-gate on the standard library; teams that prefer YAML can wrap the same schema. Required reviewers should sit on docs/errors/owned/, because that directory is the incident-facing surface. The overlay below is a labeled example and is not a recommendation of real severity numbers.

{
  "version": 1,
  "codes": {
    "AUTH_TOKEN_EXPIRED": {
      "severity": "SEV-4",
      "pages_oncall": false,
      "pii_class": "user_identifier",
      "customer_copy": "Your session expired. Sign in again to continue.",
      "runbook_steps": [
        "Do not rotate signing keys for isolated expiry spikes.",
        "Check client clock skew only when expiry clusters on one device class."
      ],
      "related_codes_ack": []
    },
    "PAYMENT_GATEWAY_TIMEOUT": {
      "severity": "SEV-2",
      "pages_oncall": true,
      "pii_class": "payment_identifier",
      "customer_copy": "A payment processor timed out. No extra charge was created from this retry.",
      "runbook_steps": [
        "Compare gateway success rate against the five-minute baseline dashboard.",
        "Page only when timeouts exceed the documented provider SLO, not on a single charge."
      ],
      "related_codes_ack": []
    },
    "QUOTA_EXCEEDED": {
      "severity": "SEV-3",
      "pages_oncall": false,
      "pii_class": "tenant_identifier",
      "customer_copy": "This workspace hit its current quota. Raise the limit or wait for reset.",
      "runbook_steps": [
        "Confirm the tenant is not retrying a stuck batch that double-counts usage."
      ],
      "related_codes_ack": []
    },
    "INTERNAL_INVARIANT": {
      "severity": "SEV-1",
      "pages_oncall": true,
      "pii_class": "none",
      "customer_copy": "We hit an unexpected internal error. The team was paged.",
      "runbook_steps": [
        "Treat this as a bug, not a retryable user error.",
        "Capture the component field from logs without pasting raw payloads into the status page."
      ],
      "related_codes_ack": []
    }
  }
}

Step 4 — Fail the join when paging codes lack runbooks

The join-gate is the actual product of this workflow, not the generated Markdown that people skim. It should fail closed when code sets differ, when paging is true without steps, and when customer copy reprints the log template. It should also fail when customer copy still contains format placeholders, because those placeholders are telemetry, not sentences a customer should see. Print machine-readable violations so CI logs stay greppable during a release freeze.

#!/usr/bin/env python3
"""Join generated error facts to a human-owned overlay. Labeled proposal."""
from __future__ import annotations

import json
import sys
from pathlib import Path
from typing import Any

ROOT = Path(__file__).resolve().parents[1]
GENERATED = ROOT / "docs" / "errors" / "generated" / "index.json"
OVERLAY = ROOT / "docs" / "errors" / "owned" / "overlay.json"

ALLOWED_SEVERITY = {"SEV-1", "SEV-2", "SEV-3", "SEV-4", "SEV-5"}
ALLOWED_PII = {
    "none",
    "user_identifier",
    "tenant_identifier",
    "payment_identifier",
}
REQUIRED = (
    "severity",
    "pages_oncall",
    "pii_class",
    "customer_copy",
    "runbook_steps",
    "related_codes_ack",
)


def load_json(path: Path) -> Any:
    return json.loads(path.read_text(encoding="utf-8"))


def join_errors(generated: dict, overlay: dict) -> list[str]:
    violations: list[str] = []
    gen_codes = [row["code"] for row in generated["rows"]]
    gen_map = {row["code"]: row for row in generated["rows"]}
    own_map = overlay.get("codes", {})
    if overlay.get("version") != 1:
        violations.append("overlay version must be 1")
    extra = sorted(set(own_map) - set(gen_codes))
    missing = sorted(set(gen_codes) - set(own_map))
    if extra:
        violations.append(f"overlay has unknown codes: {extra}")
    if missing:
        violations.append(f"overlay missing codes: {missing}")
    for code in sorted(set(gen_codes) & set(own_map)):
        row = gen_map[code]
        owned = own_map[code]
        for key in REQUIRED:
            if key not in owned:
                violations.append(f"{code}: missing field {key}")
        severity = owned.get("severity")
        if severity not in ALLOWED_SEVERITY:
            violations.append(f"{code}: invalid severity {severity!r}")
        pii = owned.get("pii_class")
        if pii not in ALLOWED_PII:
            violations.append(f"{code}: invalid pii_class {pii!r}")
        if not isinstance(owned.get("pages_oncall"), bool):
            violations.append(f"{code}: pages_oncall must be a boolean")
        steps = owned.get("runbook_steps")
        if not isinstance(steps, list) or any(not isinstance(s, str) or not s.strip() for s in steps):
            violations.append(f"{code}: runbook_steps must be a list of non-empty strings")
        elif owned.get("pages_oncall") is True and len(steps) < 1:
            violations.append(f"{code}: paging codes require at least one runbook step")
        copy = owned.get("customer_copy", "")
        if not isinstance(copy, str) or len(copy.strip()) < 12:
            violations.append(f"{code}: customer_copy is too short")
        elif copy.strip() == row["template"]:
            violations.append(f"{code}: customer_copy reprints the log template")
        elif "{" in copy or "}" in copy:
            violations.append(f"{code}: customer_copy still contains template placeholders")
        if pii == "none" and "{" in row["template"]:
            violations.append(f"{code}: template has placeholders but pii_class is none")
    return violations


def main() -> int:
    violations = join_errors(load_json(GENERATED), load_json(OVERLAY))
    if violations:
        for item in violations:
            print(f"join_error_docs: {item}", file=sys.stderr)
        return 1
    print(f"join_error_docs: ok ({len(load_json(GENERATED)['rows'])} codes)")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Step 5 — Optional model draft of technical notes only

A free model can draft technical notes from comments after the index exists, which is the only model-shaped step in this workflow. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode free-model access can draft those labeled technical notes after the compiler writes the index. The free server option can run the compiler and the join-gate as an ordinary documentation job.

Do not send customer copy, runbooks, or production identifiers to the model during that draft. Related-code suggestions belong in a side file that the overlay must acknowledge before publish. If the draft file lacks an explicit draft: true header, treat it as unlabeled model output and fail the docs job. The overlay remains the only path for severity and paging, including when the draft sounds confident.

Step 6 — Block merge on a stale generated index

Wire three commands into the documentation pipeline, in this order, and keep them off the application deploy path. First recompile or --check the index so hand edits cannot hide inside generated JSON. Then run the join-gate so a new enum member cannot ship without overlay review. Then publish Markdown from the join of both files, never from a chat transcript that paraphrased the enum.

python scripts/compile_error_index.py --check
python scripts/join_error_docs.py

A proposed CODEOWNERS rule should require the incident or support owner on overlay edits, not on the generated index. Generated files can be owned by the compiler path, which reduces review noise without hiding judgment changes. If your host runs docs jobs on a shared queue, still keep overlay contents out of prompt logs. The join-gate is cheap relative to an on-call rotation that discovers an undocumented code during a page.

Join-gate tests

The tests below are labeled fixtures and do not measure a production catalog. They encode the contract: missing overlay rows fail, and paging without steps fails. Run them with pytest tests/test_error_doc_join.py after the scripts are importable. Extend the matrix when you add a new judgment field, because an untested field will be skipped under time pressure.

# tests/test_error_doc_join.py
from copy import deepcopy

from scripts.join_error_docs import join_errors

GENERATED = {
    "row_count": 1,
    "source_path": "app/errors.py",
    "rows": [
        {
            "code": "PAYMENT_GATEWAY_TIMEOUT",
            "defining_class": "AppError",
            "lineno": 12,
            "source_path": "app/errors.py",
            "symbol": "PAYMENT_GATEWAY_TIMEOUT",
            "template": "gateway timeout on charge_id={charge_id}",
        }
    ],
}

OWNED = {
    "severity": "SEV-2",
    "pages_oncall": True,
    "pii_class": "payment_identifier",
    "customer_copy": "A payment processor timed out. No extra charge was created from this retry.",
    "runbook_steps": ["Compare gateway success rate against the five-minute baseline dashboard."],
    "related_codes_ack": [],
}


def test_join_ok_when_overlay_covers_generated_code():
    overlay = {"version": 1, "codes": {"PAYMENT_GATEWAY_TIMEOUT": deepcopy(OWNED)}}
    assert join_errors(GENERATED, overlay) == []


def test_join_fails_when_overlay_omits_a_generated_code():
    overlay = {"version": 1, "codes": {}}
    violations = join_errors(GENERATED, overlay)
    assert any("missing codes" in item for item in violations)


def test_join_fails_when_paging_code_has_no_runbook():
    owned = deepcopy(OWNED)
    owned["runbook_steps"] = []
    overlay = {"version": 1, "codes": {"PAYMENT_GATEWAY_TIMEOUT": owned}}
    violations = join_errors(GENERATED, overlay)
    assert any("paging codes require" in item for item in violations)


def test_join_fails_when_customer_copy_reprints_template():
    owned = deepcopy(OWNED)
    owned["customer_copy"] = "gateway timeout on charge_id={charge_id}"
    overlay = {"version": 1, "codes": {"PAYMENT_GATEWAY_TIMEOUT": owned}}
    violations = join_errors(GENERATED, overlay)
    assert any("reprints the log template" in item or "placeholders" in item for item in violations)

Limitations and who should not use this

This join-gate proves coverage, not correctness of severity, and it will not detect a wrong SEV-2 that a human typed in haste. It also assumes one enum module with literal ErrorDef rows, which excludes free-text logger calls and codes minted by a remote vendor at runtime. Multi-language repositories need one walker per catalog of record, plus a single overlay so support does not keep two truths. Customer copy still needs review when a status page or email is a legal-adjacent surface in your market.

Teams without stable error enums should not adopt this overlay, because the compiler will emit an empty index and fail closed. Chat logs are not a catalog of record, and fluent severity is not evidence that a code should page. Security advisories, regulated device labeling, and abuse-report language should stay outside generated notes, even when a free model is available for technical drafts. If the on-call process is a wiki with no owners, the overlay will not create ownership by existing as a file.

The join-gate is the durable output of this workflow; the model draft is optional scaffolding around extractable facts. Compile every shipped code, require a human overlay for judgment fields, and fail the docs build when those sets diverge. If you already generate API docs from source, apply the same join to error codes before the next incident review, using a free model only for labeled technical notes.