Compile Error Catalogs From Raises Clauses; Humans Sign Severity and Response

Generated troubleshooting pages fail when they mix extractable error facts with unsigned operational promises in one file. A repository already contains exception classes, raises clauses, and stable log event identifiers that scripts can list. A language model may draft customer-facing sentences from that list, but it must not invent severity, paging rules, or recovery clocks. Maintainers should merge documentation only after those operational fields carry an explicit human signature in review.

Separate extractable facts from signed operational copy

Public error documentation usually collapses four different statement classes into one Markdown troubleshooting section for readers. Extractable facts include type names, constructor fields, and log event identifiers that already exist in source. Draftable prose includes short user-visible explanations that a model can propose from those facts alone. Signed operational copy includes severity, on-call paging, legal recovery language, and any calendar date for removal.

Mixing those classes produces pages that look complete while remaining unreviewable as a change. Reviewers cannot tell whether an SLA phrase came from a runbook or from a fluent draft. Readers treat unsigned recovery clocks as contract, then file incidents against documentation that nobody signed. The working response is a field-level ownership map, not a better prompt.

The table below is the reproducible artifact for this workflow, not a writing-style preference. Every troubleshooting page should declare which columns were extracted, drafted, or signed before merge. Continuous integration should reject a published path that still contains UNSIGNED in a human-owned column. That gate is mechanical and does not depend on subjective model quality scores.

Field	Allowed source	Model may draft?	Human must sign?
Exception class name	AST / compiler	No, extract only	No
Constructor fields	AST	No, extract only	No
Log event id	Literal string constants	No, extract only	No
HTTP or RPC mapping	Routing tables	Only if missing	Yes if public
Customer-visible summary	Catalog rows	Yes	Yes
Severity (sev1–sev4)	Incident taxonomy	No	Yes
Page on-call	Roster owner	No	Yes
Remediation steps	Runbook path	Proposal only	Yes
Removal or freeze date	Release owners	No	Yes

Step 1. Extract raises clauses and literal log event identifiers

Start from the current commit rather than from a chat transcript about an earlier API shape. Walk Python modules that you already mark as public, and collect Raise nodes plus logger calls with literal event names. Persist the result as JSON so later stages never re-parse narrative Markdown for ground truth. Treat the following script as a labeled worked example, not as a measured production service.

# example: extract_error_catalog.py
"""Worked example: emit extractable error facts from a Python tree."""
from __future__ import annotations

import ast
import json
import pathlib
import sys
from typing import Any

LOG_METHODS = {"debug", "info", "warning", "error", "exception", "critical"}


class ErrorCatalogVisitor(ast.NodeVisitor):
    def __init__(self, module: str) -> None:
        self.module = module
        self.functions: list[dict[str, Any]] = []
        self._current: dict[str, Any] | None = None

    def visit_FunctionDef(self, node: ast.FunctionDef) -> None:
        if node.name.startswith("_"):
            self.generic_visit(node)
            return
        record = {
            "module": self.module,
            "qualname": node.name,
            "lineno": node.lineno,
            "raises": [],
            "log_events": [],
        }
        prev, self._current = self._current, record
        self.generic_visit(node)
        self.functions.append(record)
        self._current = prev

    visit_AsyncFunctionDef = visit_FunctionDef

    def visit_Raise(self, node: ast.Raise) -> None:
        if self._current is None or node.exc is None:
            return
        name = self._exc_name(node.exc)
        if name:
            self._current["raises"].append(name)

    def visit_Call(self, node: ast.Call) -> None:
        if self._current is None:
            self.generic_visit(node)
            return
        func = node.func
        if isinstance(func, ast.Attribute) and func.attr in LOG_METHODS:
            if node.args and isinstance(node.args[0], ast.Constant):
                value = node.args[0].value
                if isinstance(value, str) and value.startswith("evt."):
                    self._current["log_events"].append(value)
        self.generic_visit(node)

    @staticmethod
    def _exc_name(exc: ast.expr) -> str | None:
        if isinstance(exc, ast.Name):
            return exc.id
        if isinstance(exc, ast.Call) and isinstance(exc.func, ast.Name):
            return exc.func.id
        return None


def extract(root: pathlib.Path) -> list[dict[str, Any]]:
    rows: list[dict[str, Any]] = []
    for path in root.rglob("*.py"):
        if "tests" in path.parts:
            continue
        module = path.with_suffix("").as_posix().replace("/", ".")
        tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path))
        visitor = ErrorCatalogVisitor(module)
        visitor.visit(tree)
        rows.extend(visitor.functions)
    return [row for row in rows if row["raises"] or row["log_events"]]


if __name__ == "__main__":
    target = pathlib.Path(sys.argv[1] if len(sys.argv) > 1 else "src")
    json.dump(extract(target), sys.stdout, indent=2)

Execute the extractor against the package root and store JSON beside the docs tree.

python extract_error_catalog.py src > docs/_generated/error_catalog.json

The extractor will miss exceptions built through helpers, formatted log lines, and errors raised from generated stubs. That incompleteness is expected, and it is precisely why signed columns exist at all. Do not compensate by asking a model to guess missing type names from nearby prose. Re-run extraction on the merge commit whenever Python sources change in the same pull request.

Step 2. Render stubs that keep operational fields explicitly UNSIGNED

Do not paste the JSON into a blog-like README and declare the troubleshooting page finished. Render a stub that copies extractable fields verbatim and leaves operational fields marked UNSIGNED for review. The stub is the only documentation file the optional drafting stage may edit, and only in the customer-summary column. Keep severity, paging, remediation, and removal dates out of that drafting diff.

# example: render_error_stub.py
"""Worked example: turn catalog rows into unsigned troubleshooting stubs."""
from __future__ import annotations

import json
import pathlib
import textwrap

STUB = """\\
## `{qualname}`

- Exception types (extracted): {raises}
- Log events (extracted): {events}
- Customer summary (draftable): UNSIGNED
- Severity (human-owned): UNSIGNED
- Page on-call (human-owned): UNSIGNED
- Remediation (human-owned): UNSIGNED
- Removal date (human-owned): UNSIGNED
"""


def render(catalog_path: pathlib.Path, out_dir: pathlib.Path) -> None:
    rows = json.loads(catalog_path.read_text(encoding="utf-8"))
    out_dir.mkdir(parents=True, exist_ok=True)
    lines = [
        "# Troubleshooting catalog",
        "",
        "Status fields marked UNSIGNED must not ship.",
        "",
    ]
    for row in rows:
        block = STUB.format(
            qualname=row["qualname"],
            raises=", ".join(row["raises"]) or "(none)",
            events=", ".join(row["log_events"]) or "(none)",
        )
        lines.append(textwrap.dedent(block))
    (out_dir / "TROUBLESHOOTING.md").write_text("\n".join(lines), encoding="utf-8")

A later check can be a short shell gate in CI rather than a separate documentation product. Fail the build when UNSIGNED remains on any path you publish to users. Allow the token only on draft branches, never on the default-branch documentation set that customers read.

# example CI fragment
if git grep -n "UNSIGNED" -- docs/TROUBLESHOOTING.md; then
  echo "signed operational fields still unmarked" >&2
  exit 1
fi

Step 3. Draft customer summaries without touching human-owned fields

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

After the catalog exists, a drafting pass is useful only for the customer-visible summary column on each row. MonkeyCode's free model access and free server option can host that pass without placing severity or paging text in the same prompt. Feed the extractor JSON, forbid new exception names, and require the model to leave every human-owned field untouched. Treat returned summaries as proposals until a maintainer replaces UNSIGNED with signed values in review.

A prompt that stays inside the draftable column looks like the following template block. It is an unexecuted template, not a claim about a named model, quota, or hardware profile.

You receive error_catalog.json.
Write one customer summary per qualname, maximum 25 words.
Do not add exception types, log events, severity, paging, dates, or SLAs.
Leave every human-owned field exactly as UNSIGNED.

If the draft introduces a type that is absent from the JSON, discard the entire documentation patch immediately. That rejection is cheaper than debating whether the extra type merely sounds plausible to readers. Re-run extraction on the merge commit if the same patch also touched Python sources. Never let the drafting host rewrite the JSON catalog to match its own prose.

Step 4. Sign severity, paging, and remediation during review

Reviewers should not re-litigate extracted class names unless the extractor itself is demonstrably wrong. Their job is to fill severity, paging, remediation, and any removal date from runbooks they already trust. If no runbook exists, the correct merge action is to keep UNSIGNED and block publish, not to invent a clock. Record the signer as a username and date on the pull request, not inside generated JSON blobs.

Use this numbered review checklist so the signature step stays consistent across reviewers:

1. Confirm each exception name still exists in the merge-commit tree with a search.
2. Reject any customer summary that states a time-to-recovery, refund rule, or uptime percentage.
3. Copy severity from the team's existing incident taxonomy, never from the tone of the summary.
4. Set page-on-call to yes, no, or business-hours using the roster owner, not the model.
5. Paste remediation from a runbook path, or leave the field UNSIGNED and stop the publish.

Step 5. Add a unit test for the publish gate

A shell grep is enough for small trees, but a unit test documents the rule next to the extractor. The following test is a worked example that reads a rendered page and fails on leftover UNSIGNED tokens. Point it at the published path only, so draft branches can still carry unsigned stubs. Do not assert word counts or quality scores; assert ownership markers.

# example: test_publish_gate.py
"""Worked example: published troubleshooting pages must not contain UNSIGNED."""
from pathlib import Path

PUBLISHED = Path("docs/TROUBLESHOOTING.md")


def test_published_troubleshooting_has_no_unsigned_fields() -> None:
    text = PUBLISHED.read_text(encoding="utf-8")
    assert "UNSIGNED" not in text, (
        "human-owned fields still unsigned on the publish path"
    )

Wire the test to the job that builds the default-branch docs, not to every draft preview. A failing assertion then means a human-owned column shipped without a signature, which is the defect this workflow exists to catch. Passing does not prove the severity values are operationally correct; it only proves they are no longer placeholders.

Limitations

This workflow does not discover errors that never appear as raise or as a literal evt.* log string. Wrappers, protocol buffers, and HTTP problem-details documents need dedicated extractors before they enter the catalog. Dynamic messages built with f-strings will not appear, which is preferable to signing guessed identifiers in public docs. Teams that publish legally binding availability numbers need a counsel-owned source, not a Markdown stub rendered from AST.

The optional drafting pass can still write confident nonsense inside a twenty-five-word customer summary. Bounded length reduces blast radius; it does not create ground truth for operators. Human signature remains the publish gate even when summaries look dull and conservative. If extraction coverage is thin, expand the parser rather than widening the model's permission to invent types.

Who should not use this approach

Do not adopt this split if your troubleshooting corpus already comes from an incident database with signed severity fields. Safety-critical systems that require hazard analysis should not treat AST output as the hazard list for certification. Authors who want one-shot write-the-docs-from-the-repo prompts will fight the UNSIGNED gate and should pick another process. Small libraries with three public errors can maintain the ownership table by hand and skip the extractor entirely.

Keep generated error catalogs next to signed operational copy, and refuse default-branch merges while severity still says UNSIGNED. If you already extract catalogs this way, a free-server drafting session is enough to fill the customer-summary column and nothing else.