Treat Troubleshooting Docs as Two Files: an Extracted Fault Catalog and a Signed Recovery Matrix

Generated troubleshooting pages fail in review when draft descriptions and user-facing outcomes occupy the same markdown file. A catalog of exception types, status codes, and log event names can be extracted, then drafted, without promising any user outcome. Recovery language, retry guidance, and data-loss statements remain a human-owned matrix that continuous integration can refuse to publish unsigned. The remainder of this article specifies a two-file pipeline, a YAML schema, and a test that enforces the split.

Mixed ownership, not missing prose, is the failure mode

Most teams regenerate troubleshooting sections from a single prompt over the repository or from unreviewed chat output. That process mixes three statement kinds that do not share a source of truth: mechanical inventory, plausible description, and operational promise. Inventory can be compiled from raise sites and error handlers, while description can be drafted and later edited. Operational promise cannot be inferred from a stack trace, an HTTP mapping, or a short model summary.

When those three statement kinds share one file, reviewers cannot tell derived sentences from product warranties. A regenerated paragraph can change retry advice into a rollback claim without any test noticing the swap. The published page then disagrees with the transaction boundary that the running service actually implements. Splitting the catalog from the recovery matrix turns that disagreement into a build failure instead of a reading exercise.

Ownership matrix for error-path documentation

Use the following table as the publishing contract for every troubleshooting page that will ship to users. Rows a model may draft still require a human to accept or rewrite the sentence before merge. Rows marked human-owned must never be emitted into the generated catalog file at all. Cheap CI checks cover only a subset; refunds and availability claims still need a named owner.

Statement class	Source of truth	Model may draft?	Human must own	Publish gate
Exception class inventory	AST or raise sites	No; extract only	Confirm coverage	Fail if source fault is unsigned
HTTP status mapping	Framework error handlers	No; extract only	Confirm mapping	Fail if a mapped status is missing
One-sentence mechanical description	Extracted name plus handler	Yes; draft only	Edit for accuracy	Fail if reviewed is not true
Retryable or idempotent?	Transaction and queue code	No	Yes	Fail if the field is missing
Data committed or discarded	Persistence boundary	No	Yes	Fail if the field is missing
User-visible recovery steps	Product and support policy	No	Yes	Fail if the field is missing
Support window or SLA language	Contract, not code	No	Yes	Fail if present in generated files

The table is the policy artifact, and the scripts below enforce only those checks that remain cheap in CI. Dynamic exceptions, multi-language services, and dependency faults will not appear until extractors are extended. Banned-verb filters are not proof of safety; they only stop the most obvious outcome language in drafts.

Workflow: extract, draft, sign, then render

Keep the four stages serial so a draft can never publish itself as a recovery instruction. Each stage writes or updates one file, and later stages refuse to run when earlier files are incomplete. The commands assume a Python service tree; adapt the extractor if handlers live in another language.

1. Extract a fault catalog from source, not from chat

Run a deterministic extractor over the service package and write faults.generated.yaml before any model runs. Do not ask a model to list errors, because listing is a compile step with a verifiable source. Keep the generated file out of the published site until every description is reviewed and every fault has a signed recovery row.

# extract_faults.py — proposed extractor, not a production-grade scanner
import ast
import pathlib
import sys
import yaml

ROOT = pathlib.Path(sys.argv[1])
rows = []

class RaiseVisitor(ast.NodeVisitor):
    def __init__(self, path):
        self.path = path
        self.found = []

    def visit_Raise(self, node):
        name = None
        if isinstance(node.exc, ast.Call) and isinstance(node.exc.func, ast.Name):
            name = node.exc.func.id
        elif isinstance(node.exc, ast.Name):
            name = node.exc.id
        if name:
            self.found.append({
                "fault_id": name,
                "path": str(self.path),
                "lineno": node.lineno,
                "draft_description": "",
                "reviewed": False,
            })
        self.generic_visit(node)

for py in ROOT.rglob("*.py"):
    if "tests" in py.parts or py.name.startswith("test_"):
        continue
    tree = ast.parse(py.read_text(encoding="utf-8"), filename=str(py))
    visitor = RaiseVisitor(py)
    visitor.visit(tree)
    rows.extend(visitor.found)

seen = set()
unique = []
for row in rows:
    if row["fault_id"] in seen:
        continue
    seen.add(row["fault_id"])
    unique.append(row)

yaml.safe_dump({"faults": unique}, open("faults.generated.yaml", "w"), sort_keys=False)

python extract_faults.py ./src

The extractor records locations and empty description fields; it does not invent HTTP mappings, retry policy, or user guidance. Label extra heuristics, such as status-code guesses from nearby integers, as proposals until they are checked against real handlers. If your service raises faults from a shared errors package, point the extractor at that package rather than at every caller.

2. Draft descriptions on a disposable lane

Feed only the generated catalog, with secret-bearing source removed, into a drafting job that fills draft_description. Limit each draft to one mechanical sentence that restates the fault name and the raising module. The job must not write retry guidance, rollback claims, support hours, or any sentence that tells a user what happened to their data.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. A drafting job of this kind can run against MonkeyCode's free model access on the free server option. That placement keeps the catalog off a workstation GPU and off a billed API key. Do not send production connection strings, customer traces, or unsigned legal text into that lane.

After the job returns, a reviewer sets reviewed: true only when the sentence matches the raise site. Unreviewed descriptions stay unpublished even if the YAML file is committed for inspection. Treat the lane as a draft printer, not as an owner of user outcomes, and do not store customer incidents there.

3. Maintain a human-owned recovery matrix

Create recovery.signed.yaml by hand and keep it out of any regenerate script. Every fault_id in the generated catalog must have a matching signed row before a troubleshooting page may render. The dates below are sample signatures for this article, not production metrics or measured incident rates.

# recovery.signed.yaml — human-owned; never regenerated
version: 1
rows:
  - fault_id: PaymentCaptureError
    retryable: false
    data_outcome: "capture request not committed; wallet balance unchanged"
    user_recovery: "Re-open checkout and submit once; do not retry in a loop."
    signer: "payments-oncall"
    signed_at: "2026-09-20"
  - fault_id: InventoryReservationTimeout
    retryable: true
    data_outcome: "reservation lease expired; no stock decrement"
    user_recovery: "Retry after 5 seconds; abandon after three attempts."
    signer: "inventory-oncall"
    signed_at: "2026-09-20"

Replace signers with the team that owns the transaction boundary, not with the person who ran the extractor. If a fault has no owner, it has no published recovery language and the page should omit it. Adding a plausible recovery sentence without a signer is the failure mode this pipeline exists to block.

4. Gate publish with a coverage test, then render without a model

The tests below fail when source contains an extracted fault that lacks a signed recovery row, or when a generated description is still unreviewed. They also fail when the generated file contains banned outcome verbs that belong only in the signed matrix.

# test_fault_docs.py — proposed CI gate
import pathlib
import re

import pytest
import yaml

BANNED_IN_GENERATED = re.compile(
    r"\b(retry|roll(?:ed)? back|data loss|SLA|uptime|never lose)\b",
    re.I,
)


def test_every_extracted_fault_has_a_signed_recovery():
    generated = yaml.safe_load(pathlib.Path("faults.generated.yaml").read_text())
    signed = yaml.safe_load(pathlib.Path("recovery.signed.yaml").read_text())
    signed_ids = {row["fault_id"] for row in signed["rows"]}
    missing = [
        f["fault_id"]
        for f in generated["faults"]
        if f["fault_id"] not in signed_ids
    ]
    assert missing == [], f"unsigned faults: {missing}"


def test_generated_descriptions_are_reviewed_and_non_promising():
    generated = yaml.safe_load(pathlib.Path("faults.generated.yaml").read_text())
    for fault in generated["faults"]:
        assert fault.get("reviewed") is True, fault["fault_id"]
        text = fault.get("draft_description") or ""
        assert not BANNED_IN_GENERATED.search(text), fault["fault_id"]


def test_signed_rows_name_outcome_and_recovery():
    signed = yaml.safe_load(pathlib.Path("recovery.signed.yaml").read_text())
    for row in signed["rows"]:
        assert row.get("data_outcome"), row["fault_id"]
        assert row.get("user_recovery"), row["fault_id"]
        assert row.get("signer"), row["fault_id"]
        assert isinstance(row.get("retryable"), bool), row["fault_id"]

python -m pytest test_fault_docs.py -q

Render markdown only after those tests pass, and keep the renderer free of model calls. Concatenate the reviewed description with the signed recovery columns using a small template, then commit the rendered page as a build output rather than as a chat paste.

# proposed render contract — not an executed benchmark
for fault in reviewed_catalog:
    rec = signed_index[fault.fault_id]
    write("## " + fault.fault_id)
    write(fault.draft_description)
    write("- Retryable: " + rec.retryable)
    write("- Data outcome: " + rec.data_outcome)
    write("- What to do: " + rec.user_recovery)

Reviewer checklist before merge

1. Confirm each fault_id still exists at the recorded path, or delete the catalog row and the signed row together.
2. Confirm retryable matches the queue or HTTP idempotency rules in the handler, not the draft sentence.
3. Confirm data_outcome matches the persistence boundary, including partial writes and lease expiry.
4. Reject any generated description that tells the user to retry, wait on an SLA, or assume a rollback.

A reviewer who cannot name the transaction boundary should leave reviewed false and leave the recovery row unsigned. Unpublished faults are cheaper than a troubleshooting page that invents a rollback the ledger never performed. The checklist is the human counterpart to the pytest file, not a substitute for it.

Limitations and who should not use this

The extractor will miss dynamically constructed exceptions, errors raised in other languages, and faults that exist only inside a dependency. The banned-verb regex is a coarse filter, not a proof that a description is safe to show a customer. The recovery matrix can still be wrong if the signer misunderstands isolation, leases, or outbox delivery.

Do not use this approach for incident reports that include customer data, for security advisories, or for legally binding availability claims. Those documents need counsel and an incident commander, not a draft lane. Teams that lack a named owner per fault should keep troubleshooting pages unpublished rather than auto-filling recovery steps from a model.

Closing check

Before the next release, extract the fault catalog, draft descriptions on a throwaway lane, and refuse to publish any row that lacks a signer. The useful output is not more troubleshooting prose; it is a list of faults the team is not yet willing to stand behind.