Keep Severity Out of Model-Drafted Troubleshooting Pages

Error documentation fails when a model is allowed to invent severity, data-loss risk, or an authorized fix. The useful split is simple: models may draft symptom prose from reviewed facts, while humans own every remediation claim. This article gives a catalog format, a classifier script, and a review gate you can run before any draft leaves the repository. The method stays useful even if you later swap the drafting tool, because the ownership boundary lives in files rather than in a prompt.

Why generated troubleshooting pages drift

A readable troubleshooting page can still authorize the wrong action when its severity line was never reviewed by on-call. Rewriting a sentence can improve readability while quietly changing who is allowed to restart a service or notify a customer. Teams then ship a polished runbook whose severity label was never reviewed by the person who carries the pager. That gap is a documentation ownership problem, not a prompt-quality problem, and the catalog should encode it.

Three failure modes show up often enough in doc reviews to treat them as design constraints rather than personal anecdotes. First, the model copies an old mitigation that no longer matches the current flag set or deploy topology. Second, it upgrades a warning into an incident because the prompt asked for a confident tone. Third, it fills a missing field with a plausible customer impact that nobody on the team actually measured. A catalog that marks those fields as human-owned stops the draft step from touching them at all.

What each role may touch

The catalog below is a proposal for repositories that already keep machine-readable error records under version control. It is not a claim that any particular product measured these classes against a set of production incidents. Use it as a contract between the people who know the system and the tool that only writes prose.

Field group	Examples	Model may draft	Human must own
Symptom facts	error code, log key, introduced version	Yes, from the catalog only	Confirm the facts file
Explanatory prose	what the message means, where to look	Yes	Edit for accuracy
Contractual claims	severity, data-loss flag, customer notice	No	Sign before publish
Operational authority	restart, rollback, secret handling	No	Sign with a name and date

Numbered ownership rules keep the later script honest.

1. A model may paraphrase fields that already exist in the reviewed catalog, and it may not add a field.
2. A model may suggest extra checks only when they are labeled as unverified candidates, not as required steps.
3. A human must set severity, data-loss risk, customer communication, and any action that changes production state.
4. A human must reject a draft that mentions a secret value, an internal hostname, or a credential path.

Step 1: Keep facts in a catalog, not in chat

Store one record per error code in YAML, and keep narrative sentences out of the structured fact fields. The example below is illustrative and has not been executed against a live service in this article. Replace the codes with your own before you treat the file as a source of truth. Leave severity and authorized actions in the human-owned block so a later script can refuse an unsigned page.

# errors/catalog.yaml — illustrative, not a production incident record
errors:
  - code: E_QUEUE_LAG
    log_key: queue.lag_seconds
    introduced: "2024-11-02"
    symptom_facts:
      - "Consumer lag is exported as queue.lag_seconds."
      - "The page fires when lag stays above the configured threshold."
    candidate_checks:
      - "Compare lag with the last successful deploy time."
      - "Confirm the consumer process is running in the target environment."
    human_owned:
      severity: "page"
      data_loss: false
      customer_notice: "required if lag exceeds the agreed window"
      authorized_actions:
        - "Scale the consumer only after the on-call lead approves."
      owner: "unassigned"
      signed_on: null

Do not put real hostnames, tokens, or customer identifiers in this file or in its comments. The classifier will later refuse to emit a draft packet if those patterns appear, but a regex is not a complete secret scanner. Treat the refusal as a review gate, not as proof that the repository is free of secrets. Pair it with your existing secret scanner before you trust a green result from this script alone.

Step 2: Classify fields before any model sees them

The script reads the catalog and writes two artifacts, a draft packet and a separate signature packet. Draft packets contain symptoms and candidate checks only, while signature packets hold severity, data-loss, notices, and authorized actions. Signature packets should stay in the review tool rather than in any model context window or chat log. The script also fails closed if a human-owned field is empty, null, or still marked unassigned.

#!/usr/bin/env python3
"""Propose draft vs signature packets. Not a measured benchmark."""

from __future__ import annotations

import re
import sys
from pathlib import Path

import yaml

SECRETISH = re.compile(
    r"(?i)(api[_-]?key|secret|password|token|BEGIN PRIVATE|aws_)"
)
REQUIRED_HUMAN = (
    "severity",
    "data_loss",
    "customer_notice",
    "authorized_actions",
    "owner",
    "signed_on",
)


def load_catalog(path: Path) -> dict:
    data = yaml.safe_load(path.read_text(encoding="utf-8"))
    if not isinstance(data, dict) or "errors" not in data:
        raise SystemExit("catalog must be a mapping with an errors list")
    return data


def classify(entry: dict) -> tuple[dict, dict, list[str]]:
    code = entry.get("code", "")
    blob = yaml.safe_dump(entry)
    blockers: list[str] = []
    if SECRETISH.search(blob):
        blockers.append(f"{code}: possible secret material")
    human = entry.get("human_owned") or {}
    for key in REQUIRED_HUMAN:
        if key not in human or human[key] in (None, "", "unassigned", []):
            blockers.append(f"{code}: unsigned human field {key}")
    draft = {
        "code": code,
        "log_key": entry.get("log_key"),
        "introduced": entry.get("introduced"),
        "symptom_facts": entry.get("symptom_facts") or [],
        "candidate_checks": entry.get("candidate_checks") or [],
        "draft_rules": [
            "Paraphrase only these fields.",
            "Label every extra check as unverified.",
            "Do not state severity, data loss, or a production action.",
        ],
    }
    signature = {"code": code, "human_owned": human}
    return draft, signature, blockers


def main() -> int:
    catalog = load_catalog(Path(sys.argv[1]))
    drafts, signatures, blockers = [], [], []
    for entry in catalog["errors"]:
        draft, signature, problems = classify(entry)
        drafts.append(draft)
        signatures.append(signature)
        blockers.extend(problems)
    out = Path(sys.argv[2])
    out.mkdir(parents=True, exist_ok=True)
    (out / "signature_packet.yaml").write_text(
        yaml.safe_dump({"signatures": signatures}, sort_keys=False),
        encoding="utf-8",
    )
    if blockers:
        print("\n".join(blockers), file=sys.stderr)
        return 1
    (out / "draft_packet.yaml").write_text(
        yaml.safe_dump({"drafts": drafts}, sort_keys=False),
        encoding="utf-8",
    )
    print(f"wrote {len(drafts)} draft packets; signature file stays local")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Run the classifier from the repository root, and keep the signature file out of the prompt you send for drafting. The commands below assume Python 3.11 or newer and a local virtual environment on the developer machine. They do not call a network service, and they do not publish documentation to any external site by themselves. The first run should fail the draft-packet presence test until a reviewer signs the human-owned fields.

python3 -m venv .venv
.venv/bin/pip install pyyaml
.venv/bin/python scripts/split_error_catalog.py errors/catalog.yaml build/doc-packets
test -s build/doc-packets/signature_packet.yaml
test -s build/doc-packets/draft_packet.yaml

Expect a non-zero exit while the owner field is unassigned or the signed_on field is still null. That failure is the point of the gate, not a defect in YAML parsing or file permissions. The draft packet is withheld until those fields are signed, while the signature packet is still written for review. Assign a real reviewer, set a date after they read the authorized actions, and rerun the script before you request prose.

Step 3: Let a model draft only the packet you already classified

Send only draft_packet.yaml to a drafting model, and ask for one short page per error code. The page may restate symptoms in plain language and must list every candidate check as still unverified. It must not invent severity, downtime, data loss, or any command that mutates production state or data. If the model adds those claims, discard the page and tighten the packet rather than editing the invention into the docs.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access can host that drafting step when a local model is unnecessary for symptom prose. The free server option, as supplied for this draft, is a fit when you do not want to administer the machine that runs the drafting pass. This article does not state model names, quotas, hardware, duration, or permanence, because those details were not supplied as verified facts.

Keep signature_packet.yaml on the review side either way, including when the draft itself is produced on a hosted drafting pass. A minimal drafting instruction can live next to the packet so reviewers can see the constraint. It is a prompt skeleton, not evidence that any particular model obeyed the constraint in a test. Reviewers should diff the resulting page against the packet before they read it as finished documentation.

Using only draft_packet.yaml, write one troubleshooting section per code.
Paraphrase symptom_facts. Mark candidate_checks as unverified.
Do not mention severity, data loss, customers, restarts, or rollbacks.
If a fact is missing, write "not in catalog" instead of guessing.

Step 4: Humans sign remediation before the page is merged

Open the signature packet beside the drafted page, and do not merge until every human-owned field has a named owner and a date. The reviewer checks four questions, in order, and records each answer in the pull request body. This review is the product, and the drafted prose is only the readable layer on top of it. A missing signature is a failed review, even when the generated sentences sound specific and calm.

1. Does every symptom sentence trace to a field in the draft packet, with no added cause or hidden assumption?
2. Are candidate checks still labeled unverified, or did someone promote them without a supporting log line?
3. Do severity, data-loss, and customer notice match the signature packet exactly, including values left blank?
4. Is each authorized action something the named owner is allowed to approve on the current on-call rota?

If any answer is no, fix the catalog or the signature packet first, then regenerate the draft. Editing only the Markdown hides the drift until the next incident, when the page and the catalog disagree. A small diff of signature_packet.yaml in the same pull request makes that disagreement visible to reviewers who never open the model transcript. Reject pages that cite a severity the signature packet does not contain, even if the wording is clearer.

Limitations and who should skip this

The regex for secret-like strings misses encoded credentials, private URLs, and secrets that are split across fields. It also flags harmless words such as token bucket unless you tune the pattern for your vocabulary. The script does not prove that a fact is true; it only proves that a required human field was filled. A wrong severity that a human signed is still wrong, and this workflow will publish it without further protest.

Skip this approach when the error catalog is not the source of truth, or when incidents are still tracked only in chat logs. Skip it when legal or safety text must be written by counsel rather than assembled from engineering fields. Skip it for one-off blog explanations that have no operational action, because the signature gate adds process without reducing risk. Teams without a named on-call owner will only collect unsigned packets, and the classifier will keep failing until that role exists.

What to keep after the draft exists

Store the catalog, both packets, and the drafted page in version control, and treat the signature packet as the authority when they conflict. Regenerate the prose when catalog facts change, and require a new signature whenever authorized actions change. A drafting pass can be repeated whenever the draft packet changes, but it should not be treated as a standing approval. Leave the signature file in human review, and publish the page only after the four questions above all pass.