A Two-Layer Observability Docs Pipeline: Generated Field Atlas, Human-Signed Incident Meaning

Generated documentation fails incident response when field names compile cleanly while meaning, cardinality, and PII stay unsigned. Teams extract log keys from source and still ship runbooks that invent alert owners, retention, and customer impact. This article describes a two-layer docs build: a generated field atlas and a human-signed meaning overlay. The overlay is the contract; missing signatures fail the documentation pipeline the same way missing tests fail CI.

Unsigned meaning is the failure, not a missing field list

Most observability documentation rots because instrumentation changes faster than the narrative pages that explain it. A new order identifier can land in production on Tuesday while the runbook still describes last quarter's correlation path. Chat-generated prose often fills that gap with plausible owners and SLO numbers that nobody on-call has reviewed. The safer split is mechanical extraction of field identity, plus explicit human ownership of incident meaning.

Unsigned PII and cardinality are not editorial nits; they are production-risk cells with a measurable blast radius. A high-cardinality user email in a helpful example line can leak into indexes, traces, and vendor exports. A missing alert owner on a saturation field pages the wrong rotation during a regional timeout. Treat those cells as signed artifacts, not as sentences a model is allowed to invent.

Draft work versus signature work

Treat every observability doc cell as either compile-safe or signature-required before any model writes a sentence. Compile-safe cells come from source and can be regenerated on every commit without editorial judgment. Signature-required cells change incident response, privacy posture, or paging, so they must remain human-owned. The table below is the working contract for this pipeline, not a style preference.

Cell	Source of truth	Model may draft?	Human must sign?	Merge if unsigned?
Field key	Source constants	Extract only	No	Fail extract
Value shape	Source types or registry	Extract only	Confirm on rename	Fail extract
Example log line	Atlas plus redaction rules	Yes, drafts folder only	Redaction review	Fail if PII unknown
One-line description	Name and type	Yes, drafts folder only	Optional edit	Draft only, unpublished
Cardinality class	Traffic shape and index cost	Proposal only	Yes	Fail gate
PII class	Data map and legal policy	Proposal only	Yes	Fail gate
Alert owner	Team roster	No	Yes	Fail gate
Incident meaning	On-call practice	Proposal only	Yes	Fail gate
Retention	Retention policy	No	Yes	Fail gate

Do not let a drafting model write alert owners, retention windows, or PII classes into the published tree. Those values can look fluent inside a generated paragraph and still page the wrong humans. Keep proposals in docs/obs/drafts/ where reviewers can accept or delete them without merging fiction. The gate script later treats an accepted overlay row as the only publishable meaning layer.

Keep generated files and signed files on separate paths

Keep generated files and signed overlay files in separate paths so reviews cannot confuse them. The atlas is a build artifact and should never be hand-edited inside a documentation pull request. The overlay is the only file that may contain owners, PII classes, cardinality, and incident meaning. Draft descriptions live in a third file that CI never publishes until a reviewer copies selected sentences.

docs/obs/
  atlas.generated.json
  overlay.signed.yaml
  drafts/field-blurbs.generated.md
  published/field-reference.md
scripts/
  extract_log_fields.py
  gate_obs_docs.py
  render_field_reference.py
app/
  logging_fields.py

Name generated files with a .generated. marker so code review tooling can hide or lock them. Require overlay changes to go through the same owners who receive the pages those fields describe. If a service has no field registry yet, add the registry before asking any model to explain production logs. Documentation cannot compile field identity that the running program itself refuses to declare in source.

Five steps from constants to a published field reference

Step 1. Extract field identity from a source registry

Require structured logging to declare field keys as named constants or a single registry dictionary. Parsing free-form logger calls produces false positives and trains the atlas to trust unreviewed strings. The extractor below uses the ast module so ordinary refactors still keep keys visible to the docs build. Label this script as a starting pattern; adapt the registry name to the codebase under review.

# app/logging_fields.py
LOG_FIELD_REGISTRY = {
    "http.route": {"type": "string", "kind": "dimension"},
    "http.status_code": {"type": "int", "kind": "dimension"},
    "order.id": {"type": "string", "kind": "id"},
    "user.email": {"type": "string", "kind": "id"},
    "queue.depth": {"type": "int", "kind": "gauge"},
}

#!/usr/bin/env python3
"""Extract LOG_FIELD_REGISTRY from Python sources into an atlas JSON file."""
from __future__ import annotations

import argparse
import ast
import json
from pathlib import Path
from typing import Any

REGISTRY_NAME = "LOG_FIELD_REGISTRY"


class RegistryVisitor(ast.NodeVisitor):
    def __init__(self) -> None:
        self.fields: dict[str, dict[str, Any]] = {}
        self.source_file = ""

    def visit_Assign(self, node: ast.Assign) -> None:
        for target in node.targets:
            if isinstance(target, ast.Name) and target.id == REGISTRY_NAME:
                if not isinstance(node.value, ast.Dict):
                    raise SystemExit(f"{self.source_file}: {REGISTRY_NAME} must be a dict")
                self._load_dict(node.value)
        self.generic_visit(node)

    def _load_dict(self, node: ast.Dict) -> None:
        for key_node, val_node in zip(node.keys, node.values):
            if key_node is None:
                continue
            if not isinstance(key_node, ast.Constant) or not isinstance(key_node.value, str):
                raise SystemExit(f"{self.source_file}: registry keys must be string constants")
            key = key_node.value
            meta: dict[str, Any] = {"key": key, "source": self.source_file}
            if isinstance(val_node, ast.Dict):
                for mk, mv in zip(val_node.keys, val_node.values):
                    if isinstance(mk, ast.Constant) and isinstance(mv, ast.Constant):
                        meta[str(mk.value)] = mv.value
            elif isinstance(val_node, ast.Constant):
                meta["type"] = val_node.value
            self.fields[key] = meta


def extract(src: Path) -> dict[str, Any]:
    visitor = RegistryVisitor()
    for path in sorted(src.rglob("*.py")):
        tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path))
        visitor.source_file = str(path)
        visitor.visit(tree)
    return {
        "registry": REGISTRY_NAME,
        "field_count": len(visitor.fields),
        "fields": visitor.fields,
    }


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--src", type=Path, required=True)
    parser.add_argument("--out", type=Path, required=True)
    args = parser.parse_args()
    atlas = extract(args.src)
    if atlas["field_count"] == 0:
        raise SystemExit("extract produced zero fields; treat as a broken registry")
    args.out.parent.mkdir(parents=True, exist_ok=True)
    args.out.write_text(json.dumps(atlas, indent=2, sort_keys=True) + "\n", encoding="utf-8")
    print(f"wrote {atlas['field_count']} fields to {args.out}")


if __name__ == "__main__":
    main()

Point the extractor at the application package and write the atlas beside the overlay, not inside it. Re-run the command after every registry edit so deleted keys cannot linger in published documentation. A zero-field atlas is a failed extract, not an empty success, and should be treated as a broken registry. Commit hooks can run the same command, but CI remains the source of truth for merge decisions.

python scripts/extract_log_fields.py --src ./app --out docs/obs/atlas.generated.json

Step 2. Write the atlas as a build artifact

Run the extractor in CI so the atlas matches the commit under test, not a laptop checkout. Commit the overlay; do not commit hand-edited copies of the atlas unless your review culture requires a golden snapshot. If you snapshot the atlas, regenerate it and fail on drift rather than accepting silent key deletions. Deleted fields should be treated as incident-documentation bugs, not optional cleanup chores for a later sprint.

git diff --exit-code -- docs/obs/atlas.generated.json
# Proposal only: use this when you snapshot the atlas.
# Prefer regenerating on CI and publishing without a committed JSON file.

Step 3. Draft blurbs from names and types only

A drafting pass can run against the generated atlas when you need first-cut field descriptions from names and types. Disclosure: This article was prepared as part of MonkeyCode's product outreach, which is the only affiliation claimed here. MonkeyCode's free model access and free server option can host that drafting job without folding unsigned meaning into the merge gate. Keep model output in the drafts folder so only overlay signatures become mergeable documentation for on-call readers.

# Proposal: drafting prompt for field blurbs (never for overlay cells)
You receive a JSON atlas of log field keys and types.
Write one short description per key from the name and type alone.
Do not invent owners, PII classes, cardinality, retention, SLOs, or example emails.
Do not mark any description as approved or publishable.

# Proposal: write blurbs into the unpublished drafts path only.
python scripts/draft_field_blurbs.py \
  --atlas docs/obs/atlas.generated.json \
  --out docs/obs/drafts/field-blurbs.generated.md

The draft file is a reading aid for overlay authors, not an input to the renderer. Reviewers may copy a sentence into incident_meaning after they accept the risk cells around it. They may also discard the entire draft when the field name is already clear to the on-call rotation. Unpublished drafts should be grepped for @, Bearer, and account-shaped numeric strings before anyone pastes them.

Step 4. Sign cardinality, PII, owner, and incident meaning

Service owners edit overlay.signed.yaml only, and they leave every other published meaning file untouched. Cardinality is a budget decision: high means the field can explode index cost or break aggregations under load. PII is a policy decision: unknown is not a draft state, it is a failed gate. Incident meaning is one or two sentences that tell the paged engineer why the field exists during a failure.

# docs/obs/overlay.signed.yaml
fields:
  http.route:
    cardinality: low
    pii: none
    owner: platform-sre
    retention: 14d
    incident_meaning: Identifies the matched route when latency rises on one handler.
  http.status_code:
    cardinality: low
    pii: none
    owner: platform-sre
    retention: 14d
    incident_meaning: Separates client errors from dependency failures during error-budget burn.
  order.id:
    cardinality: high
    pii: indirect
    owner: checkout
    retention: 7d
    incident_meaning: Join key to checkout traces; never attach the order payload beside it.
  user.email:
    cardinality: high
    pii: direct
    owner: identity
    retention: 7d
    incident_meaning: Direct identifier; drop from info logs and keep on a redacted error path only.
  queue.depth:
    cardinality: low
    pii: none
    owner: checkout
    retention: 14d
    incident_meaning: Saturation signal for the checkout worker pool before timeout errors appear.

Direct PII rows should not carry example payloads, even when a draft blurb offered a fake address. High-cardinality rows should name a paging owner who can justify index cost, not a shared inbox that nobody reads. Retention must match the actual sink policy rather than a number that looks conventional in a table. If legal review is required, attach this overlay file, not a chat transcript from the drafting step.

Step 5. Gate unsigned high-risk cells, then render Markdown

The gate must fail closed on missing overlay rows for every atlas key, not only on keys reviewers remember. High cardinality without an owner is a page-routing defect, and the gate should break the build. Unknown PII on any exported field is a publication defect and should break the documentation build. After the gate passes, render a single reference page that concatenates atlas identity with signed meaning and omits draft blurbs.

pip install pyyaml
python scripts/gate_obs_docs.py \
  --atlas docs/obs/atlas.generated.json \
  --overlay docs/obs/overlay.signed.yaml
python scripts/render_field_reference.py \
  --atlas docs/obs/atlas.generated.json \
  --overlay docs/obs/overlay.signed.yaml \
  --out docs/obs/published/field-reference.md

#!/usr/bin/env python3
"""Fail the docs build when atlas keys lack a signed meaning overlay."""
from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path

import yaml

REQUIRED = ("cardinality", "pii", "owner", "incident_meaning", "retention")
ALLOWED_CARD = {"low", "medium", "high"}
ALLOWED_PII = {"none", "indirect", "direct"}


def load_overlay(path: Path) -> dict:
    data = yaml.safe_load(path.read_text(encoding="utf-8")) or {}
    fields = data.get("fields")
    if not isinstance(fields, dict):
        raise SystemExit("overlay.signed.yaml must contain a top-level 'fields' mapping")
    return fields


def gate(atlas: dict, overlay: dict) -> list[str]:
    errors: list[str] = []
    atlas_fields = atlas.get("fields", {})
    for key in sorted(set(overlay) - set(atlas_fields)):
        errors.append(
            f"overlay has {key} but atlas does not; remove stale meaning or restore the constant"
        )
    for key in sorted(set(atlas_fields) - set(overlay)):
        errors.append(f"atlas key {key} has no overlay row; refuse to publish")
    for key, row in overlay.items():
        if key not in atlas_fields:
            continue
        if not isinstance(row, dict):
            errors.append(f"{key}: overlay row must be a mapping")
            continue
        for field in REQUIRED:
            value = row.get(field)
            if not isinstance(value, str) or not value.strip():
                errors.append(f"{key}: signed cell '{field}' is empty")
        card = row.get("cardinality")
        pii = row.get("pii")
        if card not in ALLOWED_CARD:
            errors.append(f"{key}: cardinality must be one of {sorted(ALLOWED_CARD)}")
        if pii not in ALLOWED_PII:
            errors.append(f"{key}: pii must be one of {sorted(ALLOWED_PII)}")
        if pii == "direct" and "example" in row:
            errors.append(f"{key}: direct PII cannot ship an example payload")
    return errors


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--atlas", type=Path, required=True)
    parser.add_argument("--overlay", type=Path, required=True)
    args = parser.parse_args()
    atlas = json.loads(args.atlas.read_text(encoding="utf-8"))
    overlay = load_overlay(args.overlay)
    errors = gate(atlas, overlay)
    if errors:
        print("observability docs gate failed:", file=sys.stderr)
        for item in errors:
            print(f"  - {item}", file=sys.stderr)
        raise SystemExit(1)
    print(f"gate passed for {len(overlay)} signed fields")


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""Render atlas identity plus signed overlay into a published Markdown reference."""
from __future__ import annotations

import argparse
import json
from pathlib import Path

import yaml


def render(atlas: dict, overlay: dict) -> str:
    lines = [
        "# Observability field reference",
        "",
        "Identity cells come from LOG_FIELD_REGISTRY. Meaning cells are human-signed.",
        "",
        "| Key | Type | Cardinality | PII | Owner | Retention | Incident meaning |",
        "|-----|------|-------------|-----|-------|-----------|------------------|",
    ]
    for key in sorted(atlas.get("fields", {})):
        ident = atlas["fields"][key]
        meaning = overlay[key]
        incident = meaning["incident_meaning"].replace("|", "\\|")
        lines.append(
            f"| {key} | {ident.get('type', '')} | {meaning['cardinality']} | "
            f"{meaning['pii']} | {meaning['owner']} | {meaning['retention']} | {incident} |"
        )
    lines.append("")
    return "\n".join(lines)


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--atlas", type=Path, required=True)
    parser.add_argument("--overlay", type=Path, required=True)
    parser.add_argument("--out", type=Path, required=True)
    args = parser.parse_args()
    atlas = json.loads(args.atlas.read_text(encoding="utf-8"))
    overlay = (yaml.safe_load(args.overlay.read_text(encoding="utf-8")) or {}).get("fields", {})
    args.out.parent.mkdir(parents=True, exist_ok=True)
    args.out.write_text(render(atlas, overlay), encoding="utf-8")
    print(f"wrote {args.out}")


if __name__ == "__main__":
    main()

# Proposal: CI job. Adapt paths to the repository; do not treat this as a hosted workflow.
name: obs-docs
on: [pull_request]
jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install pyyaml
      - run: python scripts/extract_log_fields.py --src ./app --out docs/obs/atlas.generated.json
      - run: python scripts/gate_obs_docs.py --atlas docs/obs/atlas.generated.json --overlay docs/obs/overlay.signed.yaml
      - run: python scripts/render_field_reference.py --atlas docs/obs/atlas.generated.json --overlay docs/obs/overlay.signed.yaml --out docs/obs/published/field-reference.md

Reproducible gate checks

Run these four checks against the sample registry before trusting the pipeline on a real service. Each check should exit nonzero and print a row-level reason rather than a generic documentation failure. If any check stays green, the gate is documenting intent instead of enforcing it. Keep the checks in CI next to the extractor so overlay drift cannot hide in a passing Markdown preview.

1. Remove http.route from the overlay and confirm the gate exits nonzero with a missing-row error.
2. Set user.email PII to any value outside none/indirect/direct and confirm the gate rejects publication.
3. Add an overlay-only key such as legacy.trace_id and confirm the gate reports a stale meaning row.
4. Leave order.id cardinality at high with a blank owner and confirm the required-cell check refuses to merge.

Limitations

This workflow assumes that field keys are declared in one registry the parser can see. Dynamic key construction, string-concatenated attributes, and vendor auto-instrumentation will not appear until you wrap them. Cardinality labels are not measurements; they are signed claims that still need periodic review against production index stats. The pipeline does not replace a data-retention policy, a DPA, or an on-call roster source of truth.

Generated example log lines remain unsafe until a reviewer checks redaction against the signed PII class. Models will happily invent an email, a token, or a street address because those strings look like documentation. Ban personally identifying examples in the draft prompt, then still grep drafts for @, Bearer, and numeric identifiers that match account formats. If legal review is required, the overlay is the artifact to attach, not the chat transcript.

Who should not use this approach

Do not use this pipeline if the codebase has no structured field registry and no plan to add one. Scraping ad-hoc logger.info messages will generate a confident atlas that still misses the keys that matter during incidents. Do not use it as a substitute for access-control on log backends, because documentation gates cannot prevent a query that dumps raw events. Skip the model drafting step entirely when the overlay authors already know the fields and only need the extract-and-gate loop.

Teams that publish customer-facing status pages from the same store should also keep this atlas internal. Incident meaning often includes vendor names, queue depths, and customer-count heuristics that do not belong on a public status page. Split public copy into a different signed document that uses a stricter public vocabulary list. The field atlas can still feed that document, but the meaning overlay must not leak.

What this does not claim

This workflow does not claim that generated documentation is accurate because a model wrote fluent sentences. Accuracy here means the keys match source, the overlay is signed, and the build failed when either side drifted. It also does not claim a reduction in incident minutes, because that measurement needs your pager data, not a tutorial. Use the gate's fail count and overlay review lag as the operational metrics if you need numbers.

Ship the atlas from CI on the next instrumentation change, and keep every paging sentence in the signed overlay. Start with a single service field registry rather than attempting a company-wide observability documentation rewrite. The useful test is simple: delete a field constant, watch the docs build fail, and confirm on-call still owns the remaining meaning row. If the build stays green after a real key deletion, the atlas is a brochure, not a contract.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.