Split Schema Docs: AST-Built Column Atlas, Human-Signed PII and Retention

Schema documentation fails in review when generated column lists and compliance labels occupy the same markdown file. A compiler can extract mapped columns, SQL types, and nullability from Python AST without importing application modules. A language model may draft short glosses from that extracted list, but it cannot assign PII class, retention windows, or cascade meaning. Those labels belong in a human-signed policy file that continuous integration refuses to accept from any model-authored job.

The failure mode this workflow targets

Reviewers treat a data dictionary as true when column names, types, and privacy labels sit together in one generated page. Name-based heuristics then mark user_token as secret while leaving notes unlabeled, even when notes store support transcripts. Cascade flags from ondelete look like policy, yet they only record a database action, not the product consequence of orphaned rows. Mixing those layers produces docs that compile cleanly and still fail a privacy or incident review.

The split below keeps mechanical fields in a compiler output and keeps judgment fields in a signed sidecar. The compiler never imports application models, so settings modules and database engines cannot run during documentation builds. When a model is used, it reads only the facts file and writes gloss drafts that cannot satisfy the policy check. That third path exists so prose experiments cannot overwrite a signature cell during a regenerate step.

Decision table: compile lane versus signature lane

Cell	Allowed source	Owner after merge
table name, attribute, column name	AST of mapped_column / Column	compiler
declared type string	AST of the type expression	compiler
nullable, unique, index	AST keywords	compiler
foreign-key target name	AST of the ForeignKey string	compiler
ondelete token	AST keyword	compiler, token only
one-line gloss	facts file only	model draft, then human edit
PII class	never inferred from names	human
retention window	never inferred from types	human
cascade product meaning	never equal to ondelete	human
query audience	never inferred from module path	human

The decision table is the contract for the compiler script, the policy file, and the review checklist. If a cell can be computed from syntax, it does not belong in the signed file. If a cell changes legal or operational duty, it does not belong in model output. Review comments should cite a cell name from this table instead of arguing about tone.

Step 1: Pin a small mapped-column fixture

Use a fixture that resembles production models without loading a database or a settings object. The example below is labeled as an unexecuted sample for the compiler tests, not as a live schema. Keep the fixture in tests/fixtures so application startup code never owns the documentation sample. Column names in the sample should be boring and realistic rather than jokey placeholders that hide review risk.

# example_fixture: tests/fixtures/models_sample.py
from sqlalchemy import ForeignKey, String, Text
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column


class Base(DeclarativeBase):
    pass


class Account(Base):
    __tablename__ = "accounts"
    id: Mapped[int] = mapped_column(primary_key=True)
    email: Mapped[str] = mapped_column(String(255), unique=True, nullable=False)
    notes: Mapped[str | None] = mapped_column(Text, nullable=True)


class SessionRow(Base):
    __tablename__ = "sessions"
    id: Mapped[int] = mapped_column(primary_key=True)
    account_id: Mapped[int] = mapped_column(
        ForeignKey("accounts.id", ondelete="CASCADE")
    )
    refresh_token: Mapped[str] = mapped_column(String(512), nullable=False)

The fixture includes a unique email, a free-text notes column, and a cascade toward accounts. Those three shapes are enough to prove that uniqueness, nullability, and ondelete can compile while privacy cannot. Tests should refuse to import this module inside the compiler process. A compiler that imports models will execute module-level settings and is already outside this workflow.

Step 2: Compile the atlas with AST only

The compiler walks a directory, parses each .py file, and records mapped_column assignments on class bodies. It stores a JSON list plus a markdown table, and it omits every human-owned column from both outputs. Run the compiler as a project tool invoked from CI, not as an import of application modules. Missing __tablename__ constants cause the class to be skipped rather than inferred from the class name.

# compile_column_atlas.py — example tool
from __future__ import annotations

import ast
import json
from pathlib import Path


def _kw(call: ast.Call, name: str):
    for kw in call.keywords:
        if kw.arg == name and isinstance(kw.value, ast.Constant):
            return kw.value.value
    return None


def _fk_meta(call: ast.Call):
    target, ondelete = None, None

    def from_fk(fk: ast.Call):
        nonlocal target, ondelete
        if fk.args and isinstance(fk.args[0], ast.Constant):
            target = fk.args[0].value
        ondelete = _kw(fk, "ondelete")

    for arg in call.args:
        if isinstance(arg, ast.Call) and getattr(arg.func, "id", "") == "ForeignKey":
            from_fk(arg)
    for kw in call.keywords:
        val = kw.value
        if isinstance(val, ast.Call) and getattr(val.func, "id", "") == "ForeignKey":
            from_fk(val)
    return target, ondelete


def _type_str(call: ast.Call) -> str | None:
    for arg in call.args:
        if isinstance(arg, ast.Call) and getattr(arg.func, "id", None) == "ForeignKey":
            continue
        if isinstance(arg, ast.Call) and getattr(arg.func, "id", None):
            return arg.func.id
        if isinstance(arg, ast.Name) and arg.id != "ForeignKey":
            return arg.id
    return None


def compile_file(path: Path) -> list[dict]:
    tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path))
    rows: list[dict] = []
    for node in tree.body:
        if not isinstance(node, ast.ClassDef):
            continue
        table = None
        for stmt in node.body:
            if isinstance(stmt, ast.Assign):
                for t in stmt.targets:
                    if isinstance(t, ast.Name) and t.id == "__tablename__":
                        if isinstance(stmt.value, ast.Constant):
                            table = stmt.value.value
        if not table:
            continue
        for stmt in node.body:
            if not isinstance(stmt, ast.AnnAssign) or not isinstance(stmt.target, ast.Name):
                continue
            if not isinstance(stmt.value, ast.Call):
                continue
            if getattr(stmt.value.func, "id", None) != "mapped_column":
                continue
            fk, ondelete = _fk_meta(stmt.value)
            rows.append(
                {
                    "source": str(path).replace("\\", "/"),
                    "class_name": node.name,
                    "table": table,
                    "attribute": stmt.target.id,
                    "declared_type": _type_str(stmt.value),
                    "nullable": _kw(stmt.value, "nullable"),
                    "unique": _kw(stmt.value, "unique"),
                    "primary_key": bool(_kw(stmt.value, "primary_key")),
                    "fk_target": fk,
                    "ondelete_token": ondelete,
                }
            )
    return rows


def main() -> None:
    root = Path("app")
    rows: list[dict] = []
    for path in sorted(root.rglob("*.py")):
        rows.extend(compile_file(path))
    out = Path("docs/generated")
    out.mkdir(parents=True, exist_ok=True)
    (out / "column_atlas.json").write_text(
        json.dumps(rows, indent=2, sort_keys=True) + "\n", encoding="utf-8"
    )
    lines = [
        "# Column atlas (compiled)",
        "",
        "| table | attribute | declared_type | nullable | unique | pk | fk_target | ondelete_token |",
        "|---|---|---|---|---|---|---|---|",
    ]
    keys = [
        "table",
        "attribute",
        "declared_type",
        "nullable",
        "unique",
        "primary_key",
        "fk_target",
        "ondelete_token",
    ]
    for r in rows:
        cells = ["" if r.get(k) is None else str(r.get(k)) for k in keys]
        lines.append("| " + " | ".join(cells) + " |")
    (out / "column_atlas.md").write_text("\n".join(lines) + "\n", encoding="utf-8")


if __name__ == "__main__":
    main()

Command sequence for a clean compile on a checkout that already contains models under app/:

python compile_column_atlas.py
test -f docs/generated/column_atlas.json
python -c "import json; print(len(json.load(open('docs/generated/column_atlas.json'))))"

The JSON file is the only input that later stages may read for glosses or coverage checks. Markdown remains a human-readable view of the same cells and must not grow extra columns in the generate job. Commit both generated files so reviewers can diff mechanical drift without opening the policy sidecar. A regenerate that rewrites policy YAML is a pipeline bug, not a convenience.

Step 3: Keep the policy sidecar empty of compiled facts

Create docs/signed/column_policy.yaml with keys that match table.attribute and with no type fields. Reviewers fill PII class, retention, cascade meaning, and query audience during the same change that adds a column. A duplicate type column in this file is a process bug, because it will drift from AST. CODEOWNERS should list privacy or data stewards on this path so the compiler owners cannot self-sign.

# docs/signed/column_policy.yaml — human-owned; example structure
version: 1
columns:
  accounts.id:
    pii_class: "internal_key"
    retention: "account_lifetime"
    cascade_meaning: "n/a"
    query_audience: "identity_runtime"
  accounts.email:
    pii_class: "direct_identifier"
    retention: "account_lifetime_plus_30d"
    cascade_meaning: "n/a"
    query_audience: "identity_and_support_tools"
  accounts.notes:
    pii_class: "possible_unstructured_pii"
    retention: "support_window_then_purge"
    cascade_meaning: "n/a"
    query_audience: "support_only"
  sessions.id:
    pii_class: "internal_key"
    retention: "session_lifetime"
    cascade_meaning: "n/a"
    query_audience: "auth_runtime_only"
  sessions.account_id:
    pii_class: "internal_key"
    retention: "follow_account"
    cascade_meaning: "deleting an account must drop sessions; users lose devices"
    query_audience: "auth_runtime_only"
  sessions.refresh_token:
    pii_class: "credential"
    retention: "session_lifetime"
    cascade_meaning: "n/a"
    query_audience: "auth_runtime_only"

The accounts.notes row exists because a name heuristic would skip free text while a human reviewer cannot skip it. The cascade cell explains product loss for users, not the SQL token already stored in the atlas. Credential columns such as refresh_token get an audience of auth runtime, which is a duty statement rather than a type. None of those three labels can be reconstructed from the compiler JSON without inventing policy.

Step 4: Optional gloss drafts from the facts file only

Gloss text is the only place a model adds value, and only after the atlas exists. Feed column_atlas.json only, and forbid repository paths, source comments, and policy YAML as model context. Treat the model result as a draft beside the atlas, and never as a mergeable policy file. Version the prompt as a checked-in template so chat sidebars cannot widen the context window later.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access can draft those glosses from the facts file after the compiler finishes. The free server option can run the compiler without packing the rest of the tree into the prompt. Neither feature classifies PII, sets retention, or explains cascade impact; those cells stay unsigned if a model wrote them.

A narrow prompt shape looks like the following, and should be versioned as a template rather than typed into a chat window.

Source: docs/generated/column_atlas.json only.
Write a one-sentence gloss per row using table, attribute, declared_type,
nullable, unique, and fk_target.
Do not guess PII, retention, encryption, audience, or cascade meaning.
Leave any uncertain gloss as UNREVIEWED.
Output markdown table: table, attribute, gloss, review_state.

Store the model output at docs/generated/column_gloss_draft.md so it stays mechanically separate from signed policy. Humans may copy a gloss into narrative docs after they edit uncertain rows marked UNREVIEWED. Continuous integration must not require glosses for merge, so a model outage cannot block schema changes. That rule prevents subjective gloss tone from becoming an accidental release gate on schema changes.

Step 5: Gate merge on coverage, not on prose

The check loads atlas keys and policy keys, then fails on missing, extra, or model-tagged policy rows without scoring gloss quality. Example checker code below reads YAML and JSON from disk and prints every violation to stderr. A non-zero exit status is the only interface that CI should consume from this checker. The checker must not rewrite the policy file to close gaps, because that would forge a human signature.

# check_column_policy.py — example tool
from __future__ import annotations

import json
import sys
from pathlib import Path

import yaml  # example assumes PyYAML is already a test dependency

ATLAS = Path("docs/generated/column_atlas.json")
POLICY = Path("docs/signed/column_policy.yaml")
REQUIRED = ("pii_class", "retention", "cascade_meaning", "query_audience")
FORBIDDEN_POLICY_KEYS = ("declared_type", "nullable", "unique", "ondelete_token")


def main() -> int:
    rows = json.loads(ATLAS.read_text(encoding="utf-8"))
    policy = yaml.safe_load(POLICY.read_text(encoding="utf-8")) or {}
    cols = policy.get("columns") or {}
    atlas_keys = {f"{r['table']}.{r['attribute']}" for r in rows}
    policy_keys = set(cols)
    errors: list[str] = []
    missing = sorted(atlas_keys - policy_keys)
    extra = sorted(policy_keys - atlas_keys)
    if missing:
        errors.append("policy missing: " + ", ".join(missing))
    if extra:
        errors.append("policy extra: " + ", ".join(extra))
    for key, body in cols.items():
        if not isinstance(body, dict):
            errors.append(f"{key}: not a mapping")
            continue
        for fk in FORBIDDEN_POLICY_KEYS:
            if fk in body:
                errors.append(f"{key}: compiled field {fk} duplicated in policy")
        for req in REQUIRED:
            val = body.get(req)
            if not val or str(val).strip() in {"", "TODO", "UNREVIEWED"}:
                errors.append(f"{key}: {req} unsigned")
            if isinstance(val, str) and val.lower().startswith("model:"):
                errors.append(f"{key}: {req} marked model-authored")
    for line in errors:
        print(line, file=sys.stderr)
    return 1 if errors else 0


if __name__ == "__main__":
    raise SystemExit(main())

python compile_column_atlas.py
python check_column_policy.py

Wire both commands after unit tests and before any documentation publish job so unsigned columns cannot ship in HTML. Publishing the atlas without a green policy check recreates the original mixed-file failure for every reader. Skipping this coverage check is a stronger incident precursor than shipping HTML without optional gloss drafts. Keep the policy path in CODEOWNERS so the same author cannot land compiler and signature in one quiet commit.

Test plan for the compiler and the gate

These seven checks document the boundary more clearly than a prose style guide about documentation ownership. If a future generator writes policy YAML, the model-prefix test should fail before human review starts. Put the fixture and expected JSON under version control beside the compiler, not in a scratch notebook. Do not mock AST with handmade dictionaries that already include PII fields; that hides compiler leakage.

1. Parse models_sample.py and expect four attributes across the accounts and sessions tables in the atlas JSON.
2. Assert that accounts.email is unique and that accounts.notes is nullable in the compiled JSON object.
3. Assert sessions.account_id.ondelete_token equals CASCADE and that the JSON contains no PII or retention fields.
4. Remove accounts.notes from the policy file and expect a non-zero exit from the coverage checker.
5. Add a declared_type key under a policy row and expect a duplicate compiled-field failure from the checker.
6. Prefix a pii_class value with model: and expect the checker to reject that row as unsigned.
7. Run the compiler in a process that cannot import sqlalchemy even though the fixture module would need it.

Point seven is easiest with a virtualenv that omits SQLAlchemy and still runs compile_column_atlas.py against the fixture path. If that process imports the ORM, the documentation job has left the AST lane. Record the expected atlas JSON as a committed snapshot so type-token drift shows up as a diff rather than a silent docs rewrite.

Limitations and who should not use this

AST compilation misses columns built in mixins through runtime loops, __table__ constructed by hand, and hybrids that never call mapped_column. Teams with those patterns need an additional inspected-metadata job under a locked settings profile, which this article does not provide. The workflow also ignores Alembic revisions that added columns without model updates, so the atlas is not a substitute for migration review. Hybrid properties and synonyms will be absent until someone extends the compiler with explicit allowlists.

Name-to-PII shortcuts remain unsafe even when shown only as non-blocking suggestions beside the policy file. Columns named notes, metadata, or payload regularly hold identifiers after an unreviewed support-tool change. Do not send production dumps, row samples, or comment blocks that mention hosts into the gloss prompt. The gloss model should see types and names from the atlas, and it should see nothing else from the repository.

Skip this approach when a single person both compiles and signs policy without a second reviewer on the sidecar. Skip it when legal requires a formal records schedule that this YAML structure cannot represent with required fields. Skip it when the schema is generated entirely outside Python, because the AST walk will emit an empty atlas. Skip it when the goal is to let a model infer sensitivity from column names, because that goal conflicts with the gate.

The durable output is a short atlas, a shorter policy file, and a checker that treats unsigned privacy cells as build failures. Gloss drafts remain optional commentary on mechanical facts rather than a source of operational or legal duty. Reviewers should argue about cascade meaning and retention, not about whether email is a string. When those arguments happen on the policy file, the generated atlas can change without laundering a privacy decision.