A Config-Doc Coverage Matrix: Extract Flag Facts Before Any Model Writes Prose

Feature-flag documentation drifts when models invent defaults, owners, or rollback steps that source control never stated. The durable fix is a coverage matrix that separates extractable facts from sentences a human must still sign. Extracted facts include identifier, type, default, and call-site file; signed fields include blast radius, sunset, and customer impact. This article walks through a Python extractor, a YAML matrix, and tests that fail when those lanes mix.

The adjacent debate about comments versus clean code misses a more mechanical failure. Config references go stale because writers treat every sentence as interchangeable prose instead of typed fields with different owners. A flag name and its default are compile outputs. A promise about who gets the blast radius is a signature, not a completion.

What this workflow claims, and what it does not

The claim is narrow. Identifiers that already exist in code can be listed, hashed, and checked in continuous integration without asking a model to remember them. Short mechanical blurbs can be drafted from that list after the list is frozen. Anything that implies outage scope, legal wording, customer-visible naming, or a calendar date stays empty until a human fills it.

This is a proposed method, not a production case study. The snippets below are labeled examples. They are meant to be copied into a throwaway repository and run against a fixture tree, not treated as audited telemetry from a live fleet.

Decision table: draftable fields versus signed fields

Field	Source of truth	Model may draft?	Human must sign?	CI failure if empty
flag_id	Source identifier	No, extract only	No	Yes
value_type	Literal or annotation	No, extract only	No	Yes
default_value	Literal default	No, extract only	No	Yes
call_sites	File paths	No, extract only	No	Yes
mechanical_blurb	Frozen facts file	Yes, after extract	Review only	Soft warning
blast_radius	Ops knowledge	No	Yes	Yes
sunset_or_review_date	Calendar ownership	No	Yes	Yes
customer_facing_name	Product language	No	Yes	Yes
rollback_copy	Incident process	No	Yes	Yes

The table is the artifact that later tests encode. If a column says extract only, a model must not invent a value for it. If a column says human must sign, a merged document with a blank or a model-shaped placeholder is a failing build.

Step 1: Freeze a tiny, boring flag vocabulary

Pick one convention and refuse to parse folklore. The example below treats FLAG_* constants and os.getenv calls with literal keys as the only legal sources. Dynamic keys assembled at runtime are recorded as unknown and left unsigned, which is intentional. Unknown rows must not receive drafted prose.

# flag_extract.py — proposed example, not a vendor SDK
from __future__ import annotations

import ast
import hashlib
import json
from pathlib import Path
from typing import Any

ALLOWED_ROOTS = ("src", "app", "lib")


def _lit_str(node: ast.AST) -> str | None:
    if isinstance(node, ast.Constant) and isinstance(node.value, str):
        return node.value
    return None


def extract_file(path: Path) -> list[dict[str, Any]]:
    tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path))
    rows: list[dict[str, Any]] = []

    for node in ast.walk(tree):
        if isinstance(node, ast.Assign) and len(node.targets) == 1:
            target = node.targets[0]
            if isinstance(target, ast.Name) and target.id.startswith("FLAG_"):
                rows.append(
                    {
                        "flag_id": target.id,
                        "value_type": type(getattr(node.value, "value", "").__class__.__name__
                            if isinstance(node.value, ast.Constant)
                            else "unknown"),
                        "default_value": getattr(node.value, "value", None)
                        if isinstance(node.value, ast.Constant)
                        else None,
                        "call_sites": [str(path)],
                        "origin": "const",
                    }
                )
        if isinstance(node, ast.Call):
            func = node.func
            name = getattr(func, "attr", None) or getattr(func, "id", None)
            if name == "getenv" and node.args:
                key = _lit_str(node.args[0])
                default = _lit_str(node.args[1]) if len(node.args) > 1 else None
                rows.append(
                    {
                        "flag_id": key or "UNKNOWN_DYNAMIC_KEY",
                        "value_type": "str",
                        "default_value": default,
                        "call_sites": [str(path)],
                        "origin": "getenv",
                    }
                )
    return rows


def extract_repo(root: Path) -> list[dict[str, Any]]:
    merged: dict[str, dict[str, Any]] = {}
    for folder in ALLOWED_ROOTS:
        base = root / folder
        if not base.exists():
            continue
        for path in base.rglob("*.py"):
            for row in extract_file(path):
                key = row["flag_id"]
                if key in merged:
                    merged[key]["call_sites"] = sorted(
                        set(merged[key]["call_sites"] + row["call_sites"])
                    )
                else:
                    merged[key] = row
    return [merged[k] for k in sorted(merged)]


def facts_hash(rows: list[dict[str, Any]]) -> str:
    payload = json.dumps(rows, sort_keys=True, default=str).encode("utf-8")
    return hashlib.sha256(payload).hexdigest()[:16]


if __name__ == "__main__":
    root = Path(".")
    rows = extract_repo(root)
    print(json.dumps({"hash": facts_hash(rows), "flags": rows}, indent=2, default=str))

Run the extractor against a fixture, not against an unsanitized monorepo, until the hash is stable. A changing hash means the facts file is not frozen, so later drafting is not allowed. Keep unknown dynamic keys in the report so reviewers can see the gap instead of filling it with guessed names.

python flag_extract.py > artifacts/flag_facts.json
python -c "import json; print(json.load(open('artifacts/flag_facts.json'))['hash'])"

Step 2: Materialize a coverage matrix with empty signature lanes

Do not let the model create the matrix shape. Commit a schema that already contains the signed fields as empty strings or explicit needs_human sentinels. The mechanical blurb may be missing on the first pass. Blast radius may never be missing after the flag is referenced by a public route.

# docs/config_coverage.yaml — proposed schema
version: 1
facts_hash: "replace-after-extract"
rows:
  - flag_id: FLAG_ENABLE_BATCH_EXPORT
    value_type: bool
    default_value: false
    call_sites:
      - src/export/service.py
    mechanical_blurb: ""   # draftable after hash freeze
    blast_radius: ""       # human owned
    sunset_or_review_date: ""
    customer_facing_name: ""
    rollback_copy: ""
    signer: ""

The empty strings are not invitations. They are failing states. A later test distinguishes mechanical_blurb, which may be filled by a gated draft job, from the four human fields, which may not be filled by that job at all.

Step 3: Gate drafting so the model never sees empty signature columns as a prompt

Disclosure: This article was prepared as part of MonkeyCode's product outreach. After the facts hash is frozen, a drafting step may write only mechanical_blurb, and only from the extracted JSON. MonkeyCode's free model access and free server option are relevant here as a place to run the extractor on a shared box and to draft those blurbs without promoting the job into an owner of rollback language.

The prompt surface should be a facts slice, not the repository and not the YAML signature columns. A workable slice looks like the next block. Notice that blast radius, dates, and customer names are absent on purpose.

You receive JSON flags with flag_id, value_type, default_value, and call_sites.
Write mechanical_blurb as one sentence of at most 24 words.
Restate only type, default, and that the identifier is read in the listed files.
Do not invent owners, dates, severity, customers, rollback, or product names.
Return JSON {"flag_id": "...", "mechanical_blurb": "..."} and nothing else.

A merge rule then copies blurbs into the matrix only when flag_id matches and facts_hash is unchanged. If the hash moved, discard the draft and re-extract. That rule is more important than the wording of the prompt, because a prompt cannot police a mutated default.

Step 4: Encode the lanes as tests, not as review comments

Review comments do not survive Friday afternoon. Tests do. The suite below fails when source flags are missing from the matrix, when unknown keys received prose, or when human fields look like generated filler. Filler detection is deliberately crude: signed fields may not contain TBD, TODO, or the words probably and might.

# tests/test_config_coverage.py — proposed example
from __future__ import annotations

import json
import re
from pathlib import Path

import yaml

from flag_extract import extract_repo, facts_hash

HUMAN_FIELDS = (
    "blast_radius",
    "sunset_or_review_date",
    "customer_facing_name",
    "rollback_copy",
    "signer",
)
FILLER = re.compile(r"\b(TBD|TODO|probably|might|TBA)\b", re.I)
DATE = re.compile(r"^\d{4}-\d{2}-\d{2}$")


def test_facts_hash_matches_committed_matrix():
    root = Path(".")
    extracted = extract_repo(root)
    matrix = yaml.safe_load(Path("docs/config_coverage.yaml").read_text())
    assert matrix["facts_hash"] == facts_hash(extracted)


def test_every_extracted_flag_has_a_row():
    extracted = {row["flag_id"] for row in extract_repo(Path("."))}
    matrix = yaml.safe_load(Path("docs/config_coverage.yaml").read_text())
    documented = {row["flag_id"] for row in matrix["rows"]}
    missing = extracted - documented
    extra = documented - extracted
    assert not missing, f"undocumented flags: {sorted(missing)}"
    assert not extra, f"docs for flags not in source: {sorted(extra)}"


def test_unknown_keys_do_not_receive_drafted_prose():
    matrix = yaml.safe_load(Path("docs/config_coverage.yaml").read_text())
    for row in matrix["rows"]:
        if row["flag_id"] == "UNKNOWN_DYNAMIC_KEY":
            assert row.get("mechanical_blurb") in ("", None)
            for field in HUMAN_FIELDS:
                assert row.get(field) in ("", None)


def test_human_fields_are_signed_and_not_filler():
    matrix = yaml.safe_load(Path("docs/config_coverage.yaml").read_text())
    for row in matrix["rows"]:
        if row["flag_id"] == "UNKNOWN_DYNAMIC_KEY":
            continue
        for field in HUMAN_FIELDS:
            value = (row.get(field) or "").strip()
            assert value, f"{row['flag_id']} missing {field}"
            assert not FILLER.search(value), f"{row['flag_id']} {field} looks unsigned"
        assert DATE.match(row["sunset_or_review_date"]), row["flag_id"]

Wire the tests into the same job that publishes docs. Publishing without the hash check is how invented defaults reach customers. A one-line make target keeps the order honest: extract, compare, then render.

.PHONY: config-docs
config-docs:
    python flag_extract.py > artifacts/flag_facts.json
    pytest tests/test_config_coverage.py
    python tools/render_config_reference.py docs/config_coverage.yaml > site/config-reference.md

The renderer should print extracted columns as tables and signed columns as quoted blocks attributed to signer. If signer is empty, the renderer must refuse to write the page rather than emitting a blank heading.

Step 5: Render two visibly different lanes on the page

Readers should see the ownership split without reading this article. Generated rows can look like a compact table. Signed rows should look like a dated note with a name. Mixing those typographies is how teams start trusting a drafted sunset date that nobody chose.

## FLAG_ENABLE_BATCH_EXPORT

| Fact | Value |
| --- | --- |
| Type | bool |
| Default | false |
| Call sites | `src/export/service.py` |

Mechanical blurb: Boolean flag defaulting to false; read in `src/export/service.py`.

> Blast radius (signed by sre-oncall, review 2026-10-15): batch export for internal tenants only.
> Rollback: set the flag false and drain `export-worker` before the next deploy window.

That page remains useful if every drafting tool disappears. The table still compiles from source. The quoted block still requires a person who can be asked about the drain step.

Limitations the tests cannot hide

Regex-free AST extraction still misses flags assembled from prefixes, remote consoles, or generated code. Those identifiers will not appear in flag_facts.json, so the matrix cannot protect them. Teams that define flags only in a vendor dashboard should not use this method as their source of truth.

Mechanical blurbs can still overclaim if call sites are incomplete or if a default is a function call rather than a literal. The extractor records those as unknown instead of guessing, which leaves gaps. Gaps are cheaper than a wrong default in a public reference.

Human fields can be filled with confident fiction. The filler regex is a seatbelt, not counsel. Legal product names, accessibility language, and incident copy still need the owners who already approve those sentences outside documentation tooling.

Free model access does not create a review queue. Free server capacity does not freeze a facts hash. If either job can write blast_radius, the lane split has already failed, regardless of where the job runs.

Who should not use this approach

Do not use this workflow when flags are not in the repository at all. Do not use it for security advisories, pricing, or contractual uptime language, because those sentences are not config facts. Do not use it as a substitute for an incident runbook when rollback requires privileges the docs signer does not hold.

Small libraries with three settings and a maintainer who already updates a README by hand will spend more time on the matrix than on the product. The method starts to pay when the flag list changes faster than review memory, and when CI can fail a merge without a meeting.

What to keep when the drafting job is offline

Keep the extractor, the hash, and the tests. Keep the empty signature columns as failing states. Discard any blurb whose hash no longer matches. That leftover system still prevents undocumented flags from shipping, which is the original failure mode this article set out to contain.

If a shared box is already in the loop for extractors, MonkeyCode's free server option is one place to park the compiler beside the test job, with blurbs remaining optional and unsigned fields remaining mandatory.