If a Doc Claim Cannot Compile, Do Not Let a Model Draft It

Documentation generation goes wrong when a model writes claims that no repository file can refute. Volume is not the interesting failure mode here; untestable sentences are the actual failure mode. A practical gate is simple: a model may draft a claim only when that claim compiles against an artifact. Humans must own every sentence whose evidence is a promise, an incident, or a legal constraint.

Most generation pipelines score fluency, completeness, or reviewer time, and then publish the resulting pages. None of those scores ask whether a later engineer could disprove the sentence from the tree. If the answer is no, the sentence is not documentation; it is an untracked product assertion. Treat that distinction as a compiler problem for claims rather than as a writing-quality problem at review.

Evidence classes that actually gate generation

A claim compiles when a deterministic checker can resolve its pointer to a repository object today. Symbols, tests, OpenAPI operations, and configuration keys are compilable because they exist as files. Service levels, incident narratives, pricing, and legal statements do not compile, because their source is a decision. The table below is the policy; generation is forbidden from inventing extra classes at draft time.

Evidence class	Example pointer	Model may draft?	Human must sign?
code_symbol	src/auth/session.py:Session.ttl	yes	no
test_id	tests/test_session.py::test_ttl_expires	yes	no
openapi_operation	GET /v1/sessions	yes	no
config_key	SESSION_TTL_SECONDS	yes	no
sla	monthly availability target	no	yes
incident	INC-2041 customer-facing outage	no	yes
legal	data-residency statement	no	yes
pricing	included minutes on a free tier	no	yes
architecture_intent	sharding postponed until a named trigger	no	yes

The first four classes share one property: a checkout either contains the object or it does not. The last five classes share the opposite property: the object lives in a decision record, a contract, or a person's head. Mixing those properties in one generated paragraph is how support pages acquire confident falsehoods.

A four-step workflow

1. Split pages into claim units, not into headings

Inventory the documentation tree by file, then split each file into claim-sized units rather than sections. A claim is one sentence that could be true or false without depending on neighboring sentences. Record the claim identifier, the file path, and a proposed evidence class in a YAML manifest. Leave architecture essays in the human-signed bucket until a pointer actually exists in the tree.

2. Require a resolvable pointer or a human signature

Attach a pointer that a script can resolve: a path with a symbol, a test node id, or an operation. Reject vague pointers such as "see the auth package" because they cannot fail a checkout. For human-signed claims, require a person, a date, and a ticket or policy URL instead of a file. Do not let the model fill those fields; empty signature fields must fail the gate.

3. Draft only files whose rows are entirely compilable

Allow a model to draft only files whose manifest rows are entirely in the compilable classes. Feed the model the resolved snippets, not the whole repository, so the draft cannot wander into promises. After the draft, re-run the checker so new sentences without markers cannot land in the branch. Keep human-signed files out of the generation job even when the surrounding page looks similar in tone.

4. Make the checker the merge gate, not the reviewer

Install the checker as a required CI job on documentation paths and on the provenance manifest itself. A red build must mean an unresolved pointer, a missing signature, or a model file with a human class. A green build does not mean the sentence is wise; it means the sentence is allowed to exist. Reviewers then spend attention on signed claims and on whether the pointers still match product intent.

Worked example: markers, a manifest, and a stdlib checker

Label the following as a local, executable example rather than as production metrics. Each claim in Markdown carries a structured HTML comment so the page remains readable while staying machine-checkable.

---
generator: model
---

<!-- claim-id: sess-ttl-default class: config_key pointer: SESSION_TTL_SECONDS -->
The default session time-to-live is read from `SESSION_TTL_SECONDS`.

<!-- claim-id: sess-ttl-test class: test_id pointer: tests/test_session.py::test_ttl_expires -->
`test_ttl_expires` expires a session after that configured interval.

Human-owned files use a different front matter block and must not set generator: model.

---
generator: human
---

<!-- claim-id: sess-sla class: sla signed_by: sre-oncall signed_at: 2026-09-05 source: POL-SLA-004 -->
Session authentication availability is covered by the public monthly SLA.

The manifest is the allow-list. Unknown classes are errors, which prevents a drafting job from growing a private taxonomy.

# docs/provenance.yml
allowed_compilable:
  - code_symbol
  - test_id
  - openapi_operation
  - config_key
allowed_human:
  - sla
  - incident
  - legal
  - pricing
  - architecture_intent
config_keys:
  - SESSION_TTL_SECONDS
openapi_operations:
  - GET /v1/sessions

The checker below uses only the Python standard library so a documentation CI job can vendor it cheaply. It walks docs/, parses front matter and claim comments, then fails the three cases that actually matter for generation control.

#!/usr/bin/env python3
"""check_doc_provenance.py — fail unmarked, unresolvable, or unsigned claims."""
from __future__ import annotations

import argparse, pathlib, re, sys
from typing import Dict, List

try:
    import yaml  # optional; fallback parser below for the tiny manifest subset
except ImportError:
    yaml = None

CLAIM_RE = re.compile(
    r"<!--\s*claim-id:\s*(?P<id>\S+)\s+class:\s*(?P<cls>\S+)"
    r"(?:\s+pointer:\s*(?P<pointer>\S+))?"
    r"(?:\s+signed_by:\s*(?P<signed_by>\S+))?"
    r"(?:\s+signed_at:\s*(?P<signed_at>\S+))?"
    r"(?:\s+source:\s*(?P<source>\S+))?\s*-->",
    re.I,
)
FM_RE = re.compile(r"^---\n(.*?)\n---\n", re.S)

COMPILABLE = {"code_symbol", "test_id", "openapi_operation", "config_key"}
HUMAN = {"sla", "incident", "legal", "pricing", "architecture_intent"}


def load_manifest(path: pathlib.Path) -> Dict:
    text = path.read_text(encoding="utf-8")
    if yaml:
        data = yaml.safe_load(text)
        return data
    # Tiny fallback: enough for the example file when PyYAML is absent.
    data = {"allowed_compilable": [], "allowed_human": [],
            "config_keys": [], "openapi_operations": []}
    current = None
    for raw in text.splitlines():
        line = raw.strip()
        if not line or line.startswith("#"):
            continue
        if line.endswith(":") and not line.startswith("-"):
            current = line[:-1]
            continue
        if line.startswith("- ") and current:
            data.setdefault(current, []).append(line[2:].strip())
    return data


def parse_front_matter(text: str) -> Dict[str, str]:
    match = FM_RE.match(text)
    meta: Dict[str, str] = {}
    if not match:
        return meta
    for line in match.group(1).splitlines():
        if ":" in line:
            key, value = line.split(":", 1)
            meta[key.strip()] = value.strip()
    return meta


def resolve_pointer(cls: str, pointer: str, root: pathlib.Path, manifest: Dict) -> str | None:
    if not pointer:
        return "missing pointer"
    if cls == "config_key":
        return None if pointer in set(manifest.get("config_keys") or []) else "unknown config_key"
    if cls == "openapi_operation":
        allowed = set(manifest.get("openapi_operations") or [])
        return None if pointer in allowed else "unknown openapi_operation"
    if cls == "test_id":
        file_part = pointer.split("::", 1)[0]
        return None if (root / file_part).is_file() else f"missing test file {file_part}"
    if cls == "code_symbol":
        if ":" not in pointer:
            return "code_symbol must look like path:Name"
        rel, _name = pointer.split(":", 1)
        return None if (root / rel).is_file() else f"missing source file {rel}"
    return "unresolvable class"


def check_file(path: pathlib.Path, root: pathlib.Path, manifest: Dict) -> List[str]:
    text = path.read_text(encoding="utf-8")
    meta = parse_front_matter(text)
    generator = meta.get("generator", "")
    errors: List[str] = []
    claims = list(CLAIM_RE.finditer(text))
    if not claims:
        errors.append(f"{path}: no claim markers")
        return errors
    allowed_c = set(manifest.get("allowed_compilable") or []) or COMPILABLE
    allowed_h = set(manifest.get("allowed_human") or []) or HUMAN
    for match in claims:
        cls = match.group("cls")
        cid = match.group("id")
        loc = f"{path}#{cid}"
        if cls in allowed_c:
            err = resolve_pointer(cls, match.group("pointer") or "", root, manifest)
            if err:
                errors.append(f"{loc}: {err}")
            if generator == "model" and cls in allowed_h:
                errors.append(f"{loc}: human class inside model file")
        elif cls in allowed_h:
            if generator == "model":
                errors.append(f"{loc}: human class inside model file")
            if not match.group("signed_by") or not match.group("signed_at") or not match.group("source"):
                errors.append(f"{loc}: human claim missing signed_by/signed_at/source")
        else:
            errors.append(f"{loc}: unknown class {cls}")
    if generator == "model":
        for match in claims:
            if match.group("cls") in allowed_h:
                errors.append(f"{path}: model file contains human-signed class")
    return errors


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--root", type=pathlib.Path, default=pathlib.Path("."))
    parser.add_argument("--docs", type=pathlib.Path, default=pathlib.Path("docs"))
    parser.add_argument("--manifest", type=pathlib.Path, default=pathlib.Path("docs/provenance.yml"))
    args = parser.parse_args()
    manifest = load_manifest(args.manifest)
    errors: List[str] = []
    for path in sorted((args.root / args.docs).rglob("*.md")):
        errors.extend(check_file(path, args.root, manifest))
    for item in errors:
        print(item, file=sys.stderr)
    print(f"{len(errors)} provenance error(s)")
    return 1 if errors else 0


if __name__ == "__main__":
    raise SystemExit(main())

A minimal test plan belongs next to the checker so the gate cannot silently weaken. The cases are fixtures, not live product measurements.

# test_check_doc_provenance.py
from pathlib import Path
import check_doc_provenance as c

def write_tree(tmp: Path) -> None:
    (tmp / "docs").mkdir()
    (tmp / "tests").mkdir()
    (tmp / "tests" / "test_session.py").write_text("def test_ttl_expires():\n    pass\n")
    (tmp / "docs" / "provenance.yml").write_text(
        "allowed_compilable:\n  - test_id\n  - config_key\n"
        "allowed_human:\n  - sla\nconfig_keys:\n  - SESSION_TTL_SECONDS\n"
    )

def test_model_file_with_sla_fails(tmp_path: Path):
    write_tree(tmp_path)
    (tmp_path / "docs" / "bad.md").write_text(
        "---\ngenerator: model\n---\n\n"
        "<!-- claim-id: x class: sla signed_by: a signed_at: 2026-09-05 source: POL -->\n"
        "We promise 99.9 percent.\n"
    )
    manifest = c.load_manifest(tmp_path / "docs" / "provenance.yml")
    errors = c.check_file(tmp_path / "docs" / "bad.md", tmp_path, manifest)
    assert any("human class" in e for e in errors)

def test_compilable_pointer_passes(tmp_path: Path):
    write_tree(tmp_path)
    (tmp_path / "docs" / "ok.md").write_text(
        "---\ngenerator: model\n---\n\n"
        "<!-- claim-id: t class: test_id pointer: tests/test_session.py::test_ttl_expires -->\n"
        "The expiry test covers the configured TTL.\n"
    )
    manifest = c.load_manifest(tmp_path / "docs" / "provenance.yml")
    assert c.check_file(tmp_path / "docs" / "ok.md", tmp_path, manifest) == []

Run the example from a repository root after placing the files.

python3 check_doc_provenance.py --root . --docs docs --manifest docs/provenance.yml
python3 -m pytest test_check_doc_provenance.py -q

It refuses unknown classes, which is the entire point: generation must not grow a private taxonomy. Tests above encode the three failures that matter: missing markers, bad pointers, and unsigned human claims. Reviewers should still read signed pages; the job only removes unmarked model prose from the default path.

Where a drafting model is allowed to participate

Compilable files are the only place a drafting model should run, because the checker can still fail them. MonkeyCode's free model access and free server option can host that narrow job without expanding into signed pages. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The checker remains a local artifact and does not require that hosting path to be valid.

Prompt the model with the resolved snippet only, for example a test body or a configuration default, and ask for one marked sentence. Do not paste incident tickets, pricing sheets, or SLA numbers into that prompt, because those tokens are human classes by construction. If the returned Markdown omits a claim comment, the checker fails, which is the desired outcome rather than a retry loop.

Limitations

Provenance is not truth: a test can exist and still document the wrong product behavior in prose. Comments, example snippets, and OpenAPI descriptions drift, so a resolved pointer can still be stale. The unit of a claim is fuzzy; compound sentences will smuggle a promise beside a compilable fact. Teams that skip CODEOWNERS on documentation paths will watch those signatures become a rubber stamp.

The fallback YAML parser understands only top-level lists, so a richer manifest needs PyYAML or a stricter schema file. Symbol resolution in this example checks that the path exists, not that the Python identifier still parses, which is a deliberate cheap bound. Extending it with ast walking is reasonable; treating that extension as proof of semantic correctness is not.

Who should not use this approach

Do not use this workflow for regulated labeling, medical copy, or customer contracts that require counsel. Do not use it to generate incident reports, because those pages are the human evidence class by definition. Do not use it when the repository has no tests, no OpenAPI file, and no stable configuration keys. In that case the model has nothing to compile against, and the gate would only launder guesses.

Skip it for changelog marketing, community posts, and any page whose audience is not an engineer holding a checkout. Those pages can still be written by people; they simply should not enter a generation job that this checker would paint green. If your documentation corpus is a single README without claim boundaries, add structure first, then consider drafting.

Start with one API reference file, one manifest, and a checker job that fails on unmarked sentences. Expand generation only after the failures you see are unresolved pointers rather than missing policy. The core conclusion does not change with more models: untestable documentation is still an unsigned assertion.