Let the Model Draft Only What the Repo Can Prove: A Two-Class Documentation Pipeline

Most documentation pipelines fail in a boring way: the model writes a sentence that looks checkable, such as a flag, a config key, or an exit code, and nobody checks it. The fix I keep coming back to is to classify every sentence before drafting, then attach a different verification mechanism to each class. Derived sentences describe facts the repository already contains, so a script can prove or disprove them; owned sentences describe commitments, guarantees, and support windows, so a named human must sign them. The pipeline below builds an artifact index, drafts only from that index, blocks owned claims inside model-drafted regions, and fails CI on any derived atom it cannot resolve.

The whole thing runs offline except for the drafting step, and the test fixtures are three Markdown documents you can copy in a few minutes.

1. Why generated docs break at the atom level

Reviewers read prose, but drift happens in atoms: --dry-run, MAX_RETRIES, exit code 3, retry.budget_ms. A paragraph can stay stylistically perfect for a year while every atom inside it rots after one refactor. Human review does not catch this reliably, because the reviewer's attention is on meaning, not on identifier spelling.

So the useful question is not "is this paragraph good" but "can this specific atom be resolved against something the repo emits". That reframing is what makes the workflow automatable, and it also tells you exactly where a model must not be trusted.

2. The two-class policy

Classify each sentence by what it asserts, not by who wrote it. The table is the contract the rest of the pipeline enforces.

Class	Typical triggers	Machine check	Who signs
Derived fact	backticked flags, env vars, config keys, function names, exit codes inside fences	resolve against artifact index	model may draft; CI verifies
Owned claim	guarantees, never/always, deprecation dates, support windows, security, pricing, roadmap	keyword lint plus owner marker	named human
Narrative	motivation, tutorial framing, troubleshooting story	no mechanical check	human reviewer of record

Two rules follow. First, the drafting prompt receives the artifact index and nothing else about the product surface, so inventing a flag requires inventing it from an empty allowlist. Second, owned-claim sentences may not appear inside a model-drafted block, which turns a judgement call into a parse error.

3. Step 1: Build an artifact index from the repo

The index is a JSON file of names the documentation is allowed to reference. Generate it; never hand-maintain it.

# tools/docs_claims/index_build.py
import ast, json, pathlib, re, subprocess, sys

SKIP = {".venv", "build", "dist", "node_modules"}

def python_symbols(root: pathlib.Path) -> set[str]:
    names: set[str] = set()
    for path in root.rglob("*.py"):
        if SKIP & set(path.parts):
            continue
        tree = ast.parse(path.read_text(encoding="utf-8"))
        for node in ast.walk(tree):
            if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
                names.add(node.name)
            elif isinstance(node, ast.Assign):
                names.update(t.id for t in node.targets if isinstance(t, ast.Name) and t.id.isupper())
    return names

def cli_flags(entry: str) -> set[str]:
    out = subprocess.run([sys.executable, entry, "--help"], capture_output=True, text=True, check=True).stdout
    return set(re.findall(r"(?<![\w-])(--[a-z0-9][a-z0-9-]*)", out))

def schema_keys(path: pathlib.Path) -> set[str]:
    keys: set[str] = set()
    def walk(node):
        if isinstance(node, dict):
            props = node.get("properties")
            if isinstance(props, dict):
                keys.update(props)
            for value in node.values():
                walk(value)
        elif isinstance(node, list):
            for item in node:
                walk(item)
    walk(json.loads(path.read_text(encoding="utf-8")))
    return keys

Serialize the three sets into docs_claims_index.json with category labels, and commit the generator rather than the output. The index becomes stale only if you forget to regenerate it, which is a CI step, not a human habit.

4. Step 2: Extract atoms from the draft

Parsing Markdown with regexes is crude, but the atoms you need are all inside code spans and fences, and those are unambiguous.

FENCE = re.compile(r"```

[^\n]*\n(.*?)

```", re.S)
INLINE = re.compile(r"`([^`\n]+)`")
FLAG = re.compile(r"(?<![\w-])(--[a-z0-9][a-z0-9-]*)")
ENVVAR = re.compile(r"\b([A-Z][A-Z0-9_]{2,})\b")

def atoms(text: str) -> set[str]:
    found: set[str] = set()
    spans = INLINE.findall(text) + [line for block in FENCE.findall(text) for line in block.splitlines()]
    for span in spans:
        found.update(FLAG.findall(span))
        found.update(ENVVAR.findall(span))
    return found

Extracting from inline spans and fenced lines only keeps the false-positive rate low, because prose words never enter the candidate set. If your docs embed identifiers without backticks, fix the docs rather than widening the regex, since unmarked identifiers are unverifiable by construction.

5. Step 3: Resolve, classify, and fail the build

The resolver produces three buckets: resolved atoms, unresolved atoms, and policy violations. Ownership markers are HTML comments, so they stay invisible in rendered output.

OWNED_TRIGGERS = ("guarantee", "we will", "never", "always", "deprecat", "support window", "secure", "compliant", "SLA")
MODEL_BLOCK = re.compile(r"<!-- draft:model -->(.*?)<!-- /draft:model -->", re.S)
OWNER_MARK = re.compile(r"<!-- owner: @([a-z0-9_-]+) -->")

def owned_in_model_blocks(text: str) -> list[str]:
    hits = []
    for block in MODEL_BLOCK.findall(text):
        for sentence in re.split(r"(?<=[.!?])\s+", block):
            low = sentence.lower()
            if any(trigger.lower() in low for trigger in OWNED_TRIGGERS):
                hits.append(sentence.strip()[:90])
    return hits

def unowned_owned_claims(text: str) -> list[str]:
    missing = []
    for paragraph in text.split("\n\n"):
        low = paragraph.lower()
        if any(t.lower() in low for t in OWNED_TRIGGERS) and not OWNER_MARK.search(paragraph):
            missing.append(paragraph.strip().splitlines()[0][:90])
    return missing

Exit non-zero when unresolved is non-empty or either violation list is non-empty. Print atom names and line numbers only, because a report long enough to skim will be skimmed.

6. Step 4: Prove the gate with fixtures

Three fixtures are enough to trust the gate, and all three are cheap to maintain.

# tests/test_docs_claims.py
import subprocess, sys

def run(path):
    return subprocess.run([sys.executable, "-m", "tools.docs_claims.resolve", path], capture_output=True, text=True)

def test_clean_page_passes():
    assert run("tests/fixtures/clean.md").returncode == 0

def test_unknown_flag_fails():
    result = run("tests/fixtures/unknown_flag.md")
    assert result.returncode == 1 and "--fast-mode" in result.stdout

def test_owned_claim_in_model_block_fails():
    result = run("tests/fixtures/owned_claim.md")
    assert result.returncode == 1 and "guarantee" in result.stdout.lower()

Run python -m tools.docs_claims.index_build > docs_claims_index.json before the tests, and cache the index between jobs. When a fixture fails for the wrong reason, the message tells you whether the index or the extraction is wrong, which is the debugging shortcut that makes this pipeline tolerable on a busy repository.

7. Where the model fits, and where it does not

Drafting is the one step that benefits from a hosted model and a machine that is not your laptop, because regeneration runs on a schedule and should not depend on whoever is awake. I run the drafting job on MonkeyCode's free server option and call its free model access for the rewrite step, keeping the index builder and the resolver inside my own CI. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The division is deliberate: the model receives the index as its only product-surface context, and every atom it emits returns to a verifier I control.

What the model may draft: derived sentences whose atoms resolve, narrative framing around them, and rephrasings of existing verified text. What a human must own: guarantees, deprecation commitments, support windows, security statements, pricing, and anything a customer could reasonably quote back at you in a dispute.

Limitations and who should skip this

Single-file scripts with stable CLI surfaces gain little, because a human edit is faster than maintaining an index generator. Teams that publish prose unchanged for months should also skip it, since a gate nobody triggers is just build time. Documentation that intentionally describes planned or unreleased behavior needs a third class, or owned markers on nearly every paragraph, and that friction is real.

The resolver is syntactic: it proves a flag exists, not that the sentence describing it is true. Cross-field semantics, defaults, and interaction effects still need a reviewer, so treat a green build as permission to stop worrying about identifier drift, not as correctness.

If you want to try the drafting half without paying for inference, the free model access and free server option are enough to run this loop on a small repository; keep the verifier in your own CI and let it fail loudly.