Anchor Coverage: A CI Gate for Docs a Model May Draft but Never Own

Consider a docs PR that adds a retry section. The prose is tidy, the table is aligned, and one line reads: "Set --retry-limit to 5 to cap retries." The flag was renamed two releases ago, and nothing in CI noticed, because the pipeline lints Markdown formatting, not meaning.

That example is illustrative, not a customer story. The shape of it is common though: the review cost of generated prose is paid by humans reading sentences one at a time, and that cost does not shrink when generation gets cheaper.

DEV discussion over the last few weeks has swung between enthusiasm for what models can produce and fatigue from reviewing the output. Both reactions point at the same missing piece — a mechanical check between "the model wrote it" and "a human approved it."

Reviewing prose does not scale, checking anchors does

Most "AI docs" workflows fail in one of two boring ways. Either the model is allowed to touch files it should not, or it is allowed to state facts nobody can trace. Both are diff-shaped problems, and diffs can be gated.

The gate I use asks a narrow question: for every drafted paragraph, is there a machine-readable artifact behind it? Not "is it correct" — that is still a human judgment. Just "is it traceable," which is a question a script can answer in under a second.

Three lanes: extract, draft, own

Split every documentation change into three lanes before any model sees the file.

• Extract — facts pulled from sources that already exist: --help output, an OpenAPI schema, a config reference, exit codes in a test fixture. A script owns this lane.
• Draft — connective prose that explains extracted facts in order, with examples drawn from the fixtures. A model may own this lane, inside delimited blocks only.
• Own — commitments a human makes: support windows, security posture, deprecation dates, pricing, anything a reader could hold you to. A model may not enter this lane at all.

The important part is not the labels. It is that lanes 1 and 3 are enforced by tooling, so lane 2 can be regenerated freely without a reviewer having to re-audit the whole file.

The artifact: anchor_gate.py

Drafts are written only between markers. Every fact inside a draft block must carry an anchor like {{claim:cli.retry-limit}}. The gate resolves anchors against claims.json and refuses naked numbers.

#!/usr/bin/env python3
"""anchor_gate.py - fail the build when drafted prose outruns its evidence.

Usage:
    python anchor_gate.py docs/ claims.json --max-orphans 0.20

Exit codes: 0 = pass, 1 = gate failed, 2 = bad input.
"""
from __future__ import annotations

import json
import re
import sys
from pathlib import Path

START = re.compile(r"<!--\s*draft:start\s+id=([\w.-]+)\s*-->")
END = re.compile(r"<!--\s*draft:end\s*-->")
ANCHOR = re.compile(r"\{\{claim:([\w.-]+)\}\}")
NUMBER = re.compile(r"\d[\d,_]*(?:\.\d+)?")


def draft_blocks(text: str) -> list[tuple[str, list[str]]]:
    blocks, block_id, buf = [], None, []
    for line in text.splitlines():
        start = START.search(line)
        if start:
            block_id, buf = start.group(1), []
            continue
        if END.search(line) and block_id is not None:
            blocks.append((block_id, buf))
            block_id = None
            continue
        if block_id is not None:
            buf.append(line)
    return blocks


def paragraphs(lines: list[str]) -> list[str]:
    out, buf = [], []
    for line in lines + [""]:
        if line.strip():
            buf.append(line.strip())
        elif buf:
            out.append(" ".join(buf))
            buf = []
    return out


def main() -> int:
    if len(sys.argv) < 3:
        print(__doc__)
        return 2
    docs_dir, claims_path = Path(sys.argv[1]), Path(sys.argv[2])
    claims = json.loads(claims_path.read_text(encoding="utf-8"))
    budget = 0.20
    if "--max-orphans" in sys.argv:
        budget = float(sys.argv[sys.argv.index("--max-orphans") + 1])

    total = orphans = 0
    errors: list[str] = []

    for md in sorted(docs_dir.rglob("*.md")):
        for block_id, lines in draft_blocks(md.read_text(encoding="utf-8")):
            for para in paragraphs(lines):
                total += 1
                ids = ANCHOR.findall(para)
                unknown = sorted({i for i in ids if i not in claims})
                if unknown:
                    errors.append(f"{md.name}:{block_id}: unknown claim(s) {unknown}")
                if not ids:
                    orphans += 1
                    if NUMBER.search(para):
                        errors.append(
                            f"{md.name}:{block_id}: number without an anchor -> {para[:60]!r}"
                        )

    rate = orphans / total if total else 0.0
    print(f"draft paragraphs: {total}  orphans: {orphans}  rate: {rate:.2f}")
    if rate > budget:
        errors.append(f"orphan rate {rate:.2f} exceeds budget {budget:.2f}")
    for err in errors:
        print(f"FAIL {err}")
    return 1 if errors else 0


if __name__ == "__main__":
    sys.exit(main())

The orphan budget matters more than the hard errors. Prose always contains sentences with no fact in them — transitions, framing, a definition. A 20% ceiling keeps the gate from becoming a lint that everyone disables in week two, while still catching the paragraph that quietly invents a default.

Generate claims.json, do not type it

A hand-maintained claims file rots exactly as fast as the docs it is meant to protect. Derive it from the same artifacts the code already trusts.

{
  "cli.retry-limit": { "value": "3", "source": "app.py --help" },
  "schema.timeout_ms": { "value": "1500", "source": "openapi.yaml#/components/schemas/Retry" }
}

A twenty-line extractor over argparse help text and schema keys is usually enough to start. If a fact cannot be extracted, that is a signal: either it belongs in the own lane, or the source of truth is missing and the doc should not claim it yet.

The list a model must never write

Write this list into the repo as human_owned.txt and have the generator refuse to touch those sections.

Content	Model may draft?	Human owner
Flag names, defaults, exit codes	Yes, anchored	Extraction script
Example invocations from fixtures	Yes, anchored	CI fixture suite
Deprecation and support windows	No	Product or support owner
Security, data retention, compliance	No	Security owner
Pricing, quotas, licensing	No	Whoever signs the contract
"Why we chose this design" narrative	Draft only, must be edited	The engineer who decided

That last row is where most teams get it wrong. A model can produce a plausible rationale for a design it has never seen, and plausible rationale is the hardest kind of wrong to review.

Running the draft pass where it costs nothing to re-run

Regeneration is the whole point of this workflow: when the schema changes, you re-draft instead of hand-patching. That only stays true if the draft pass is cheap enough to run on every PR rather than monthly.

MonkeyCode is one option here — the project is described as open source, with free model access and a free server option, which is what makes a per-PR draft pass financially uninteresting to argue about. The operator of this account states a free tier of 10 million tokens; treat that as a starting figure to verify against the project's current terms, not a capacity plan.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free server option fits the other half of the loop: the gate and the extractor need somewhere to run as a check. A draft job that calls a model, writes only inside marked blocks, then runs anchor_gate.py is a small, boring workload — which is exactly what you want on a free box rather than on the cluster that runs production traffic.

Estimate your own burn instead of trusting a round number. Token counts vary by tokenizer, but a rough planning heuristic for English prose plus code is four characters per token:

def rough_tokens(text: str) -> int:
    # planning heuristic only; measure with your model's tokenizer
    return len(text) // 4

Run that over the prompts for your five largest docs, multiply by regenerations per week, and you have a number you can defend. If it does not fit a free tier, gate the draft pass to changed pages only — the extractor already tells you which anchors moved.

A four-case test plan for the gate itself

A gate nobody tests becomes a gate that passes everything.

1. Clean block — one paragraph, one valid anchor, no numbers outside it. Expect exit 0.
2. Unknown anchor — {{claim:cli.ghost-flag}} not in claims.json. Expect exit 1 with the block id.
3. Naked number — "retries default to 5" with no anchor. Expect exit 1.
4. Honest orphan — a framing sentence with no facts, budget set to 0.0. Expect exit 1 only on the rate line, which proves the budget is actually read.

Add a fifth once you trust it: regenerate the same block twice with the same inputs and diff the output. Non-determinism is not a bug, but a block that changes wording every run will churn your git history and train reviewers to ignore the diff.

Limitations, and who should skip this

Anchor coverage measures traceability, not truth. A wrong extractor produces confidently anchored lies, and the gate will pass them.

The approach also assumes a machine-readable source of truth exists. If your product's behavior is defined by a PDF, a support inbox, or one person's memory, the extract lane is empty and this workflow has nothing to stand on.

Skip it if your docs are primarily narrative — architecture decisions, incident write-ups, tutorials where the value is the author's judgment. And skip it if a formal review process already binds every sentence to a signer; adding anchors on top is duplicate work with extra syntax.

Start with one directory

Pick the docs directory with the most flags and defaults, write the marker comments, and run the gate in report-only mode for a week. The number it prints — orphan rate — is the most useful documentation metric I know of that a script can compute without understanding a single sentence.

If you try the per-PR draft pass, run it on a small page set first and keep the human-owned list longer than you think you need.