DEV Community

Avery Lin
Avery Lin

Posted on

Cite-or-Own: An Allowlist for Evidence-Bound Documentation Drafts

Generated documentation stays trustworthy when every model-written section cites a repository artifact that a checker can open. Claims that lack that evidence remain human work, even when drafting tokens are inexpensive or locally hosted. A cite-or-own allowlist encodes the split as YAML, markers, and a CI command. Remove any vendor name from this workflow and the same ownership rule still holds.

Ownership is decided before a prompt exists

Blanket generation treats a README as one blob, which hides where invented claims usually appear. Support hours, compatibility promises, and security posture rarely live in a single source file. Procedure steps, flag tables, and endpoint summaries often do, because code and schemas already constrain them. An allowlist makes that split explicit before any draft is requested.

The contract is a YAML map from heading paths to two fields: owner and evidence. Sections marked model must list paths or commands that a checker can resolve on disk. Sections marked human must contain a human marker and must never contain a model draft fence. That rule is mechanical, so a pull request can fail without a subjective review argument.

Do not ask a model to classify ownership, because the classification is itself a product claim. A human maintainer fills the YAML after an inventory script lists headings. Missing contract rows should fail closed, since an unlabeled section is an unlabeled claim.

Decision table: what a model may draft

Use the table as a starting policy, then tighten the human column for regulated products. If evidence cannot be named as one path or one committed snapshot, the heading is not model-eligible yet.

Section kind Typical evidence Owner Model may draft?
API path and method tables OpenAPI or route modules model Yes, after a schema parse
CLI flag lists Committed --help snapshot model Yes, if the snapshot is versioned
Install commands Package manifest or Makefile model Yes, quoted from those files
Runnable examples Files under examples/ model Yes, only if CI executes them
Architecture intent No in-repo proof required human No
SLA and support hours Policy outside the repo human No
Security slogans such as "we encrypt X" Threat model, not adjectives human No
Migration warnings Changelog plus human review human Notes only; human owns the final text

The table is not a universal standard for every documentation set. Teams that publish contractual language should expand human-owned rows rather than reuse marketing copy from a draft. Evidence that needs production access at check time is also a poor fit, because CI should not scrape live systems to bless prose.

Numbered workflow

1. Inventory headings before anyone generates prose

Walk the documentation tree and emit a heading inventory instead of editing paragraphs first. A small script can list H2 and H3 paths so the contract file stays aligned with the pages. Review the skeleton in a normal diff, then fill owner and evidence by hand.

python3 tools/docs_allowlist.py inventory --root docs --out docs/ownership.yml
Enter fullscreen mode Exit fullscreen mode

Keep docs/ownership.yml in version control beside the markdown it governs. When a writer adds a heading without a contract row, the next check fails closed. That failure is cheaper than discovering an unsupported claim after a customer quotes the page.

2. Attach evidence a checker can open without a network

Evidence must be a repository path, a documented command with a committed snapshot, or an explicit empty list with owner: human. Vague citations such as "the backend" or "recent discussions" are not evidence. Prefer snapshots under docs/_evidence/ so the checker remains deterministic in CI.

# docs/ownership.yml
version: 1
pages:
  - path: docs/cli.md
    sections:
      - heading: "Command reference"
        owner: model
        evidence:
          - type: command
            run: ["python3", "-m", "mycli", "--help"]
            snapshot: docs/_evidence/cli-help.txt
      - heading: "When to use this tool"
        owner: human
        evidence: []
        marker: "<!-- human-owned -->"
Enter fullscreen mode Exit fullscreen mode

If a command snapshot drifts, regenerate the snapshot in a separate change, then allow a model to redraft only the matching fenced section. Humans still write "When to use this tool", which encodes product judgment rather than flag text. Mixing those jobs in one prompt is how unsupported promises enter a page.

3. Draft only allowlisted sections, with bounded context

Copy the cited snapshot into a bounded prompt and request a replacement for that heading block alone. Do not paste human-owned remainder text into the prompt context, because the model will treat it as editable source. A free local or hosted drafting path is enough, because the task is transformation of cited text, not invention of policy.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can host that bounded drafting step without adding a paid inference dependency. The allowlist still applies if you use another runner; the product is a convenience, not the ownership rule.

Wrap model output in fences the checker understands, and keep human sections on a different marker:

<!-- human-owned -->
## When to use this tool

Use this CLI when a script must print the same fields a human already selected in the dashboard.
Do not promise multi-region failover here; that claim has no evidence file in this repository.

<!-- model-draft:start id="cli-command-reference" -->
## Command reference

| Flag | Meaning |
| --- | --- |
| `--json` | Print machine-readable output |
<!-- model-draft:end -->
Enter fullscreen mode Exit fullscreen mode

A heading may not carry both markers. If a migration section needs a model-prepared changelog digest, keep the digest fenced and leave the customer-facing warning outside the fence for a human author.

4. Fail the build on ownership leaks

The checker should block four classes of error, each mapping to a concrete leak. Unlabeled headings, model fences inside human sections, missing evidence files, and stale snapshot hashes are all failures. Optional warnings may note human sections that still contain TODO tokens, but warnings must not replace the hard checks.

python3 tools/docs_allowlist.py check --root docs --contract docs/ownership.yml
python3 tools/docs_allowlist.py test   # runs the fixture plan below
Enter fullscreen mode Exit fullscreen mode

Run the same commands in CI on documentation paths only, so a docs failure stays readable. Do not hide the job inside a large unit-test matrix where ownership errors look like flaky application tests.

5. Split review, not just generation

Model-only diffs under fences can follow the code-owner path for the evidence files they cite. Human-owned sections should require a reviewer who can defend the claim in a customer conversation. That split reduces review load without pretending a model can sign a promise. If a change touches both kinds of section, review the human hunks first and treat the fenced draft as secondary.

Artifact: checker, fixtures, and a test plan

The script below is an example you must adapt to your heading parser. It is not a documentation platform. It inventories headings, compares snapshot hashes, and enforces marker rules.

#!/usr/bin/env python3
"""docs_allowlist.py — example cite-or-own checker. Not production-hardened."""
from __future__ import annotations

import argparse, hashlib, json, re, subprocess, sys
from pathlib import Path

try:
    import yaml
except ImportError:
    yaml = None

HEADING = re.compile(r"^(#{2,3})\s+(.+?)\s*$", re.M)
FENCE_START = re.compile(r"<!--\s*model-draft:start\s+id=\"([^\"]+)\"\s*-->")
FENCE_END = "<!-- model-draft:end -->"
HUMAN_MARK = "<!-- human-owned -->"

def sha256(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()

def load_contract(path: Path) -> dict:
    if yaml is None:
        raise SystemExit("PyYAML is required for this example checker")
    return yaml.safe_load(path.read_text(encoding="utf-8"))

def inventory(root: Path) -> list[dict]:
    pages = []
    for md in sorted(root.rglob("*.md")):
        text = md.read_text(encoding="utf-8")
        headings = [m.group(2).strip() for m in HEADING.finditer(text)]
        pages.append({"path": str(md), "headings": headings})
    return pages

def section_body(text: str, heading: str) -> str:
    parts = re.split(r"(?=^#{2,3}\s+)", text, flags=re.M)
    for part in parts:
        first = part.splitlines()[0] if part.strip() else ""
        if first.lstrip("# ").strip() == heading:
            return part
    return ""

def check(root: Path, contract: dict) -> list[str]:
    errors = []
    indexed = {p["path"]: p for p in contract.get("pages", [])}
    for md in sorted(root.rglob("*.md")):
        rel = str(md).replace("\\", "/")
        if rel not in indexed:
            errors.append(f"unlabeled page: {rel}")
            continue
        text = md.read_text(encoding="utf-8")
        found = {m.group(2).strip() for m in HEADING.finditer(text)}
        for section in indexed[rel].get("sections", []):
            heading = section["heading"]
            if heading not in found:
                errors.append(f"missing heading {heading!r} in {rel}")
                continue
            body = section_body(text, heading)
            owner = section["owner"]
            if owner == "human":
                if HUMAN_MARK not in body:
                    errors.append(f"human section {heading!r} lacks marker in {rel}")
                if FENCE_START.search(body):
                    errors.append(f"model fence inside human section {heading!r} in {rel}")
            elif owner == "model":
                if not FENCE_START.search(body) or FENCE_END not in body:
                    errors.append(f"model section {heading!r} lacks draft fences in {rel}")
                for ev in section.get("evidence", []):
                    snap = Path(ev["snapshot"])
                    if not snap.is_file():
                        errors.append(f"missing snapshot {snap} for {heading!r}")
                        continue
                    if ev.get("type") == "command":
                        proc = subprocess.run(ev["run"], check=False, capture_output=True, text=True)
                        current = (proc.stdout or "") + (proc.stderr or "")
                        if sha256(current.encode()) != sha256(snap.read_bytes()):
                            errors.append(f"stale snapshot {snap} for {heading!r}")
            else:
                errors.append(f"unknown owner {owner!r} for {heading!r}")
        contracted = {s["heading"] for s in indexed[rel].get("sections", [])}
        for extra in found - contracted:
            errors.append(f"unlabeled heading {extra!r} in {rel}")
    return errors

def write_fixtures(tmp: Path) -> Path:
    docs = tmp / "docs"
    ev = docs / "_evidence"
    ev.mkdir(parents=True)
    (ev / "cli-help.txt").write_text("usage: mycli [--json]\n", encoding="utf-8")
    (docs / "cli.md").write_text(
        "<!-- human-owned -->\n## When to use this tool\nUse it for exports.\n\n"
        "<!-- model-draft:start id=\"cli-command-reference\" -->\n"
        "## Command reference\n`--json` prints JSON.\n<!-- model-draft:end -->\n",
        encoding="utf-8",
    )
    contract = tmp / "ownership.yml"
    contract.write_text(
        "pages:\n  - path: docs/cli.md\n    sections:\n"
        "      - heading: When to use this tool\n        owner: human\n        evidence: []\n"
        "      - heading: Command reference\n        owner: model\n        evidence:\n"
        "          - type: file\n            snapshot: docs/_evidence/cli-help.txt\n",
        encoding="utf-8",
    )
    return contract

def run_tests() -> int:
    import tempfile
    with tempfile.TemporaryDirectory() as raw:
        tmp = Path(raw)
        contract_path = write_fixtures(tmp)
        # Fixture paths are written relative to tmp; check from tmp as cwd-equivalent root.
        errors = check(tmp / "docs", load_contract(contract_path))
        # Rewrite page paths in the fixture contract to match check() keys.
        data = load_contract(contract_path)
        data["pages"][0]["path"] = str((tmp / "docs" / "cli.md").resolve())
        # Use relative paths consistent with rglob output.
        data["pages"][0]["path"] = str(tmp / "docs" / "cli.md")
        errors = check(tmp / "docs", data)
        print(json.dumps({"fixture_errors": errors}, indent=2))
        return 1 if errors else 0

def main() -> int:
    parser = argparse.ArgumentParser(description="Cite-or-own docs allowlist")
    sub = parser.add_subparsers(dest="cmd", required=True)
    inv = sub.add_parser("inventory")
    inv.add_argument("--root", type=Path, required=True)
    inv.add_argument("--out", type=Path, required=True)
    chk = sub.add_parser("check")
    chk.add_argument("--root", type=Path, required=True)
    chk.add_argument("--contract", type=Path, required=True)
    sub.add_parser("test")
    args = parser.parse_args()
    if args.cmd == "inventory":
        pages = inventory(args.root)
        args.out.write_text(json.dumps(pages, indent=2), encoding="utf-8")
        print(f"wrote {len(pages)} pages to {args.out}")
        return 0
    if args.cmd == "test":
        return run_tests()
    errors = check(args.root, load_contract(args.contract))
    for err in errors:
        print(f"ERROR: {err}")
    print(f"{len(errors)} error(s)")
    return 1 if errors else 0

if __name__ == "__main__":
    sys.exit(main())
Enter fullscreen mode Exit fullscreen mode

Fixture test plan

Label these cases so a teammate can rerun them without guessing intent. The test subcommand only proves the happy-path fixture; extend it before you trust the checker on a live tree.

  1. Happy path: human marker present, model fences present, snapshot file exists, zero errors.
  2. Unlabeled heading: add an ## SLA section with no contract row and expect a closed failure.
  3. Ownership leak: place a model-draft fence inside a human section and expect a failure.
  4. Missing snapshot: delete docs/_evidence/cli-help.txt and expect a missing-evidence error.
  5. Stale command snapshot: change --help output without updating the file and expect a stale-hash error when type: command is used.

Run inventory against a copy of your docs/ tree before you enable the check on the default branch. Empty inventories usually mean the heading regex does not match your title style, not that the tree lacks documentation.

Limitations

The checker does not prove that a fenced draft is correct, only that a cited file exists and markers match the contract. A model can still misread a snapshot and emit a wrong flag description. Snapshot hashing also fails if the command is nondeterministic, including help text that embeds dates, versions, or terminal color codes.

Heading comparison is literal. If a writer changes Command reference to Commands, the contract row no longer matches and CI fails closed. That strictness is intentional, but it will annoy teams that rewrite titles for tone. Nested tabs, generated Docusaurus MDX, and HTML headings are outside this example parser.

The workflow also does not replace legal review. A human-owned marker is a process control, not a signature that a claim is safe to publish. If your documentation is part of a contract, keep counsel on those pages regardless of how clean the allowlist looks.

Who should not use this approach

Skip this allowlist if the repository has no citable artifacts for the pages you publish. Changelog-free product blogs, vision decks, and incident narratives are human documents; generating them under model fences creates false confidence. Skip it if CI cannot run the documented commands, because stale-snapshot detection would require production credentials.

Single-maintainer wikis that change once a quarter may not need YAML contracts. The overhead is easier to justify when several people merge documentation and at least one source of truth already exists in code. If your team cannot name a human owner for claim sections, fixing staffing matters more than adding a checker.

What this changes in review

Cite-or-own does not make generated documentation cheap in every sense. It makes unsupported claims expensive again, which is the point. Model drafts remain useful where a file, schema, or committed snapshot already says the same thing in a stricter language. Human authors keep the sentences a customer would treat as a promise.

If you already isolate drafting this way, a free model path is optional rather than required. The durable artifact is the contract, the markers, and the failing check, not the particular runner that filled a fence.

Top comments (0)