The prose in a documentation page is rarely what misleads a reader; the code example is. A model can produce a careful paragraph about retry semantics and still hand you a snippet that stopped importing two releases ago, and ordinary review will not catch it because the snippet looks correct at a glance. This post describes a workflow in which examples are never drafted at all: each fenced block is pinned to a region of a real test file, verified by hash during the build, and spliced into the page by a generator, while a model writes only the connective prose around them. The division is simple to state and awkward to fake: the repository owns everything a reader might copy, and the model owns everything a reader might skip.
1. The failure mode is silent, so make it loud
Examples decay for boring reasons: a function gains a required argument, a module is renamed, an error type moves into a subpackage. Markdown tooling does not object, because a fence is just text between three backticks, and a human reviewer reading for accuracy will approve the paragraph and skim the code inside it. The question worth automating is narrow and cheap to answer: does this block still match a region of a file that the test suite actually executes? If the answer is checked on every build, example drift stops being a documentation problem and becomes a red build, which is the only kind of documentation problem teams reliably fix.
The design constraint that follows is the interesting part. If the model can edit both the example and its hash, the check proves nothing, because the model can keep them consistent while both drift from reality. So hashes live in a lock file that is written only from executed source files, and the check runs in a mode that refuses to rewrite it.
2. Step 1 — Give every example a source pointer
Replace hand-written fences with fences that carry provenance in the info string. CommonMark allows extra words after the language name, and most renderers ignore them, so the page still renders normally.
python source="examples/auth/test_login.py#L12-L19"
markdown
Three rules make this survivable over time. First, the pointer must reference a test file, not an application module, so a passing suite guarantees the block is at least imported and called. Second, the region must be a contiguous line range, which keeps the extraction dumb and reviewable. Third, no block may appear in a page without source=; that rule alone removes the class of examples that were typed by hand into a chat window and pasted into the docs.
3. Step 2 — Freeze the regions in a lock file
The lock file maps a stable block identifier to the source path, the line range, and the digest of the extracted text. It is generated, committed, and reviewed like any other build artifact.
{
"docs/quickstart.md#block-3": {
"source": "examples/auth/test_login.py",
"lines": [12, 19],
"sha256": "sha256:0f3c…"
}
}
Keeping the digest out of the page is deliberate. A contributor regenerating the lock file is making a visible diff that a reviewer can connect to a code change, whereas a digest embedded in prose invites silent self-consistency. If your team already pins generated API references, this is the same discipline applied to human-facing snippets.
4. Step 3 — Write the verifier
The script below walks Markdown files, extracts every fenced block that declares a source, recomputes the digest, and compares it against the lock file. Treat it as a starting point and test it against your own repository layout before wiring it into CI.
#!/usr/bin/env python3
"""docs_examples.py - verify that fenced docs blocks still match tested source."""
import argparse, hashlib, json, pathlib, re, sys
FENCE = re.compile(r"^```
(?P<lang>[\w+-]*)(?P<meta>[^\n]*)\n(?P<body>.*?)^
```$",
re.MULTILINE | re.DOTALL)
POINTER = re.compile(r'source="(?P<path>[^"#]+)(?:#L(?P<start>\d+)-L(?P<end>\d+))?"')
def blocks(markdown_text):
for index, match in enumerate(FENCE.finditer(markdown_text), start=1):
pointer = POINTER.search(match.group("meta") or "")
yield index, pointer, match.group("body")
def extract(root, pointer):
lines = (root / pointer.group("path")).read_text(encoding="utf-8").splitlines()
start = int(pointer.group("start") or 1)
end = int(pointer.group("end") or len(lines))
return "\n".join(lines[start - 1:end]) + "\n"
def digest(text):
return "sha256:" + hashlib.sha256(text.encode("utf-8")).hexdigest()
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--root", default=".")
parser.add_argument("--lock", default="docs/examples.lock.json")
parser.add_argument("--write", action="store_true")
args = parser.parse_args()
root = pathlib.Path(args.root)
lock_path = root / args.lock
lock = json.loads(lock_path.read_text()) if lock_path.exists() else {}
seen, failures = {}, []
for page in sorted(root.glob("docs/**/*.md")):
for index, pointer, body in blocks(page.read_text(encoding="utf-8")):
key = f"{page.relative_to(root).as_posix()}#block-{index}"
if pointer is None:
failures.append(f"{key}: fenced block without a source pointer")
continue
try:
expected = digest(extract(root, pointer))
except FileNotFoundError:
failures.append(f"{key}: source {pointer.group('path')} not found")
continue
seen[key] = {"source": pointer.group("path"),
"lines": [int(pointer.group("start") or 1),
int(pointer.group("end") or 0)],
"sha256": expected}
if key in lock and lock[key]["sha256"] != expected:
failures.append(f"{key}: block drifted from {pointer.group('path')}")
if digest(body) != expected:
failures.append(f"{key}: published text differs from source")
for key in set(lock) - set(seen):
failures.append(f"{key}: locked entry has no matching block")
if args.write and not failures:
lock_path.write_text(json.dumps(seen, indent=2, sort_keys=True) + "\n")
return 0
for failure in failures:
print(failure, file=sys.stderr)
return 1 if failures else 0
if __name__ == "__main__":
raise SystemExit(main())
The second comparison is the one people forget: the digest of the published block must equal the digest of the source region. Without it, a page can point at the right file and still contain paraphrased code that nobody executed.
5. Step 4 — Splice examples, do not generate them
With pointers in place, the generator becomes trivial and the model's job shrinks. Ask for prose that contains placeholder tokens such as <!-- example:examples/auth/test_login.py#L12-L19 --> and let a small build step replace each token with the current extracted region. A twelve-line splice is enough:
import re, pathlib
TOKEN = re.compile(r"<!-- example:(?P<path>[^#>]+)#L(?P<a>\d+)-L(?P<b>\d+) -->")
def render(page_text, root=pathlib.Path(".")):
def swap(match):
lines = (root / match.group("path")).read_text().splitlines()
body = "\n".join(lines[int(match.group("a")) - 1:int(match.group("b"))])
return f"```
{% endraw %}
python source=\"{match.group('path')}#L{match.group('a')}-L{match.group('b')}\"\n{body}\n
{% raw %}
```"
return TOKEN.sub(swap, page_text)
The practical effect is that a model never sees example bodies it could rewrite, and a reviewer never approves code that was not executed. Prose quality still varies, but the failure you ship to readers is no longer a snippet.
6. Step 5 — Fail on orphan fences
The verifier already rejects a block without source=, which covers the common case of a contributor pasting a snippet from a chat window. Two related gates are worth adding while the plumbing is fresh: reject pointers to files outside examples/ or tests/, and reject regions shorter than three lines, since single-line snippets usually mean someone wanted to show a call without writing the test that proves it. Both checks are a few lines each and they keep the convention from eroding under deadline pressure.
7. What the model drafts, and what a human signs
| Artifact | May be model-drafted | Must be human or test-owned | Mechanical check |
|---|---|---|---|
| Section headings and ordering | Yes | Sign-off on scope | Outline review |
| Connective prose and transitions | Yes | Accuracy of claims | Claim grep plus review |
| Code samples | No | Test file region | Digest match |
| Error messages and outputs | No | Captured from an execution | Digest match |
| Version and pricing statements | Draft only | Primary-source citation | Link check |
The table is the whole argument in compressed form: anything a reader can copy, run, or quote is owned by a file that a test or a primary source controls.
8. Where a free model tier fits, and where it does not
This workflow still needs prose, and prose is the part a model is genuinely good at. I draft section transitions and troubleshooting narratives using MonkeyCode's free model access, feeding it the heading list and the example identifiers but never the block bodies. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The availability claims I am relying on are operator-supplied: free model access and a free server option, and you should confirm the current terms yourself before making a tooling decision. The server option is the relevant half here if you would rather not stand up a separate runner for the splice and verification steps, though the scripts above run fine in any container that has your repository checked out.
What the free tier is not is a reason to loosen the gate. A cheaper draft does not make unreviewed code examples safer, and the entire point of the hash check is that it does not care how the surrounding prose was produced.
9. Limitations and who should skip this
The regex fence parser is deliberately naive: it will mis-handle nested fences, ~~~ delimiters, and fences inside HTML blocks. If your docs use those heavily, swap the regex for a CommonMark parser rather than extending the pattern. Line-range pinning also creates churn, because inserting a blank line above an example invalidates every downstream pointer in that file; keeping examples in small, dedicated files under examples/ is what makes the approach tolerable.
Skip this workflow if your documentation is mostly conceptual and contains few runnable snippets, if your examples necessarily span several files and cannot be reduced to one contiguous region, or if you cannot run a build step on documentation changes. It also fits poorly on teams without a test suite, because the guarantee rests entirely on somebody else's assertions.
10. Rollout checklist
- Add
source=to the ten most-copied fences and leave the rest for a follow-up; partial adoption still removes real failures. - Generate the lock file once from executed files and commit it with the pointer changes in a single reviewable diff.
- Run the verifier in check mode in CI and confirm it fails on a deliberately edited block before trusting it.
- Change the drafting prompt to request placeholder tokens instead of code, and check that no orphan fence reaches
main. - Revisit the gate when contributors start asking for exceptions; the first exception request is usually a legitimate signal that an example needs a test, not a waiver.
Top comments (0)