The rule I apply is short: every fenced block in a documentation file is either executed by CI or labeled illustrative with a named owner. The model may propose the code in either case, but only a human may assert what that code prints. That single constraint removes the most common defect in model-drafted docs, which is a confident example that has never run.
This is a narrower claim than "generate your documentation", and it is deliberately narrower, because the artifact below fits on one screen and blocks a whole class of review arguments. The executor is plain Python with no dependencies, and the gate treats an unlabeled fence as a policy violation rather than a style nit.
1. Why the fence, and not the paragraph, is the unit of trust
A wrong sentence in prose costs a reader a minute of confusion, while a wrong command costs them an afternoon of debugging their own environment. Code blocks therefore carry a different burden of proof than the paragraphs around them, and reviewers tend to treat prose and code as one artifact. Separating the two lets you apply human attention where the cost of error is concentrated.
Models are genuinely good at the connective prose in a tutorial: the framing sentence, the "now that you have X" transition, the summary of what changed. They are unreliable about the observable consequences of a command, especially for CLIs, SDKs, and config schemas that changed after the training data. That asymmetry, not general quality, is what the gate encodes.
The third reason is review economics. A reviewer who is asked to verify six commands by hand will verify two and skim four, and a reviewer who is shown which blocks already ran will spend the saved time on the parts a machine cannot check. Gate first, review second.
2. Label every fence with one of three verbs
The convention needs to survive a diff without extra tooling, so the label lives in the fence info string and nowhere else.
| Label | Meaning | Who may author it |
|---|---|---|
runnable |
Executed in a throwaway directory; non-zero exit fails the build | Model may propose, human must confirm it runs |
output |
The stdout/stderr a human actually observed for the runnable block directly above |
Named human only |
illustrative owner=@handle |
A sketch that must never execute: pseudocode, partial config, a truncated JSON payload | Model may draft, owner vouches for shape |
Optional attributes stay minimal and mechanical: timeout=120 when a block is slow, and match=contains when the real output is noisy and only a substring is stable. Resist adding more attributes, because every attribute is a thing a reviewer must interpret under time pressure.
A useful side effect is that illustrative forces a decision. When a model invents a plausible curl example and the author cannot run it, the block becomes illustrative owner=@handle and the doc says so implicitly. Readers who copy it know they are on their own.
3. The executor: one file, standard library only
The parser walks lines, collects fences, and runs each runnable block in a fresh temporary directory so that one example cannot leak state into the next.
#!/usr/bin/env python3
"""docex: every fenced block in docs is either executed or explicitly labeled."""
from __future__ import annotations
import argparse, json, re, subprocess, sys, tempfile
from dataclasses import dataclass
from pathlib import Path
FENCE = re.compile(r"^```
(?P<lang>[\w+-]*)\s*(?P<meta>.*)$")
ATTR = re.compile(r"(\w+)=(\S+)|(?<!\w)(runnable|illustrative|output)(?!\w)")
@dataclass
class Block:
lang: str
meta: dict
flags: set
body: str
line: int
def parse(path: Path) -> list[Block]:
blocks: list[Block] = []
cur: Block | None = None
body: list[str] = []
for lineno, raw in enumerate(path.read_text(encoding="utf-8").splitlines(), 1):
if cur is None:
m = FENCE.match(raw)
if not m:
continue
meta, flags = {}, set()
for key, val, flag in ATTR.findall(m.group("meta")):
if flag:
flags.add(flag)
else:
meta[key] = val
cur, body = Block(m.group("lang"), meta, flags, "", lineno), []
elif raw.startswith("
```"):
cur.body = "\n".join(body)
blocks.append(cur)
cur = None
else:
body.append(raw)
return blocks
RUNNERS = {"python": ([sys.executable], ".py"), "bash": (["bash"], ".sh"), "sh": (["sh"], ".sh")}
def execute(blocks: list[Block], workdir: Path, default_timeout: int = 30):
results = []
for b in blocks:
if b.lang not in RUNNERS:
results.append((b, "unsupported-language", ""))
continue
prefix, ext = RUNNERS[b.lang]
script = workdir / f"snippet_{b.line}{ext}"
script.write_text(b.body, encoding="utf-8")
try:
p = subprocess.run(
prefix + [script.name], cwd=workdir, capture_output=True, text=True,
timeout=int(b.meta.get("timeout", default_timeout)),
)
except subprocess.TimeoutExpired:
results.append((b, "timeout", ""))
continue
results.append((b, "ok" if p.returncode == 0 else "nonzero-exit", p.stdout + p.stderr))
return results
def normalize(text: str) -> str:
return "\n".join(ln.rstrip() for ln in text.strip().splitlines() if ln.strip())
def matches(expected: Block, actual: str) -> bool:
want, got = normalize(expected.body), normalize(actual)
return want in got if expected.meta.get("match") == "contains" else want == got
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("paths", nargs="+", type=Path)
ap.add_argument("--report", type=Path, default=Path("docex-report.json"))
args = ap.parse_args()
violations, failures, executed, compared = [], [], 0, 0
for path in args.paths:
blocks = parse(path)
for b in blocks:
if not b.flags:
violations.append(f"{path}:{b.line}: fence has no label")
elif "illustrative" in b.flags and "owner" not in b.meta:
violations.append(f"{path}:{b.line}: illustrative fence needs owner=@handle")
runnable = [b for b in blocks if "runnable" in b.flags]
if not runnable:
continue
with tempfile.TemporaryDirectory() as td:
results = execute(runnable, Path(td))
for i, (b, status, output) in enumerate(results):
executed += 1
if status != "ok":
failures.append({"file": str(path), "line": b.line, "status": status,
"output": output[-1500:]})
continue
stop = runnable[i + 1].line if i + 1 < len(runnable) else sys.maxsize
expected = next((x for x in blocks if "output" in x.flags and b.line < x.line < stop), None)
if expected is None:
continue
compared += 1
if not matches(expected, output):
failures.append({"file": str(path), "line": b.line, "status": "output-mismatch",
"expected": normalize(expected.body),
"actual": normalize(output)[-1500:]})
report = {"executed": executed, "compared": compared,
"violations": violations, "failures": failures}
args.report.write_text(json.dumps(report, indent=2), encoding="utf-8")
print(f"docex: executed={executed} compared={compared} "
f"failures={len(failures)} violations={len(violations)}")
if failures:
return 1
return 2 if violations else 0
if __name__ == "__main__":
raise SystemExit(main())
Two design choices deserve a note. Blocks run in document order, so a tutorial that builds on itself works, but each file gets its own temporary directory, so a stale dist/ from a previous example cannot rescue a broken one. And output comparison normalizes trailing whitespace and blank lines only, because stripping more than that is how a gate quietly stops checking anything.
4. Wire it into CI and read the exit codes
Run the tool over tracked Markdown rather than a glob, so deleted files cannot keep passing from a stale checkout.
python tools/docex.py $(git ls-files 'docs/**/*.md' 'README.md') --report docex-report.json
| Exit code | Meaning | Merge policy |
|---|---|---|
| 0 | Every block labeled, every runnable block exited 0, outputs matched | Allowed |
| 1 | A block failed, timed out, or printed something other than its output
|
Blocking |
| 2 | A fence is unlabeled, or illustrative has no owner |
Blocking |
Exit code 2 is the one that catches generated documentation at scale, because a model asked to "add an example" will add a fence without a label every time. Treat that as expected behavior and let the gate correct it, instead of writing another paragraph in the prompt that the model will ignore on the next run.
When a mismatch appears, the report tells you whether the code broke or the documentation lied, and those are different fixes. A nonzero-exit on a block that ran fine last week usually means a dependency or CLI flag changed; an output-mismatch with an unchanged exit code usually means the prose around it is now stale.
5. Decide who owns what before the first draft
| Documentation element | Model drafts | Human owns |
|---|---|---|
| Explanatory prose between examples | Yes | Reviewer signs off on accuracy |
| Runnable example code | Yes, as a proposal | Confirms it executes and accepts the output block |
| Expected output, error text, exit codes | No | The person who ran it |
illustrative sketches |
Yes | Named owner vouches for shape |
| Version numbers, deprecation dates, support windows | No | Release owner |
If you want one sentence to give a new contributor, use this one: the model may write the paragraph that introduces a command, and the human writes the block that claims what the command does. Everything else in the table is bookkeeping around that boundary.
6. Where free model access and the free server option fit
The drafting half of this workflow is a good match for the free model access in MonkeyCode, because the tasks are small, well-scoped, and cheap to discard: rewrite an introductory paragraph, propose a second example for a config file, or convert a changelog entry into a migration step. The executor half is a good match for the free server option, since the job needs a Python interpreter, a temporary directory, and nothing else. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
What the free tier does not change is the boundary itself. A free model still cannot observe what your CLI prints on a machine it has never seen, so the human-owned output block stays human-owned regardless of who paid for the tokens. Budget accordingly: drafting gets cheaper, verification does not.
7. Limitations and who should not adopt this
Executing documentation code means your CI executes code from pull requests, which is a security decision, not a formatting one. Do not run this on untrusted forks without a disposable sandbox, and do not treat a free server option as a security boundary, because it is a convenience for trusted drafts and not an isolation guarantee.
Some examples resist execution entirely: anything requiring cloud credentials, paid accounts, hardware, or a GUI. Those belong in illustrative with an owner, and the honest label is more useful to readers than a fake green check.
Finally, skip this approach if your documentation is mostly conceptual, if your quickstart cannot run inside a temporary directory, or if you have no CI that can tolerate a multi-minute job. For a purely narrative docs site, the label table alone is still worth adopting, even without the executor.
If you build the executor, the report format is the part worth copying: file, line, status, expected, actual. That output is what turns a documentation review from an argument about tone into a diff about facts.
Top comments (0)