Model-generated documentation fails most often in its code examples, not in its prose, because prose can be plausible while examples must run. A doc that describes a function incorrectly is a readability problem, but an example that crashes on copy-paste destroys reader trust. The practical fix is to treat every annotated code block as a test case and execute it in CI. MonkeyCode's free model access drafts the docs, and its free server option runs the verification loop on every change. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Why examples rot while prose passes review
Human reviewers are good at judging whether a sentence is clear, but they are bad at mentally executing a snippet with a dozen imports. A model can write a grammatical paragraph about a method signature that no longer exists, and the error stays invisible until someone pastes it. Executing the example turns that invisible error into a numbered failure, which is the only format a review process can act on. This is not another claim-checking pass over the prose, because a claim can be true in words while its example still fails to run.
The workflow: five steps
Step 1 — Draft with a runnable convention. Ask the model to mark every example it wants you to trust with run in the fence info string, so a block opens as python run.
Step 2 — Extract with a parser. A small script reads the Markdown and collects every annotated block with its file name and character offset.
Step 3 — Execute in a subprocess. Each block runs with a timeout, so an infinite loop becomes a failure instead of a hung CI job.
Step 4 — Report as a table. The script prints one line per block with a pass, fail, or skip status, plus the tail of any error output.
Step 5 — Sign with a manifest. A human reviews the failures and records the disposition for each one in a YAML file that lives next to the docs.
The artifact: docrunner.py
The runner below is deliberately small, with no dependencies beyond the Python standard library, so it can run on any server or CI image.
#!/usr/bin/env python3
"""docrunner.py — extract annotated code blocks from Markdown and run them."""
import re
import subprocess
import sys
from pathlib import Path
BLOCK_RE = re.compile(
r"```
(?P<lang>\w+)(?P<run>\s+run)?\n(?P<body>.*?)
```",
re.DOTALL,
)
def extract_blocks(path: Path):
text = path.read_text(encoding="utf-8")
for match in BLOCK_RE.finditer(text):
if match.group("run"):
yield {
"lang": match.group("lang"),
"body": match.group("body").strip(),
"offset": match.start(),
}
def run_block(block: dict, cwd: Path) -> dict:
lang = block["lang"]
if lang == "python":
cmd = [sys.executable, "-c", block["body"]]
elif lang in {"bash", "sh"}:
cmd = ["bash", "-c", block["body"]]
elif lang == "node":
cmd = ["node", "-e", block["body"]]
else:
return {"status": "skipped", "reason": f"no runner for {lang}"}
try:
proc = subprocess.run(cmd, cwd=cwd, capture_output=True, text=True, timeout=30)
except subprocess.TimeoutExpired:
return {"status": "fail", "reason": "timeout after 30s"}
return {
"status": "pass" if proc.returncode == 0 else "fail",
"returncode": proc.returncode,
"stderr": proc.stderr[-500:],
}
def main() -> int:
docs_dir = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("docs")
cwd = Path(sys.argv[2]) if len(sys.argv) > 2 else Path(".")
failures = 0
for doc in sorted(docs_dir.rglob("*.md")):
for block in extract_blocks(doc):
result = run_block(block, cwd)
status = result["status"].upper()
print(f"[{status}] {doc}:{block['offset']} ({block['lang']})")
if result["status"] == "fail":
failures += 1
print(result.get("stderr") or result.get("reason"))
print(f"{'-' * 40}\nTotal failures: {failures}")
return 1 if failures else 0
if __name__ == "__main__":
sys.exit(main())
Usage is two arguments: the docs directory and the working directory where examples should execute.
python docrunner.py docs/ .
A generated doc with an annotated example looks like this; the import below is illustrative:
## Quickstart
```python run
from payments import create_client
client = create_client(api_key="test")
print(client.status())
```
Adding the runner to CI
The runner becomes useful only when it executes on every pull request, so the loop needs a permanent home. A minimal GitHub Actions job can call the script with two arguments and fail the build when any example fails. The same job can run on MonkeyCode's free server option, which removes the cost question from the decision to add the check. The job fails the build on the first failing example, so the author sees the error in the pull request instead of discovering it in production.
name: docs-examples
on: [pull_request]
jobs:
run-examples:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: python docrunner.py docs/ .
What the results mean
| Runner result | Most likely cause | Human action |
|---|---|---|
| pass | example matches the current code | keep, no further action |
| fail with exit code | wrong API, typo, or stale import | regenerate the block, then re-run |
| fail with timeout | infinite loop or network dependency | rewrite the example to be deterministic |
| skipped | language has no runner | add a runner or remove the run annotation |
Some failures will be environmental rather than textual, such as a network call that times out in CI but works locally. Treat those as a signal to make the example deterministic, not as a license to delete the check. A pass does not mean the example is good documentation, only that it is honest about what the code does today. The manifest turns the report into an ownership record:
# docs/sign-off.yaml
reviewed_by: "Avery Lin"
date: "2026-08-28"
examples:
- file: "docs/quickstart.md"
offset: 412
status: pass
- file: "docs/api.md"
offset: 88
status: fail
action: "regenerate"
note: "model used a deprecated parameter"
What the model may draft and what the human must own
The split that matters is not prose versus code, but draft versus disposition. The model may draft the prose, examples, annotations, and even a first manifest version, because all of those are cheap to regenerate when wrong.
The human must own the disposition of every failure, because only a person can decide whether an example should be regenerated, rewritten, or deleted. The human must also own the semantic accuracy of the prose, since a passing example does not prove that the surrounding explanation is correct. Signing the manifest is the moment where the human accepts responsibility for what the doc claims.
Why free infrastructure changes the loop
MonkeyCode's free model access means the drafting step has no per-word cost, so you can regenerate a failing example instead of hand-patching it with guesswork. The free server option gives the loop a permanent home on every pull request, because a metered verification step vanishes exactly when the change is largest. The script itself is plain Python and runs anywhere, so the core value of the workflow does not depend on any vendor.
Limitations and who should skip this
Executable examples are not a substitute for semantic review, because a snippet can pass while the surrounding prose still misleads the reader. The script only verifies annotated blocks, so an example the model forgot to mark will escape the loop entirely. Teams without CI or a stable test environment should skip this workflow, because the value comes from running examples on every change, not once. Documentation that is purely conceptual, with no code at all, gains nothing from this pipeline.
The human still signs
Treating generated examples as tests changes the incentive structure of the drafting step, because the model's mistakes surface as failures instead of reader frustration. The human still owns the final judgment, which now happens against a concrete report rather than a vague feeling that something is wrong. Start with one doc, annotate five examples, and let the runner tell you what the model actually got right.
Top comments (0)