Every Code Block Is a Test: A Free-Tier Doc Example Gate

#ai #documentation #python #testing

A documentation example that never executes is a promise the team cannot keep. The cheapest fix is a mechanical gate, not more careful review: mark the blocks that must run, execute them in CI, and block the merge when any of them breaks. This article builds that gate with a small Python runner, then shows why free model access changes how often a team can afford to regenerate and re-verify those examples.

Reviewed for Readability, Never Executed for Truth

Model-drafted documentation makes the verification gap worse. A language model writes plausible examples, and a human reviewer reads them for sense, but neither step actually runs the code. A config snippet can drift from the real API for months while every reader assumes it works. The only test it ever passed was the reviewer's visual scan. A recurring theme in recent DEV discussions is that AI promotes every developer to reviewer while nobody tests the reviewer. Doc examples have exactly the same blind spot: reviewed for readability, never executed for truth.

The failure mode is measurable. When an example breaks, the cost appears far from the change that caused it. A reader copies the snippet, hits an error, and opens an issue that maintainers must triage. The original PR merged cleanly because the example looked correct, not because it was correct. A run gate moves that verification left, to the moment the doc changes, where the fix is still cheap.

The Ownership Split, Applied to Executability

The draft/own split that applies to generated prose also applies to generated examples, but the boundary is different. The model may draft the example body, the command sequence, and the placeholder values; the human must own the execution contract. That contract has three parts: the marker that declares a block runnable, the assertion that defines success, and the environment assumptions that CI can actually satisfy.

Without the human-owned contract, the gate is either silent or noisy. If every fenced block runs, the pipeline fails on illustrative pseudocode and nobody trusts the signal. If no block runs, the gate is decoration. The marker is the human's signature: it says this block is a test, not an illustration, and the team is willing to be woken up when it fails.

The Artifact: A Runner That Treats Markdown as Tests

The gate needs three pieces: a marker convention, a runner script, and a CI job. Here is the full workflow.

Mark runnable blocks in your docs with the run marker after the language tag.
Add scripts/example_runner.py to the repository.
Wire the runner into CI so it runs on every pull request that touches docs.
When a PR changes a doc, have a model draft the updated example, a human verify the marker and assertions, and CI execute the block against the real toolchain.

The marker convention is a language tag followed by the word run inside the opening fence:

```bash run
git status --porcelain
```

The runner is deliberately small. It extracts marked blocks, executes each one in a temporary directory, and returns a non-zero exit code on the first failure.

#!/usr/bin/env python3
"""Run marked code blocks from Markdown docs as a regression gate.

Only blocks tagged with the `run` marker are executed:

    ```

bash run
    git status --porcelain


    ```

Blocks tagged with `skip` are counted but not executed.
"""
import re
import subprocess
import sys
import tempfile
from pathlib import Path

RUN = re.compile(r"```

(?P<lang>\w+)\s+run\s*\n(?P<body>.*?)

```", re.DOTALL)
SKIP = re.compile(r"```

(?P<lang>\w+)\s+skip\s*\n(?P<body>.*?)

```", re.DOTALL)

def execute(lang: str, body: str, cwd: Path) -> int:
    if lang in {"bash", "sh", "shell"}:
        return subprocess.run(["bash", "-e", "-c", body], cwd=cwd).returncode
    if lang == "python":
        return subprocess.run(["python", "-c", body], cwd=cwd).returncode
    print(f"  warn: unsupported language '{lang}', treating as pass")
    return 0

def main() -> int:
    paths = sys.argv[1:] or ["README.md"]
    total = skipped = failed = 0
    for name in paths:
        text = Path(name).read_text()
        for lang, body in RUN.findall(text):
            total += 1
            with tempfile.TemporaryDirectory() as tmp:
                code = execute(lang, body, Path(tmp))
            if code != 0:
                failed += 1
                print(f"FAIL {name}: {lang} block exited {code}")
        skipped += len(SKIP.findall(text))
    print(f"summary: {total} executed, {skipped} skipped, {failed} failed")
    return 1 if failed else 0

if __name__ == "__main__":
    sys.exit(main())

The local command is the same command CI runs, so the feedback loop is identical on a laptop and in the pipeline.

python3 scripts/example_runner.py docs/ README.md

The CI job is a standard two-step workflow that triggers only when documentation paths change.

# .github/workflows/docs-examples.yml
name: docs-examples
on:
  pull_request:
    paths: ["docs/**", "README.md"]
jobs:
  run-examples:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python3 scripts/example_runner.py docs/ README.md

A doc example that breaks the build gets the same attention as a unit test that breaks the build, because it is one.

Why Free Model Access Changes the Regeneration Economics

The gate only stays honest if examples are regenerated when APIs change, and regeneration has a cost that teams quietly avoid. This is the same pattern that makes metered AI reviews a hidden tax: when a verification step costs money, teams skip it on the riskiest changes, exactly where it matters most. MonkeyCode's free model access and free server option remove that incentive for the doc loop. Disclosure: This article was prepared as part of MonkeyCode's product outreach. That combination removes the per-call cost from the regeneration step, so the team can regenerate an example on every doc PR and let CI decide whether the new draft is executable.

The workflow then becomes a tight loop. The model drafts the updated example from the changed API surface, the human adjusts the run marker and the assertion, and the runner executes the block against the real toolchain. If the example fails, the model receives the failure output and produces a corrected draft, which the human reviews again. The free tier matters here not because it saves a large bill. It removes the psychological threshold that makes teams skip the loop on a Friday-afternoon doc fix.

Limitations and Who Should Skip This

The gate is not a sandbox. The runner executes code in a temporary directory on the CI host, so examples that need secrets, network access, or long-running services must be designed for that environment or marked skip. Unsupported languages pass by default, which keeps the gate quiet but means a typo in the marker can silently downgrade a test into decoration. The summary line reports skipped blocks, so the human still sees the count.

Teams whose docs are mostly conceptual prose with rare examples will get little value from the gate. Teams without a docs CI job should build that first. The runner also cannot judge whether an example is good advice; it only proves that the example executes. That distinction matters: an executable example can still teach a bad pattern, and the human review owns that judgment.

The Contract Is the Point

A doc example is a test, and the marker is the contract that makes it one. Mark the blocks that must run, execute them in CI, and treat a broken example as a broken build. The model drafts, the human owns the execution contract, and free tooling makes the loop affordable on every PR. If your README contains code blocks that have never run, point the runner at one file this week and count the promises that break.

DEV Community