Make Generated Docs Executable: A CI Harness That Runs Every Example

A model-written documentation draft becomes trustworthy only when every example in it executes, and a small extraction harness can turn those examples into a regression suite that fails CI before your users do. Generated documentation has a specific failure mode that traditional prose review almost never catches before the merge button is pressed. Models are fluent at producing plausible commands, configuration snippets, and API calls that look correct but silently diverge from the actual codebase. The result is a doc set that reads well and misleads precisely when a reader copies the first block they see.

This article walks through a documentation-generation workflow where the model drafts the full text, a free server runs every example, and a human owns the triage of failures. The pipeline uses MonkeyCode's free model access for the drafting pass and its free server option for the verification loop, which keeps the whole cycle at zero marginal infrastructure cost. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why examples are the only claims that matter

Prose claims are cheap to verify against the source but expensive to trust without a concrete execution. A sentence like "the client retries with exponential backoff" requires reading the implementation, checking the default configuration, and confirming the behavior under a simulated failure. An example block, by contrast, is a contract: it either runs and produces the documented output, or it fails in front of a user who copied it verbatim.

In practice, examples are also the most copied part of any documentation page because they offer immediate, paste-ready value. When a reader pastes a snippet into their terminal, the snippet's behavior becomes the documentation, regardless of what the surrounding prose says. That behavior makes example execution the single highest-value automated check a documentation pipeline can run.

The workflow at a glance

The pipeline has four stages, and each stage has a clear owner who is accountable for the output.

1. Draft: the model writes the first pass of every doc page from the repository's public API surface.
2. Extract: a script pulls every fenced code block out of the Markdown and tags it with a risk class.
3. Execute: a sandboxed runner executes each example on the free server and records pass or fail.
4. Triage: a human reviews the failures, fixes genuine doc bugs, and signs off on ambiguous cases.

The ownership split is the part most teams skip because it feels like process overhead rather than engineering work. The model may draft prose and propose fixes, but the human owns the classification rules and the final decision on every failed example.

Stage one: extract every code block

The extraction script is deliberately small because a verification tool that nobody can read will never be trusted. The following Python version finds all fenced blocks and emits a tab-separated manifest that the runner consumes.

import re
import sys
from pathlib import Path

def extract_blocks(markdown_path: Path):
    text = markdown_path.read_text(encoding="utf-8")
    pattern = re.compile(r"```

(\w+)\n(.*?)

```", re.DOTALL)
    for index, (language, code) in enumerate(pattern.findall(text)):
        yield markdown_path, index, language, code.strip()

if __name__ == "__main__":
    for path in map(Path, sys.argv[1:]):
        for source, index, language, code in extract_blocks(path):
            print(f"{source}#example-{index}\t{language}\t{code[:60]!r}")

Run it against your docs directory and you get a manifest of every example that currently exists in the repository. The manifest becomes the input to the classification table, which decides how much trust each example deserves.

Stage two: classify by blast radius

Not every example deserves the same verification budget, and treating them uniformly wastes human attention on trivial snippets. A shell command that lists files is low risk, while a configuration snippet that mutates production state is high risk. The classification table below maps each class to an automated check and a human owner, and it works well for CLI-heavy projects.

Class	Example type	Automated check	Human owns
A	CLI flags, schema, config	Run with --help or validate against schema	Flag semantics and defaults
B	API usage snippets	Compile and run in a sandbox	Intent and expected output
C	Conceptual prose, pseudocode	None	Every word

Class A examples get a deterministic check because they have a ground truth in the tool itself. Class B examples get an execution check plus a human review of whether the output actually matches the surrounding prose. Class C examples get no automated check at all, and the human owns them completely because prose cannot be executed.

Stage three: run the harness on the free server

The runner executes every Class A and Class B example in a clean environment and fails the build on the first error. A minimal shell runner for the two most common example languages looks like this.

#!/usr/bin/env bash
set -uo pipefail
for manifest in "$@"; do
  while IFS=$'\t' read -r source language code; do
    case "$language" in
      sh|bash) bash -c "$code" >/tmp/example.log 2>&1 ;;
      python)  python3 -c "$code" >/tmp/example.log 2>&1 ;;
      *)       echo "SKIP $source (unsupported language)"
               continue ;;
    esac
    if [ $? -eq 0 ]; then
      echo "PASS $source"
    else
      echo "FAIL $source"
      cat /tmp/example.log
      exit 1
    fi
  done < "$manifest"
done

The free server option matters here because the harness needs a clean, disposable environment for every run. Provisioning a dedicated runner for a docs check feels wasteful until you realize that stale examples cost more support time than the runner costs to operate. With MonkeyCode's free server, the verification loop runs on every pull request without touching your cloud budget.

Stage four: human triage, not human transcription

When the harness fails, the model can propose a fix, but the human owns the classification of the failure. The triage has exactly three outcomes: the example was wrong and the doc gets fixed, the example was right and the test environment was wrong, or the example was ambiguous and the prose must be rewritten to remove the ambiguity.

The third outcome is the one that improves the documentation over time because it exposes API design problems. Ambiguous examples are a signal that the underlying interface is unclear, and every triage session produces a small list of API improvements that the team can schedule. The documentation pipeline thus becomes a source of product feedback rather than a publishing step with no downstream signal.

Limitations and who should skip this

This workflow assumes your documentation contains runnable examples, so it is a poor fit for purely conceptual docs or for projects where examples require licensed tools that cannot run in a sandbox. The harness also cannot verify prose claims, which is why Class C content still needs a human review pass. Finally, the extraction script is intentionally naive: examples that depend on external services will fail in the sandbox even when they are correct, and those failures must be marked as skippable rather than deleted.

Teams that already have a strong documentation review culture may find the harness redundant, and that is a valid reason to skip it. Teams that merge generated docs without executing a single example are the ones that need this pipeline, because the cost of a wrong example is paid by every reader who copies it. If you already generate docs with a model, run the extraction script on your last merged pull request and count how many examples would have failed.