Generated Docs Should Fail CI: A Contract Test for Documentation Claims

AI-generated documentation has a credibility problem: the prose reads well, but the examples may never have run. A doc that passes a human skim can still contain a command that fails, a flag that was renamed, or an import that does not exist. The fix is not more review; it is a contract test that executes every code block in the generated Markdown and fails the build when a claim breaks. This article shows a small, reproducible harness and a clear ownership split between what a model may draft and what a human must own.

The ownership split

Before you automate verification, decide who owns each part of the documentation artifact. A model can draft narrative explanations, API descriptions, and common usage patterns from the codebase, but it should not own the execution proof. A human must own version pins, security-sensitive examples, invariants, and the test harness itself, because those require context the model does not have. The following decision table keeps the boundary explicit.

Artifact	Model may draft	Human must own
Prose explanation	Yes	Final wording
Code snippets	Yes	Execution proof
Version pins	No	Yes
Security examples	No	Yes
Test harness	No	Yes
CI integration	No	Yes

A contract test that runs the docs

Most doc review processes check formatting and tone, not whether the examples execute. The script below changes that by extracting every fenced code block, running the ones it can, and exiting nonzero on the first failure. It treats a documentation claim as a testable contract: if the code block does not run, the documentation does not merge.

#!/usr/bin/env python3
'''doc_contract.py - execute fenced code blocks from generated docs.'''
import argparse
import re
import subprocess
import sys
from pathlib import Path

FENCE = re.compile(r'```

([\w+-]+)?(\s+doc-contract:skip)?\n(.*?)

```', re.DOTALL)

def blocks(md: Path):
    text = md.read_text()
    for number, match in enumerate(FENCE.finditer(text), 1):
        lang = (match.group(1) or 'text').lower()
        skip = bool(match.group(2))
        code = match.group(3)
        if not skip and lang in {'bash', 'sh', 'shell', 'python'}:
            yield number, lang, code

def run(lang, code):
    if lang in {'bash', 'sh', 'shell'}:
        return subprocess.run(
            ['bash', '-euo', 'pipefail', '-c', code],
            capture_output=True, text=True, timeout=30,
        )
    if lang == 'python':
        return subprocess.run(
            [sys.executable, '-c', code],
            capture_output=True, text=True, timeout=30,
        )
    return None

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('files', nargs='+', type=Path)
    args = parser.parse_args()
    failures = []
    verified = 0
    for path in args.files:
        for number, lang, code in blocks(path):
            verified += 1
            result = run(lang, code)
            if result is not None and result.returncode != 0:
                failures.append((path, number, lang, result.stderr))
    for path, number, lang, stderr in failures:
        print(f'{path}:{number} [{lang}] failed:\n{stderr}')
    print(f'Verified {verified} blocks; {len(failures)} failures.')
    sys.exit(1 if failures else 0)

if __name__ == '__main__':
    main()

This script skips blocks tagged with doc-contract:skip and ignores languages it cannot execute. Add that marker to pseudo-code, partial snippets, or examples that need a live service. The timeout=30 argument prevents a hanging example from stalling the pipeline.

The workflow

1. Draft with a model: generate a first-pass doc whose code blocks reflect the current API.
2. Mark non-executable blocks: add doc-contract:skip to pseudo-code or partial examples.
3. Run the contract test locally: python doc_contract.py README.md.
4. Wire it into CI: add a job that runs the script on every PR that touches docs.
5. Treat failures as review comments: the human either fixes the code, updates the doc, or adds a skip with justification.

A minimal GitHub Actions step looks like this:

- name: Verify generated docs
  run: |
    python doc_contract.py $(git ls-files '*.md')

This assumes a Unix runner with bash and Python available. The same command works in most CI systems that let you run shell scripts.

Where the free tier fits

MonkeyCode's free model access and free server option let you prototype this loop without a paid plan. You can use the model to draft the initial doc, and the free server option gives you a place to run the verification script on a small repository. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The value of the workflow does not depend on the vendor; the same script works with any model or any CI runner.

Limitations and who should not use this

Executing a code block proves only that it runs, not that it is semantically correct. A snippet can pass and still recommend a deprecated pattern or omit an error path. The script also ignores non-executable languages, so a doc with many text blocks gets only partial coverage. Finally, the doc-contract:skip marker is an escape hatch that can hide broken examples if reviewers do not ask why it was added.

Teams that document pseudo-code or design sketches should not enable this test, because every block would need a skip marker. Teams that cannot run untrusted code in CI should also avoid it, unless they isolate the job in a container or a dedicated runner. Security-sensitive APIs need a human to review examples before any execution, so the contract test becomes a second gate, not a replacement for review.

Conclusion

Documentation generated by a model should be treated as a draft with a test suite, not as a finished artifact. A contract test that runs every code block turns prose claims into executable checks and gives reviewers a concrete reason to reject a PR. The ownership split remains the hard part: the model drafts, the human owns the invariants, and the CI pipeline enforces the boundary. Start with one README, add the script, and let the failures tell you where the documentation is lying.