Test-First AI Documentation: A Workflow That Keeps Generated Docs Honest

AI code assistants have made documentation fast to produce and easy to ignore. The issue is not speed; it's trust. Models can write a polished docstring that describes a function that no longer exists, or an example that fails on the first run. The more docs we generate, the more stale those docs become if nothing checks them.

This article describes a test-first documentation workflow. It treats documentation like code: each claim passes an automated check before it is considered done. You will find a small Python artifact that extracts docstring stubs, a validation script that catches broken examples, and a decision table that separates what a model may draft from what a human must own.

Why AI-Generated Docs Fail Silently

Consider a typical flow: you ask a language model to document a function. It returns a docstring with an example. The example contains a parameter name that was renamed in the last commit. The docstring looks plausible, so no one questions it.

The problem is the absence of a feedback loop. Code has compilers, linters, and tests. Documentation only has the cursor and the reader's patience.

The fix is to give documentation the same feedback loop. Run the examples. Check that documented names exist. Compare the documented behavior with the actual behavior.

The Three Test Gates

Gate 1: Presence

Every public function should have a docstring. This is simple to enforce with pydocstyle or a tiny AST script. If a new function lands without a docstring, CI fails.

Gate 2: Accuracy

Docstring examples must be executable. Python's doctest is the classic tool, but you can also build custom checks. For example:

def format_bytes(size: int) -> str:
    """
    Convert a size in bytes to a human-readable string.

    >>> format_bytes(1024)
    '1.0 KiB'
    >>> format_bytes(1536)
    '1.5 KiB'
    """
    ...

Running python -m doctest module.py turns those examples into tests. If the function changes behavior, the docs fail loudly.

Gate 3: Freshness

When a function signature changes, its docstring should be flagged. You can write a CI script that compares the set of public names defined in the code with the set of names mentioned in the documentation. Here is a minimal version:

# check_doc_coverage.py
import ast, pathlib, sys

def public_functions(path):
    tree = ast.parse(pathlib.Path(path).read_text())
    return {
        node.name for node in ast.walk(tree)
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef))
        and not node.name.startswith("_")
    }

def documented_names(path):
    text = pathlib.Path(path).read_text()
    return {name for name in public_functions(path) if f"`{name}`" in text}

if __name__ == "__main__":
    if len(sys.argv) != 2:
        raise SystemExit("usage: check_doc_coverage.py <file.py>")
    funcs = public_functions(sys.argv[1])
    documented = documented_names(sys.argv[1])
    missing = funcs - documented
    if missing:
        raise SystemExit(f"Undocumented public functions: {missing}")

This is a heuristic, not a proof. A docstring could mention a function name and still be wrong. But it catches the most common drift.

Using a Free-Tier Model to Create the Draft

Drafting docstrings is a good job for a language model because it is repetitive and low-risk. The high-risk part is reviewing the result. That's why the workflow separates drafting from ownership.

MonkeyCode is an open-source project that provides free model access (currently advertised as 10 million tokens, check the official README for the latest quota) and a free server option, which is useful for this kind of batch job. Instead of paying per request, you can run a nightly script that scans new public functions and sends their signatures to the model for a draft. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A typical prompt might be:

Given this function signature:
    def calculate_total(items: list[float], tax_rate: float) -> float
Write a docstring with:
1. One sentence describing what it does.
2. Args and return sections.
3. One doctest example.
Do not invent edge cases.

The model output should be treated as draft. It goes into the docstring, but the CI gates decide whether it survives.

What a Human Must Own

Even with three test gates, some information cannot be verified by executing examples. A human reviewer owns:

• Public API contracts: exact parameter semantics, ordering guarantees, exceptions raised.
• Breaking changes: migration steps from old to new behavior.
• Security and safety invariants: mention of auth, data handling, or resource limits.
• Rationale: why a design decision exists, not just what the code does.

The table below summarizes what can be delegated to a model and what should stay with a human.

Documentation content	Model can draft?	Human must review?
Repeating parameter descriptions	Yes	No, if example passes
Usage examples for stable functions	Yes	Yes, check for outdated calls
Return-value units and ranges	No	Yes
Exception and edge-case behavior	No	Yes
Migration and deprecation notes	No	Yes
"Why" explanations	No	Yes

The Full Pipeline

1. On every pull request, a CI job extracts changed public functions.
2. A script generates empty docstring stubs for those functions.
3. A free-tier model (e.g., via MonkeyCode's free model access) fills in the stubs.
4. doctest runs every example in the docstrings.
5. A coverage checker asserts every public function appears in the docs.
6. A human reviews only the rows marked "Yes" in the table above.

This keeps the model's contribution useful but bounded. It also gives reviewers a checklist instead of a blank page.

Limitations

• The custom freshness check is a heuristic. It cannot understand meaning, so a wrong yet plausible explanation can pass.
• Doctests fail on functions with non-deterministic output or heavy I/O. You may need to exclude those examples from the test set.
• The free tier on any service may have rate limits and uptime constraints. Verify the current quotas on the official project before building a production pipeline.
• For teams with strict compliance requirements, this workflow is not a substitute for expert review. If a wrong docstring could cause harm, do not rely on automated checks alone.

Who Should Not Use This

You should not use this workflow if:

• Your documentation must meet regulatory audit standards or document safety-critical systems.
• Your team already has a maintainer who writes docs by hand and the docs are never stale. Automating will add process overhead.
• You cannot tolerate the small risk that a generated example passes doctest but misleads readers.

The Takeaway

Documentation is code. Give it tests, give it CI gates, and give a model only the parts it can fail safely. The free tier of a tool like MonkeyCode makes the experimentation cheap, but the discipline comes from your pipeline, not from the model.

Comments

[Kane Lim]

Hello Glad to see you, I am Kane Lim from Hong Kong. I have over 10 years of development experience. I am writing this because your post was interesting.

The strongest idea here is treating documentation as an executable artifact rather than generated prose. I would extend the pipeline with contract testing and AST based signature diffing instead of relying primarily on textual heuristics.

For example, extract function signatures, annotations, defaults, raised exceptions, and referenced symbols into a structured contract. The generated documentation can then be validated against that contract during CI. For examples involving external systems or nondeterministic output, use hermetic fixtures and snapshot assertions rather than ordinary doctest execution.

I would also version documentation contracts alongside API versions and trigger targeted regeneration only when the dependency graph changes. This reduces unnecessary model calls while making documentation drift observable.

One additional metric worth tracking is documentation validity rate across releases. That turns documentation quality into an engineering signal instead of subjective review.

Excellent workflow. The separation between model drafting and human ownership is exactly what makes this approach practical.