Mutate the Docs: A Mutation Score for AI-Generated Documentation

A paragraph that reads like correct documentation is weak evidence that the documentation is correct, and the weakness grows when a language model wrote the paragraph. Human review is the usual safety net, yet the reviewer is rarely tested the way we test code, so a plausible but wrong claim can slip into a merged document and survive there for months. This article builds a claim-mutation harness that scores your documentation review process by mutating every verifiable statement and measuring how many of those mutations the process would catch.

The Untested Component in the Docs Pipeline

Code review has mutation testing: you alter a program and check whether the test suite notices the alteration. Documentation review has no equivalent, so teams generally trust that a senior engineer will spot a fabricated function name or a slightly wrong status code. The model is benchmarked constantly, but the review process is not, which is the wrong side of the pipeline to trust. The fix is not to read more slowly; it is to quantify how many drifted claims the current review would detect.

A mutation score for docs gives you exactly that number. It is the documentation analogue of mutation testing for code patches, and it tells a different story than a model benchmark. A high model score with a low review score means every error is caught at the last possible moment, or never.

What Counts as a Verifiable Claim

Two classes of document statements can be checked mechanically against source code: backticked symbols such as auth.get_user() and explicit HTTP status codes such as HTTP 200. Both have an unambiguous ground truth inside the repository, either as an AST definition or as a status_code assignment. Everything else, including performance numbers, "fast", "robust", and "production-ready", is not machine-verifiable and must remain the human reviewer's responsibility.

The rule of thumb is simple: a claim is verifiable when a script can prove it true or false using only the repository. If generated docs contain almost no verifiable claims, the harness scores near zero, and that result is itself the finding. The document gives a reviewer too few hooks to catch a drift, which means the next hallucinated sentence will be accepted.

Building the Source of Truth

The harness first inventories every function, class, and HTTP status code that actually exists in the source tree. The inventory is a short Python walk over the AST and a regex scan, and it serves as the referee for every later judgment:

def build_truth(source_dir: Path) -> set:
    truth = set()
    for path in source_dir.rglob("*.py"):
        text = path.read_text()
        for code in re.findall(r"status_code\s*=\s*(\d{3})", text):
            truth.add(f"HTTP {code}")
        try:
            tree = ast.parse(text)
        except SyntaxError:
            continue
        for node in ast.walk(tree):
            if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
                truth.add(f"{path.stem}.{node.name}()")
    return truth

Generate the inventory from the feature branch under review, never from main, so the referee cannot drift from the code the docs describe.

Extract, Mutate, and Score

The full harness extracts claims, mutates each one, and prints a score. A claim is UNSUPPORTED when it names something absent from source, OK when its mutation is also absent, and ESCAPED when the mutation exists in the source too:

#!/usr/bin/env python3
"""claim_mutator.py — mutation score for documentation review."""
import argparse
import ast
import re
from pathlib import Path

CLAIM_PATTERNS = {
    "symbol": re.compile(r"`([a-zA-Z_][\w.]*\(`\)`)"),
    "http": re.compile(r"\bHTTP\s+(\d{3})\b"),
}

def build_truth(source_dir: Path) -> set:
    truth = set()
    for path in source_dir.rglob("*.py"):
        text = path.read_text()
        for code in re.findall(r"status_code\s*=\s*(\d{3})", text):
            truth.add(f"HTTP {code}")
        try:
            tree = ast.parse(text)
        except SyntaxError:
            continue
        for node in ast.walk(tree):
            if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
                truth.add(f"{path.stem}.{node.name}()")
    return truth

def extract_claims(doc_text: str):
    for kind, pattern in CLAIM_PATTERNS.items():
        for match in pattern.finditer(doc_text):
            yield kind, match.group(1)

def mutate(kind: str, value: str) -> str:
    if kind == "symbol":
        return value[:-2] + "_missing()"
    if kind == "http":
        return f"HTTP {int(value) + 1}"
    return value

def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("doc", type=Path, help="Markdown file to score")
    parser.add_argument("source", type=Path, help="Python source directory")
    args = parser.parse_args()

    truth = build_truth(args.source)
    results = []
    for kind, value in extract_claims(args.doc.read_text()):
        if value not in truth:
            results.append(("UNSUPPORTED", kind, value))
        elif mutate(kind, value) not in truth:
            results.append(("OK", kind, value))
        else:
            results.append(("ESCAPED", kind, value))

    if not results:
        print("no verifiable claims found; score is undefined")
        return 1

    score = sum(1 for status, _, _ in results if status != "ESCAPED") / len(results)
    for status, kind, value in results:
        print(f"{status:11s} {kind:7s} {value}")
    print(f"score={score:.2f} claims={len(results)}")
    return 0 if score >= 0.8 else 1

if __name__ == "__main__":
    raise SystemExit(main())

Running it looks like this:

$ python claim_mutator.py README.md src/
UNSUPPORTED symbol  auth.get_user()
OK         symbol  auth.login()
OK         http    HTTP 200
ESCAPED    http    HTTP 201
score=0.75 claims=4

The statuses map directly to review decisions:

Status	Meaning	Review implication
UNSUPPORTED	The claim names something absent from source	The doc is already drifting; block the PR
OK	The claim exists and its mutation would not survive	The reviewer has enough hooks to catch drift
ESCAPED	A mutated claim would still satisfy the inventory	The doc or the inventory lacks discriminating detail; read this line first

A score of 0.8 is a reasonable default gate, expressed as the process exit code, so the harness can plug straight into CI.

Wiring the Workflow with a Free-Tier Stack

The practical loop has four steps, and a free-tier stack fits it well because the cost center is not compute but iteration. The redraft loop runs on the free model access in MonkeyCode, which is enough for regenerating doc sections around a fixed inventory, and the free server option in the same project can host the scheduled checker without a paid runner. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

1. Generate the truth inventory from the branch under review and pass it to the model together with the old document; ask for a redraft that preserves every symbol and status code verbatim.
2. Run claim_mutator.py against the generated document and fix the output until no UNSUPPORTED claims remain.
3. Repeat the redraft until the mutation score reaches at least 0.8; each ESCAPED line tells the model exactly which sentence is too vague.
4. Deploy the checker on the free server option as a scheduled job that runs on every docs pull request and reports the score back to the merge check.

The ownership split stays the same as in any serious docs pipeline: the model may draft everything that the machine can verify, and the human owns every judgment the machine cannot make. Performance promises, migration rationale, and security caveats remain outside the harness by design, so the score is not a substitute for reading; it is a filter that makes reading cheaper.

Limitations and Who Should Skip This

The mutation score measures verifiability, not semantic truth, so a document can score perfectly while its explanation still misleads the reader. The AST inventory assumes module-qualified names and a conventional single-package layout, and dynamic API surfaces will need extra patterns before the score means anything. Teams with stable, rarely changing documentation gain little from this workflow, and a project whose docs contain no code symbols should treat a zero score as a signal to write more concretely rather than to ship the harness. If you adopt it, expect the first run to be embarrassingly low, keep the threshold explicit, and let the escaped lines drive the next redraft.