Gate Model-Refreshed Docs With Three Verdicts: Verified, Stale, Unverifiable

Most teams that let a model touch documentation do not have a writing problem; they have a verification problem. The document reads well, the prose is clean, and nothing on the page tells you whether --max-retries still exists in the code. The useful control is not another review pass but a machine-checkable verdict per claim, and there are exactly three: Verified, Stale, and Unverifiable.

This article builds that gate. You will get a small dependency-free script that extracts factual claims from Markdown, resolves them against the source tree, and fails the build only on STALE. You will also get a zone convention that decides which sections a model may regenerate and which a named human must own, plus the limitations that make this approach wrong for some repositories.

Why regeneration alone makes drift worse

A model asked to "refresh the README" will happily rewrite a flag that was deleted two releases ago, because the prompt is the only ground truth it has. Silent staleness is the failure mode that hurts: the document stays confident, the reader follows it, and the error surfaces as a support ticket. Diff size is a poor proxy for that risk, since a one-character version bump and a fabricated endpoint look identical in review.

The gate below inverts the order. Extract the claims first, resolve each against the repository, and only then decide who should rewrite the sentence. Verification becomes a precondition for drafting rather than a chore afterward.

Step 1: Split the document into two zones

Not every sentence is a candidate for model regeneration. Mechanical claims derive from the tree, and a model can redraft them safely once the evidence is attached. Intent claims — why a tradeoff was chosen, what the project guarantees, who the feature is not for — have no source file to check against, so a human must own them by name.

Claim class	Example	Who drafts	What closes it
Flags, env vars	--max-retries, QUEUE_URL	Model	Token found in source
Versions, endpoint paths	v2.4.1, /api/v2/jobs	Model	Match in manifest or router
Numeric limits	"batch of 500"	Model, marked provisional	Constant in code, or human sign-off
Guarantees, tradeoffs	"safe to retry twice"	Named human	Owner attestation
Compatibility promises	"drop-in for v1"	Named human	Owner attestation

Mark the boundary in the file itself so tooling and reviewers read the same signal. Inline fences are enough, and they survive most Markdown renderers as comments.

<!-- model:begin id=cli-flags -->
Run with `--max-retries 3` and set `QUEUE_URL` in the environment.
<!-- model:end -->

## Why the retry budget is capped

Human-owned section. The cap exists because downstream consumers deduplicate on
job id, and a longer window widens the duplicate surface faster than it improves
throughput. Owner: @queue-team

Step 2: Extract and resolve claims

The script below is deliberately narrow. It recognises four claim classes that are cheap to verify and expensive to get wrong, then emits a JSON report and a non-zero exit code when anything is STALE. Broaden the regexes only after you have watched the false-positive rate on your own docs.

#!/usr/bin/env python3
"""docsdrift.py - flag documentation claims that no longer match the source tree."""
import json, re, sys
from pathlib import Path
from difflib import get_close_matches

CLAIMS = {
    "flag":     re.compile(r"(?<![\w-])--[a-z][a-z0-9-]{1,30}"),
    "env":      re.compile(r"\b[A-Z][A-Z0-9]{2,}(?:_[A-Z0-9]+)+\b"),
    "version":  re.compile(r"\bv?\d+\.\d+(?:\.\d+)?\b"),
    "endpoint": re.compile(r"(?<![\w.])/(?:api|v\d)/[a-z0-9/_{}-]+"),
}
SOURCE_SUFFIXES = {".py", ".go", ".ts", ".rs", ".yaml", ".toml", ".json"}
DOC_SUFFIXES = {".md", ".mdx"}
SKIP = {".git", "node_modules", "dist", "build", ".venv", "vendor"}

def files(root, suffixes):
    for p in root.rglob("*"):
        if p.is_file() and p.suffix in suffixes and not SKIP & set(p.parts):
            yield p

def inventory(root):
    """Every token that actually exists somewhere in the source tree."""
    seen = {k: set() for k in CLAIMS}
    for path in files(root, SOURCE_SUFFIXES):
        text = path.read_text(errors="ignore")
        for kind, rx in CLAIMS.items():
            seen[kind].update(rx.findall(text))
    return seen

def verdict(token, kind, known):
    if token in known[kind]:
        return "VERIFIED", None
    near = get_close_matches(token, known[kind], n=1, cutoff=0.8)
    if near:
        return "STALE", near[0]
    return "UNVERIFIABLE", None

def main(root=".."):
    known = inventory(Path(root))
    report = []
    for doc in files(Path(root), DOC_SUFFIXES):
        for lineno, line in enumerate(doc.read_text(errors="ignore").splitlines(), 1):
            if "human:owned" in line:
                continue
            for kind, rx in CLAIMS.items():
                for token in rx.findall(line):
                    state, suggestion = verdict(token, kind, known)
                    if state != "VERIFIED":
                        report.append({
                            "doc": str(doc), "line": lineno, "kind": kind,
                            "claim": token, "verdict": state, "did_you_mean": suggestion,
                        })
    print(json.dumps(report, indent=2))
    return 1 if any(r["verdict"] == "STALE" for r in report) else 0

if __name__ == "__main__":
    sys.exit(main(*sys.argv[1:]))

Run it from the repository root and read the report, not just the exit code:

python docsdrift.py . > docs-report.json; echo "exit=$?"
jq '[.[] | select(.verdict=="STALE")] | length' docs-report.json
jq -r '.[] | "\(.doc):\(.line) \(.claim) -> \(.did_you_mean // "no candidate")"' docs-report.json | head -40

The did_you_mean field is the part that earns its place. A STALE verdict with a close candidate is usually a rename that nobody propagated; an UNVERIFIABLE verdict is the more interesting signal, because it means the doc asserts something the tree cannot confirm at all.

Step 3: Turn the verdicts into routing rules

Three verdicts map onto three actions, and keeping the mapping mechanical prevents arguments in review threads.

1. VERIFIED — the sentence may stay, or a model may redraft it inside a model:begin fence.
2. STALE — block the merge, attach the suggested token, and let the model regenerate only that fenced block from the evidence line.
3. UNVERIFIABLE — never auto-edit. Add it to a human queue with the file and line, and either supply evidence or move the sentence into a human-owned section.

Wire the first two into a pre-commit hook so the cost lands at the author's desk, and run the full sweep nightly so drift that arrives through source changes is caught even when nobody touches the docs.

# .git/hooks/pre-commit
#!/usr/bin/env bash
python docsdrift.py . > /tmp/docs-report.json || {
  echo "docsdrift: stale claims found; see /tmp/docs-report.json" >&2
  exit 1
}

# .github/workflows/docsdrift.yml
name: docsdrift
on:
  schedule: [{cron: "0 6 * * *"}]
  pull_request: {paths: ["docs/**", "**/*.md"]}
jobs:
  drift:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python docsdrift.py . > docs-report.json
      - uses: actions/upload-artifact@v4
        if: always()
        with: {name: docs-report, path: docs-report.json}

Step 4: Give the drafting step a bounded job

Once the report exists, the model's task shrinks to something checkable: rewrite the flagged fenced block so it matches the evidence, and change nothing outside the fence. That constraint matters more than prompt wording, because an agent that can only edit inside markers cannot quietly rewrite a guarantee it does not understand. This is where limited free model access is genuinely useful: drafting a handful of fenced blocks from a supplied evidence line is a small, repetitive job, so a free-tier allowance is enough to keep the loop running without metering it.

If you adopt MonkeyCode for that step, the free server option is the more interesting half of the offer, since the nightly drift sweep is exactly the kind of small scheduled job people abandon for lack of somewhere to run it. Host the runner there, keep the report as an artifact, and treat the output as a proposal until a human merges it. Quota, hardware, and runtime terms are set by the project and they change, so confirm the current terms on the project page before you build a schedule around them. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Limitations and who should skip this

Narrow regexes cut both ways. A claim class you did not enumerate stays invisible, so treat VERIFIED as "no contradiction found" rather than "correct". Version strings in prose collide with version strings in examples, which is why the ignore path matters and why you should start with flags and env vars only.

The approach also assumes a source tree you own. Repositories that document an external service, or whose docs live in a separate repository from the code, will produce mostly UNVERIFIABLE output until you add a token inventory source. Teams without a nightly job budget should skip the scheduled sweep and keep the pre-commit hook alone; a gate nobody watches is worse than no gate.

Finally, do not use this to launder accountability. The script decides what is checkable, not who is responsible for a promise. Compatibility claims, deprecation timelines, and security guarantees still need a named human owner, and the fence markers are there to make that boundary reviewable rather than implicit.

If you want to try the drafting half of this loop, the MonkeyCode project page lists the current free model access and free server terms, and a ten-minute test is enough to see whether it fits your drift report.