Docs Rot Unevenly: Rank Sections by Drift Before You Regenerate Them

A docs PR touched 41 files last quarter. Two reviewers approved it in an afternoon. Three weeks later a support engineer pasted a config example from that same page into a customer ticket, and the flag in it had been renamed twice since.

That wasn't a hallucination. The model wrote a correct sentence about the code as it existed on the day it ran. The failure was scheduling: nobody regenerated that section after the rename, because nothing in the pipeline knew the section had become expensive to leave alone.

Most AI doc pipelines I've seen obsess over the wrong axis. They argue about whether the model may write, and skip the question of when a section deserves another cycle. This post is about the second question.

The unit is the section, not the file

A page about database migrations contains one paragraph that decays weekly and five that decay yearly. Regenerating the whole file to fix the paragraph is how you get churn-heavy diffs that reviewers stop reading.

So make the section addressable. A one-line HTML comment above the heading carries the scope and the blast radius:

## Rotating the ingest key

<!-- scope: services/ingest/**, infra/keys/** | risk: high -->

1. Drain the consumer group...

The marker is inert in rendered Markdown, diffable, and greppable. scope lists the globs whose changes make this text suspect. risk is a human judgment about what happens when the text is wrong, not about how complex the text is.

Score drift with git, not with vibes

The script below ranks every marked section by how much its scope has moved in a window. It shells out to git log, so there is no index to maintain and no service to keep alive.

#!/usr/bin/env python3
"""rank_doc_drift.py - rank doc sections by staleness risk."""
from __future__ import annotations

import argparse
import json
import re
import subprocess
from dataclasses import asdict, dataclass
from pathlib import Path

MARKER_RE = re.compile(
    r"<!--\s*scope:\s*(?P<globs>[^|>]+?)\s*(?:\|\s*risk:\s*(?P<risk>low|med|high))?\s*-->"
)
HEADING_RE = re.compile(r"^(?P<level>#{2,3})\s+(?P<title>.+?)\s*$")

RISK_WEIGHT = {"low": 1.0, "med": 3.0, "high": 8.0}


@dataclass
class Section:
    doc: str
    heading: str
    globs: list[str]
    risk: str = "low"
    commits: int = 0
    files: int = 0

    @property
    def score(self) -> float:
        # Starting weights. Tune them against your own incident history.
        return round(RISK_WEIGHT[self.risk] * (self.commits + 0.5 * self.files), 2)


def parse_doc(path: Path) -> list[Section]:
    sections: list[Section] = []
    current: Section | None = None

    for line in path.read_text(encoding="utf-8").splitlines():
        marker = MARKER_RE.search(line)
        if marker and current is not None:
            current.globs = [g.strip() for g in marker["globs"].split(",") if g.strip()]
            current.risk = marker["risk"] or "med"
            continue

        heading = HEADING_RE.match(line)
        if heading:
            current = Section(doc=str(path), heading=heading["title"], globs=[])
            sections.append(current)

    return [s for s in sections if s.globs]


def git(*args: str) -> str:
    return subprocess.run(
        ["git", *args], capture_output=True, text=True, check=True
    ).stdout


def measure(section: Section, since: str) -> None:
    hashes = git("log", f"--since={since}", "--format=%H", "--", *section.globs).split()
    section.commits = len(hashes)
    if not hashes:
        return
    names = git(
        "log", f"--since={since}", "--name-only", "--format=", "--", *section.globs
    )
    section.files = len({n for n in names.split() if n})


def main() -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("docs", nargs="+", type=Path)
    ap.add_argument("--since", default="90 days ago")
    ap.add_argument("--check", action="store_true")
    ap.add_argument("--max-high", type=int, default=0)
    ap.add_argument("--json", action="store_true")
    args = ap.parse_args()

    sections = [s for doc in args.docs for s in parse_doc(doc)]
    for s in sections:
        measure(s, args.since)
    sections.sort(key=lambda s: s.score, reverse=True)

    if args.json:
        print(json.dumps([{**asdict(s), "score": s.score} for s in sections], indent=2))
    else:
        for s in sections:
            print(f"{s.score:7.2f} {s.risk:4} {s.commits:3}c {s.files:3}f  {s.heading}  ({s.doc})")

    if args.check:
        hot = [s for s in sections if s.risk == "high" and s.commits > args.max_high]
        if hot:
            print(f"\n{len(hot)} high-risk section(s) drifted past budget:")
            for s in hot:
                print(f"  - {s.doc} :: {s.heading}")
            return 1
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Run it, then gate on it:

python3 rank_doc_drift.py docs/ --since="30 days ago"

# fail the build when a high-risk section moved at all
python3 rank_doc_drift.py docs/ --since="14 days ago" --check --max-high 0

Two properties matter here. The score is a ranking, not a measurement of truth, so do not print it as a quality metric. And 0.5 * files is deliberately weaker than commit count: forty commits to one file is a stronger staleness signal than one commit to forty files.

Three lanes, three levels of trust

The score decides which lane a section enters. The risk label decides what the model is allowed to produce inside that lane.

Risk	Model may draft	Human must own	Merge rule
low	Full prose, rewritten freely	A read-through	1 reviewer
med	Prose plus a patch against current text; every literal (path, flag, default) must appear in the diff	Each literal value, verified against source	2 reviewers, diff vs code
high	Headings and a list of questions only	All commands, numbers, warnings, ordering	Section owner signs off; replay test required

A high section is anything where a wrong sentence causes data loss, a security decision, or a billing surprise. Auth flows, deletion paths, migration cutovers, quota behavior. Those sections are exactly where a fluent paragraph is most dangerous, because it reads like it was verified.

The prompt contract that keeps the lane honest

Give the model less freedom than you think it needs. The draft request carries four things and nothing else:

1. The current section text, verbatim.
2. The git log --patch for the scope window, truncated to the hunks that touch exported symbols or config keys.
3. An instruction to output a unified diff against the section, not a rewritten file.
4. A constraint list: no identifiers absent from the patch, no new commands, no removed warnings, no changed defaults.

Point 3 is the load-bearing one. A patch is reviewable at a glance and cannot silently delete the paragraph a support engineer depends on. A full rewrite can.

Where the free tier actually fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The queue script decides what to regenerate; something still has to hold the checkout, run the loop, and produce drafts. That is a small, bursty, stateless job, which is a good shape for cheap infrastructure: the operator describes MonkeyCode as offering free model access and a free server option, including a stated ten-million-token allowance.

Treat that allowance as a budget, not a guarantee. Measure your own cycle cost before you plan a schedule — a section regenerate-and-diff cycle on a medium doc page plausibly lands in the low thousands of tokens, which means the arithmetic is about how often you sweep, not about whether you can. Verify the current terms rather than designing around a number you read in a post, including this one.

A nightly loop on the free server is ~30 lines:

set -euo pipefail
git clone --depth 200 "$DOCS_REMOTE" work && cd work
python3 rank_doc_drift.py docs/ --since="7 days ago" --json > queue.json
python3 drain_queue.py queue.json --lane-budget low=20,med=5

drain_queue.py is yours to write: take the top N per lane, call the model, write patches to a scratch branch, open one PR per lane. Keep the three lanes as three separate PRs so a high review never blocks a low typo fix.

A test plan you can run this week

The script is easy. Trusting it is the work.

1. Replay. Take last quarter's doc commits. For each regenerated section, check whether the pre-regeneration text was already wrong at that commit. The ratio you get is your baseline staleness rate — record it, don't estimate it.
2. False demand. Count sections the ranker flags that a human reviewer judges fine as-is. A high false-demand rate means your globs are too broad, not that the model is bad.
3. Time to detection. For each real doc error found, how many days between the code change and the fix? This is the number the whole workflow exists to shrink.
4. Literal accuracy. In med sections, sample the changed literals and verify against source. Report it as a count, not a percentage of "accuracy" in general.

Run all four before and after enabling the loop on the same repo. Anything else is a demo.

Where this breaks

• External systems. Git churn says nothing about a third-party API you document. Those sections need a dated source link and a manual review cadence, not a commit counter.
• Monorepos. Unscoped globs make every section look hot. Scope to the narrowest directory that actually owns the symbol.
• Docs as contract. If your docs carry legal or compliance weight, no model should produce the diff, and the reviewer is not optional.
• Marker discipline. A section without a marker is invisible to the ranker, and invisible sections are the ones that rot silently. Add a lint that fails on unmarked ## headings in governed directories.
• Risk inflation. If everything is high, the queue is useless. Cap high at some fraction of sections and make the label a conversation, not a reflex.

Who should skip this

If your docs live in a wiki with no git history, the ranker has no signal — start by exporting to Markdown in a repo. If the whole docs tree is under five pages, read it monthly and save yourself the pipeline. And if your team already struggles to get one reviewer on a docs PR, adding a second lane-specific reviewer will not help; fix the review capacity first.

The uncomfortable part of generated documentation isn't the writing. It's admitting that a paragraph can be accurate, approved, and still be wrong six weeks later because nobody scheduled it. Rank the sections. Then decide who signs.

If you run the ranker on your own tree, the interesting output isn't the score list — it's the risk labels you had to invent because none of the three fit. That's usually where the real documentation debt is hiding.