A test that fails when your docs lie

Ilya Arbabi — Thu, 05 Nov 2026 00:05:27 +0000

Five days after I published the docs for my app, 36 of its 37 pages described code that had changed since.

Code has tests and words don't. My code changes many times a day, mostly through AI coding agents, and the commits passed their tests while the words drifted without anything going red.

It isn't only the docs. Three sets of words describe the app:

the public docs
the descriptions the in-app assistant reads to know what each screen does
the tool descriptions its MCP server gives to other people's AI agents

When any of them is wrong, someone acts on it. A customer looks for a button that moved. The assistant promises a feature that changed. Another AI calls a tool the way it worked last week.

Words declare what they describe

Each page, or each set of words, keeps two things in a small map: the files it describes, and when someone last read it against them.

{
  "billing/plans": {
    "covers": ["src/billing/**", "server/plans.ts", "!server/plans.test.ts"],
    "checked": "2026-11-04T21:37:39Z"
  }
}

Git already knows when those files changed. So the check is one question: did any covered file get a commit after checked?

import { execFileSync } from "node:child_process";

type Entry = { covers: string[]; checked: string };

// "src/billing/**" -> ":(glob)src/billing/**", "!x" -> ":(exclude,glob)x"
const pathspec = (p: string) =>
  p.startsWith("!") ? `:(exclude,glob)${p.slice(1)}` : `:(glob)${p}`;

function staleCommits(entry: Entry): string[] {
  // git silently reads a date it can't parse as some other date
  if (Number.isNaN(Date.parse(entry.checked))) throw new Error(`bad date: ${entry.checked}`);
  const out = execFileSync(
    "git",
    ["log", `--since=${entry.checked}`, "--format=%h %cs %s", "--", ...entry.covers.map(pathspec)],
    { encoding: "utf8" },
  );
  return out.split("\n").filter(Boolean);
}

--since filters on the committer date, so the report prints the committer date too (%cs). There is no model call in it, so the same history always gives the same report: for each page, the commits that touched what it covers since it was last read.

The gate

A report you can ignore gets ignored, so it runs as a test in the normal suite. If any entry is stale, the suite is red and the deploy waits.

To make it green, someone reads the page against the listed commits, fixes the words if they're wrong, and marks it:

bun scripts/docs-stale.ts --mark billing/plans

That sets checked to now, and the mark goes in the same commit as the fix. That commit is always later than the mark, so the check counts the commit that sets a page's checked as read.

Two gotchas I hit

A pre-commit run can't see what the commit will make stale. git log only sees commits, so the check passed before the commit and failed right after it. Now it also reads the uncommitted changes in the working tree, and the marks go in with the change.

A rebase dates your marks out. The check compares checked with commit dates, and a rebase or cherry-pick gives every commit it rewrites a new committer date. Twice, pages I had marked before a rebase came back stale after it. So: rebase first, mark last, push right away.

What it caught in one release

One release this week worked through eleven bug reports: approved email sequences that didn't actually send, several screens, and what the app says it can read. The gate flagged 24 docs pages, nine of the assistant's descriptions and 31 MCP tool descriptions.

Most of them still held, so reading and marking took a minute each. Some didn't. The docs still said each email in a sequence waits for approval. And one MCP tool told other people's AI that the app reads PDF guides, on a server that can't run the sandboxed reader those files need. No other check looks at those words, so both would have gone live.

What it doesn't do

It works by file, not by meaning. It says "go look", never "this sentence is wrong", and reading is still the work. A page that covers a busy file gets flagged often, which is the price of a check that is cheap and gives the same answer every time. For the pages that matter most, there is also an optional mode that asks a model which sentences no longer match the screens. It costs a few cents a run, so it runs when I ask, not on every commit.