Ban Naked Numbers: A Token Gate for AI-Drafted Docs

A docs PR landed in March with one changed sentence: the queue client's retry backoff was now "250 ms".

The code had moved to 800 ms two releases earlier. Reviewers read the sentence, found it grammatical, and approved it. Three weeks later a support thread quoted our own page back at us while debugging a timeout cascade.

Prose mistakes cost a reader a paragraph. Numeric mistakes cost them an afternoon.

Prose errors are cheap; numbers are not

Look at what a language model actually gets wrong in documentation. It rarely produces broken English. It produces plausible constants.

• A duration that was true last quarter.
• A version string one minor release behind.
• A URL that looks like the rest of your URLs.
• A rate limit quoted from a different service in the same monorepo.

You can measure your own exposure before trusting anything in this post, including my premise:

rg -o --no-heading -N '\b\d+(?:\.\d+)?\s?(?:ms|s|MB|GB|%|x)\b|\bv?\d+\.\d+(?:\.\d+)?\b' docs/ \
  | wc -l

If that number is in the hundreds, human review of prose is not a control. It is a hope.

Split the page: draftable prose vs owned claims

The working rule I use is narrow. A model may arrange language. A human must own every string a reader could copy into a terminal, a config file, or a capacity plan.

Element	Model may draft	Human owns	Gate
Transitions, ordering, examples of shape	yes	no	none
Durations, sizes, rates, percentages	key only	value + source	token check
Versions, image tags	key only	value + source	token check
URLs	key only	value + reachability	token check
Deprecation and security statements	first pass	wording and truth	review checklist
Commands that mutate state	no	yes, after execution	separate runbook test
Screenshots	no	yes	none

The important part is not the table. It is that the model is not asked to remember a number — it is asked to reference one.

The artifact: a fact registry plus claim tokens

1. Register the claim, with provenance

facts.json is the only file a human signs for numeric truth. Each entry carries where it came from and when someone last checked.

{
  "facts": {
    "queue.backoff_ms": {
      "value": 800,
      "unit": "ms",
      "source": "src/queue/retry.ts:41",
      "verified_at": "2026-10-24",
      "owner": "runtime-team"
    },
    "api.default_timeout_s": {
      "value": 30,
      "unit": "s",
      "source": "config/defaults.yaml:12",
      "verified_at": "2026-10-24",
      "owner": "platform"
    },
    "release.current": {
      "value": "2.7.0",
      "source": "package.json:3",
      "verified_at": "2026-10-24",
      "owner": "release"
    }
  }
}

2. Let the model draft only slots

The prompt contract is short: write Markdown, never write a number, version, or URL directly — emit a token instead. Unresolved needs become a request, not a guess.

The client retries with a backoff of {{fact:queue.backoff_ms}} by default.
Requests time out after {{fact:api.default_timeout_s}} unless overridden.

<!-- fact-request: queue.retry_jitter_ms — model could not find a source -->

That trailing comment is the useful failure. The model admitting it does not know is worth more than a confident 50 ms.

3. Fail the PR on a naked number

claimcheck.py is stdlib-only and runs in two stages. Stage one scans the raw draft; stage two scans the rendered page and confirms every surviving claim traces back to a registered value.

#!/usr/bin/env python3
"""claimcheck.py - only registered claims may appear in AI-drafted docs.

  python claimcheck.py docs/queue-retry.md --facts facts.json --stage draft
  python claimcheck.py docs/queue-retry.md --facts facts.json --stage rendered
"""
from __future__ import annotations

import json
import re
import sys
from pathlib import Path

FENCE = re.compile(r"^```

.*?^

```", re.S | re.M)
TOKEN = re.compile(r"\{\{fact:([a-z0-9_.-]+)\}\}")
CLAIM = re.compile(
    r"\b\d+(?:[.,]\d+)?\s?(?:ms|s|sec|secs|seconds?|min|minutes?|h|hours?"
    r"|MB|GB|KB|TB|%|x|rps|req/s|tokens?)\b"
    r"|\bv?\d+\.\d+(?:\.\d+)?\b"
    r"|https?://[^\s)\]>\"']+",
    re.IGNORECASE,
)


def load_facts(path: str) -> dict:
    return json.loads(Path(path).read_text())["facts"]


def as_text(fact: dict) -> str:
    unit = fact.get("unit")
    return f"{fact['value']} {unit}".strip() if unit else str(fact["value"])


def strip_examples(md: str) -> str:
    # Fenced blocks are verified by executing them, not by this gate.
    return FENCE.sub("", md)


def check_draft(md: str, facts: dict) -> list[str]:
    problems = []
    for key in sorted(set(TOKEN.findall(md))):
        if key not in facts:
            problems.append(f"unknown fact key {key!r}")
    body = TOKEN.sub("", md)
    for m in CLAIM.finditer(body):
        line = body.count("\n", 0, m.start()) + 1
        problems.append(f"draft line {line}: naked claim {m.group(0)!r}")
    return problems


def render(md: str, facts: dict) -> str:
    return TOKEN.sub(lambda m: as_text(facts[m.group(1)]), md)


def check_rendered(rendered: str, facts: dict) -> list[str]:
    allowed = [as_text(f).replace(" ", "") for f in facts.values()]
    problems = []
    for m in CLAIM.finditer(rendered):
        hit = m.group(0).replace(" ", "")
        if not any(hit.startswith(a) for a in allowed):
            problems.append(f"rendered claim {m.group(0)!r} matches no fact")
    return problems


def main(argv: list[str]) -> int:
    try:
        md_path, facts_path, stage = argv[1], argv[3], argv[5]
        facts = load_facts(facts_path)
        md = strip_examples(Path(md_path).read_text())
        if stage == "draft":
            problems = check_draft(md, facts)
        else:
            problems = check_rendered(render(md, facts), facts)
    except (IndexError, KeyError, OSError, json.JSONDecodeError) as exc:
        print(f"input error: {exc}", file=sys.stderr)
        return 2
    for p in problems:
        print(p)
    print(f"{stage}: {len(problems)} problem(s), {len(facts)} fact(s)")
    return 1 if problems else 0


if __name__ == "__main__":
    sys.exit(main(sys.argv))

The first run on the draft that started this post looks like this (illustrative output from a synthetic three-page docs set, not a benchmark):

$ python claimcheck.py docs/queue-retry.md --facts facts.json --stage draft
draft line 14: naked claim '250 ms'
draft line 21: naked claim 'https://internal.example/runbooks/queue'
draft: 2 problem(s), 6 fact(s)
$ echo $?
1

Two lines back to the model, one of them a token, and the stale number is gone — not corrected, removed from the model's authority.

4. Show the reviewer the provenance

The gate blocks bad pages. It does not tell a reviewer whether 800 ms is still 800 ms. Print that before approval:

python - <<'PY'
import json, re, pathlib
facts = json.loads(pathlib.Path("facts.json").read_text())["facts"]
md = pathlib.Path("docs/queue-retry.md").read_text()
for key in sorted(set(re.findall(r"\{\{fact:([a-z0-9_.-]+)\}\}", md))):
    f = facts[key]
    print(f"{key:24} {str(f['value']):>8}  {f['source']:<28} {f['verified_at']}  {f['owner']}")
PY

api.default_timeout_s           30  config/defaults.yaml:12       2026-10-24  platform
queue.backoff_ms               800  src/queue/retry.ts:41         2026-10-24  runtime-team
release.current              2.7.0  package.json:3                2026-10-24  release

Now the approval question is answerable in one line: "did the runtime team sign a change to backoff_ms?" If not, the page is wrong or the fact file is stale, and either way it is not a merge.

Failure analysis: what the gate catches, and what it never will

Three honest limits.

1. A registered value can be wrong. The gate proves provenance, not truth. If someone writes 800 with a source pointer that says 500, you have a process problem no regex will fix.
2. Prose can be false without containing a number. "The client retries until the queue drains" carries no digits and can still be a lie. That is a review checklist item, not this tool's job.
3. Omission is invisible. A stale caveat that was deleted reads as a clean page. Diff-level review, not claim scanning, is what catches removal.

This is why the split matters more than the script. The script only enforces a boundary a human already agreed to.

Where free model access and a free server actually fit

The draft stage above is a bounded generation job: fixed template, prose only, tokens required, fact-request list as output. That shape is exactly what you can afford to run on every docs PR rather than on a quarterly cleanup. MonkeyCode's free model access makes that economics trivial for a side project or an unfunded OSS repo. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A weak model is not a blocker here, because its worst habit — inventing constants — is precisely what stage one rejects. What you get instead is a list of unknown fact key failures, each one a question for the owning team, and a page with fewer confident sentences.

The free server option matters for the scheduled half of the workflow. A nightly job drafts one page from a template, runs --stage draft, and opens a PR only if the problem count is zero. That job needs a stable-ish place to run more than it needs CPU. Two caveats before you build that: treat free capacity as a convenience, not an SLA, and never point it at a private repo whose contents you cannot export. Check the current limits on the project's own pages rather than trusting any number in an article — including this one.

Who should not use this

• Docs generated from config. If the numbers already live in defaults.yaml, render the table from config. Do not gate prose that should not be prose.
• Teams without a named fact owner. The gate will block PRs and nobody will be empowered to unblock them. Fix ownership first.
• Repos one person fully knows. For five pages with a single reviewer, the rg count and a careful read is cheaper than a registry.
• Legal, pricing, and marketing copy. Different failure modes, different review.

The one-sentence version

If a reader can copy it into a terminal or paste it into an incident doc, a human signs it — the model gets to write the sentence around it.

If you want to try the draft stage without a budget conversation, MonkeyCode's free model access and free server option are a reasonable place to start; the fact registry, the two-stage gate, and the reviewer checklist stay yours either way.

Comments

[Kent Bodrov]

One mutation test I'd add: swap two valid fact keys. A sentence about queue backoff could reference api.default_timeout_s; the rendered value is registered, but attached to the wrong claim. Would the gate catch that? Binding each claim or template slot to an allowed fact key would cover a failure that checking registry membership alone misses.