The Model Drafts Prose; You Own the Numbers: Tracing Numeric Claims in Generated Docs

A documentation draft is only safe to generate when every sentence in it has a named owner, and that owner is decided by sentence type rather than by how fluent the prose reads. Models restate what a repository already says quite well, and they are unreliable at deciding why it says it, so the useful split follows that asymmetry. This post turns the split into one small checker that fails a build when a drafted number has no line of evidence behind it, plus a seven-step workflow for keeping the human half honest.

The scope is deliberately narrow: numbers, file paths, and command names are machine-checkable, while intent, trade-offs, and deprecation reasoning are not. Narrow scope is what makes the check enforceable on every pull request instead of once a quarter.

Split draft rights by falsifiability, not by section

Classify each block of a page before any model sees it, then let the class decide who may write the first draft. The table below is the version we can defend; treat the middle column as policy rather than as a claim about any specific model.

Block type	Example	First draft	Who signs off
Restatement of code	make dev starts the worker	model, given the excerpt	CI, because the command must exist
Defaults and limits	30 second visibility timeout	model, given file:line	the checker script
Rationale and trade-offs	why the timeout was raised	human only	named maintainer
Migration and deprecation	old flag removed in a release	human only	maintainer plus release owner
Security and data handling	what is logged, what is retained	human only, never drafted	security reviewer

Two rules fall out of the table. A drafted sentence that contains a number must carry a pointer to the line that produces the number, and a sentence that explains a decision must carry a person. Everything else is prose style and can be reviewed as prose.

The artifact: a checker that refuses orphaned numbers

The script below is short on purpose. It parses a facts block from a Markdown file, re-reads the repository at the current commit, and fails if a quoted number is absent from the cited window. It is a working sketch rather than a polished tool, so read the regexes before trusting them with unusual number formats.

#!/usr/bin/env python3
# tools/doc_facts.py
"""Fail a docs build when a drafted number has no line of evidence."""

from __future__ import annotations

import datetime as dt
import re
import subprocess
import sys
from pathlib import Path

FACTS = re.compile(r"<!--\s*facts\s*(.*?)-->", re.S)
NUM = re.compile(r"(?<![\w.])\d+(?:\.\d+)*")


def parse_block(block: str) -> list[dict]:
    facts, cur = [], None
    for raw in block.splitlines():
        line = raw.strip()
        if not line or line.startswith("#"):
            continue
        if line.startswith("- "):
            cur = {}
            facts.append(cur)
            line = line[2:]
        if cur is None:
            continue
        key, _, value = line.partition(":")
        cur[key.strip()] = value.strip().strip('"').strip("'")
    return facts


def numbers(text: str) -> set[str]:
    return set(NUM.findall(text))


def window(repo: Path, rel: str, lineno: int, pad: int = 2) -> str:
    lines = (repo / rel).read_text(encoding="utf-8").splitlines()
    lo = max(0, lineno - 1 - pad)
    hi = min(len(lines), lineno + pad)
    return "\n".join(lines[lo:hi])


def check(fact: dict, repo: Path, today: dt.date) -> list[str]:
    kind, text = fact.get("kind", ""), fact.get("text", "")
    fid = fact.get("id", "?")
    if kind == "code":
        rel, _, line = fact.get("source", "").partition(":")
        if not rel or not line.isdigit():
            return [f"{fid}: code fact needs source=path:line"]
        try:
            snippet = window(repo, rel, int(line))
        except OSError as exc:
            return [f"{fid}: cannot read {rel}: {exc}"]
        return [
            f"{fid}: {n!r} not in {rel}:{line} (+/-2 lines)"
            for n in numbers(text)
            if n not in snippet
        ]
    if kind == "external":
        errs = []
        if not fact.get("url", "").startswith("http"):
            errs.append(f"{fid}: external fact needs a url")
        try:
            age = (today - dt.date.fromisoformat(fact.get("retrieved", ""))).days
        except ValueError:
            return [f"{fid}: retrieved must be YYYY-MM-DD"]
        if age > 180:
            errs.append(f"{fid}: retrieved {age} days ago, refresh it")
        return errs
    if kind == "human":
        return [] if fact.get("owner") else [f"{fid}: human fact needs an owner"]
    return [f"{fid}: kind must be code, external, or human"]


def main(paths: list[str]) -> int:
    repo, fails = Path.cwd(), 0
    head = subprocess.run(
        ["git", "rev-parse", "--short", "HEAD"],
        capture_output=True, text=True, check=True,
    ).stdout.strip()
    today = dt.date.today()
    for p in paths:
        doc = Path(p).read_text(encoding="utf-8")
        block = FACTS.search(doc)
        if not block:
            stray = numbers(doc)
            if stray:
                print(f"{p}: numbers {sorted(stray)} without a facts block")
                fails += 1
            continue
        facts = parse_block(block.group(1))
        covered = set().union(*(numbers(f.get("text", "")) for f in facts))
        for fact in facts:
            for err in check(fact, repo, today):
                print(f"{p}: {err}")
                fails += 1
        for stray in sorted(numbers(FACTS.sub("", doc)) - covered):
            print(f"{p}: prose number {stray!r} has no fact entry")
            fails += 1
        print(f"{p}: {len(facts)} facts checked at {head}")
    return 1 if fails else 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

The draft it is designed to police looks like this, with a prose sentence and a verification block kept in the same file so review cannot separate them.

# Queue worker tuning

The worker default is a 30 second visibility timeout, and it retries 3 times before dead-lettering.

<!-- facts
- id: f1
  kind: code
  text: "visibility timeout is 30 seconds"
  source: src/queue/config.py:14
- id: f2
  kind: code
  text: "retries 3 times"
  source: src/queue/config.py:15
- id: f3
  kind: human
  text: "timeout raised for slow batch jobs"
  owner: platform-oncall
-->

Seven steps from excerpt bundle to merged page

1. Freeze the source commit in the pull request body, because every pointer below is relative to that commit and a rebase can move line 14.
2. Build an excerpt bundle so the model sees the code it is allowed to describe. git grep -n 'TIMEOUT\|RETRY' -- 'src/queue/*' > /tmp/excerpts.txt gives you a bundle you can paste or attach.
3. Ask for two outputs in one response: the Markdown prose and the facts block. State explicitly that numbers absent from the excerpts must not appear in the prose.
4. Pre-filter before review. grep -oE '[0-9]+(\.[0-9]+)*' /tmp/excerpts.txt | sort -u gives the allowed set, and any drafted number outside it is a rejected draft rather than a slow review comment.
5. Run python tools/doc_facts.py docs/queue-tuning.md locally and fix the facts block first, not the prose, because the prose is usually right and the pointer is usually wrong.
6. Route the human facts to their named owners in review, so the trade-off paragraph has an author instead of an approver.
7. Paste the commit hash and the checker output line into the pull request, which tells the next reader exactly what the numbers were verified against.

In CI the same command runs on changed files only, which keeps the job under a few seconds on modest repositories.

name: docs
on: [pull_request]
jobs:
  facts:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: python tools/doc_facts.py $(git diff --name-only origin/main...HEAD -- 'docs/**/*.md')

Where free drafting access actually helps

Step 3 is the only step that needs a model, and it is also the step you want to rerun cheaply, because the first draft usually fails the pre-filter and gets regenerated two or three times. MonkeyCode provides free model access and a free server option, which is enough to move that regeneration loop off a paid API key and off your laptop; the repository-reading work in steps 4 and 5 stays where the checkout is. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The limitation is structural rather than promotional: verification requires the repository at a pinned commit, so it belongs in local tooling or CI, and no remote drafting environment can substitute for it. Drafting remotely and verifying locally is the split that keeps the two halves honest.

Failure modes to expect

• Line drift after a refactor: the pointer resolves and the number moves, so rerun the checker on every commit instead of once at authoring time.
• Formatted numbers: 1,000 and 1_000 will not match a source line that writes 1000, so normalise separators before comparing.
• Semantic gaps: a sentence can quote the right number and draw the wrong conclusion, and no checker will catch that. That is why the rationale row in the table stays human-owned.
• Recency theatre: an external fact with a fresh retrieval date is not a verified fact, only a dated one.

Who should skip this

Teams whose documentation is mostly narrative, marketing, or tutorial prose will get little from a number checker and should invest in a vocabulary lint instead. Single-maintainer repositories without CI will find the human owner field is always the same name, which makes the ownership column decorative. Projects whose pages carry no numbers at all should not add a facts block just to satisfy a tool.

If you maintain pages that quote defaults, limits, or version numbers, start with the table and the script above on one page this week, and see how many drafted sentences survive a commit-pinned check before you extend it to the rest of the docs tree.