Make your README numbers fail the build: a reproducible-figures gate in Rust

Every README with a benchmark number in it has the same bug. The number was true once.

I got tired of that on ferrumdeck, which is a Rust enforcement plane for AI agent tool calls, so the figures on its README are now regenerable by a make target that exits non-zero when a number drifts. Not a target that prints numbers for you to eyeball. One that fails.

This post is the pattern, the exact commands, and the one design decision that took the longest to get right.

Run it

git clone https://github.com/sattyamjjain/ferrumdeck
cd ferrumdeck
make reproduce-readme-figures-fast

That is about 10 seconds, and it checks the rate figures and both spend figures. The full version, which adds the latency table, takes about 70 seconds warm and roughly 2.5 minutes on a cold checkout because it builds --release first:

make reproduce-readme-figures

Everything is deterministic and offline. No services, no API keys, no model calls, no money moved. If you have a Rust toolchain, you have everything.

What it actually checks

Four things, and they are checked differently on purpose.

The rate figures. On an AgentDojo-style indirect-prompt-injection corpus (17 attack, 8 benign), the governed configuration blocks 17/17 with 8/8 benign-task utility retained. On a fixed spend-overrun trajectory, 4/4 unsafe actions blocked against 0/4 ungoverned.

The spend figures. The governed run of that trajectory costs 85 cents against 184 cents ungoverned, because stopping the runaway loop saves more than the governance overhead costs. And on the payment path, three unsafe AP2 mandates blocked an authorized spend from $150.95 to $0.40, each blocked on a distinct control.

The latency table. The deny-by-default allowlist check is 183 ns at p50 on an Apple M4 in --release, measuring the decision path only.

Whether any of them moved.

The design decision worth stealing

Rates are compared exactly. Latencies are compared within a band. That asymmetry is the whole thing.

The rate benchmarks are seeded, offline, and LLM-free. There is no clock and no sampling anywhere in that path. So a rate that moves at all is a behaviour change, and any drift fails. 17/17 becoming 16/17 is not noise; it is a regression, and treating it as noise is how a security guarantee quietly stops holding.

Latency is a measurement of a busy machine. Run-to-run variance on p95 reached 4.4× under load here. If you assert an exact nanosecond figure, you have built a test that fails on a Tuesday when your laptop is indexing, and a test that fails for reasons unrelated to the code gets muted within a week. A muted test is worse than no test, because it still renders green. So p95 gets a wider band than p50, and both bands are sized to catch an algorithmic regression rather than background noise.

The bit I would not skip if I were copying this: every row prints its measured/published ratio whether it passed or not. A figure drifting toward the edge of its band is visible three releases before it crosses. Pass/fail alone throws that signal away.

The machine problem, handled honestly

Absolute nanoseconds are a property of the silicon. The reference machine is stated in the README (Apple M4, --release) and that statement is load-bearing for the latency rows only.

On a different CPU, the check widens to an order-of-magnitude comparison rather than a strict one, and the output tells you which regime it applied. A nanosecond figure that hard-failed on someone else's laptop would be a broken gate, not a strict one, and the difference between those two matters more than the strictness does.

Wiring it up

Two pieces.

A Makefile target that runs the comparison and propagates the exit code, so it works identically in CI and on a laptop:

reproduce-readme-figures:
    cargo build --release
    cargo test --release -p fd-evals --test readme_figures -- --nocapture

And a nightly workflow that runs the full target and opens a tracking issue on drift rather than emailing a failure into the void:

- name: Reproduce README figures
  run: make reproduce-readme-figures
- name: Open drift issue
  if: failure()
  uses: actions/github-script@v7
  with:
    script: |
      github.rest.issues.create({
        owner: context.repo.owner,
        repo: context.repo.repo,
        title: `README figure drift on ${context.sha.slice(0,8)}`,
        body: 'A published figure moved. Run `make reproduce-readme-figures` locally for the ratio table.'
      })

An issue survives. A red run in a tab you closed does not.

Why bother

Because the alternative is that your README is a claim nobody can check, including you.

I found the version of this bug that started me down this path in a different repo of mine last week: a CI check compared a rule count in the docs against a generated badge, and the badge had stopped regenerating. Two stale numbers, in perfect agreement, forever. The check was green, and it was measuring nothing. Absence of a fresh measurement was silently equivalent to agreement.

So if you copy one thing, copy this: assert freshness next to equality. Any comparison against a cached, generated or fetched value needs both, and the freshness line is the one everybody leaves out.

Try to break it

make reproduce-readme-figures is the honest entry point. If a figure fails on your hardware, I want the issue, because either my band is wrong or my claim is.

Repo: github.com/sattyamjjain/ferrumdeck. Apache-2.0. Early and single-maintainer, and the README says so before it says anything else.