The spec for a small test project of mine says the items endpoint supports POST /items — create an item. The code registers the path with GET only. Nothing crashed. No test failed. The spec just quietly became fiction, while remaining the document anyone new — human or agent — would trust first.
That gap has a name (drift) and, I eventually decided, it deserved a receipt. So I built spec-receipt: a deterministic, stdlib-only checker that extracts what a Markdown spec declares, extracts what the code declares, and prints a verdict per item, with file-and-line evidence on both sides.
pipx install spec-receipt
spec-receipt check path/to/repo
What it checks
Five surface classes, in both directions:
-
CLI flags (with stated defaults and choice sets) — from
argparseandclick, via AST - CLI commands — subparsers and click groups
-
HTTP routes — Flask, FastAPI, Django URL declarations; an endpoint's identity is its (method, path) pair, which is how
POST /itemsgets caught -
Config keys — example configs and
config[...]/config.get(...)reads, presence only -
Environment variables — constant-name reads, including negations ("never reads
AWS_*")
Spec items come back IMPLEMENTED, MISSING, CONTRADICTED or — for prose requirements, which get traced to anchors in code/tests rather than judged — TRACED / UNTRACED. Code surfaces the spec never mentions come back UNDOCUMENTED.
The verdicts are stingy on purpose. MISSING is issued only when a surface class was completely enumerated and the item is absent from it. CONTRADICTED needs hard evidence on both sides. A prose requirement is never declared absent: natural language can't prove a behaviour absent, so it's UNTRACED, and the receipt says why.
Then I ran it on real repos, and it lied to me
Before releasing, I dogfooded it on three real spec'd projects — its own design document, a sibling tool (skill-claims), and a public spec-kit-layout project — hand-checking every hard verdict.
Coverage passed. Precision didn't. All three CONTRADICTED verdicts were false.
The tool hadn't misjudged any real declaration. It had read the examples inside my design document — the section that illustrates what a spec item looks like, an illustrative --mode with a stated default, a quoted "no --legacy" — as if they were declarations, and formed verdicts against pairs that should never have existed. It also traced one feature's FR-001 to another feature's tests, and two requirement IDs to a comment block that explicitly lists them as having no anchor. A drift checker that cries wolf is worse than no checker, so v1.0.0 waited.
The fix was one narrow class — extraction scoping, not the verdict engine:
- Sections and lines that present examples or define the notation are illustrative, never declarative
- Template placeholders (
/path,SCREAMING_CASE) name the notation, not a surface - Quoted text is the document talking about wording, not wording that declares
- Sample output in a spec is not configuration
- Requirement anchors are scoped to the ID's own spec, and a comment that denies anchoring is never an anchor
The rerun, same repos, same commands: zero false CONTRADICTED anywhere, every trace correctly scoped or honestly UNTRACED, mean coverage 92.6%. Two requirements are now honestly UNTRACED where the first run had traced them to that denial comment — lower numbers, correct receipts. Every failure shape became a regression test; the suite is 87 tests.
Honest limits
Static evidence only — the tool executes nothing, so runtime-built surfaces can hide. Config values are never compared, only presence. Requirements are traced, never semantically judged. Python implementations only. Each disclosed trade-off of the scoping rules (a genuine declaration sharing a line with "e.g." is missed, for instance) is printed in the README and in every receipt's Limits section.
It's MIT, has zero dependencies, and prints no score — a receipt, not a rating. The two fixtures in the repo show both directions verbatim: one repo in exact agreement with its spec, one drifted in six planted ways.
Watch it run (27 seconds): the demo video runs the checker on the repo's own drifted fixture — 17 spec items receipted: 11 IMPLEMENTED, 1 MISSING, 3 CONTRADICTED (the --mode default, the --legacy negation, and POST /items), closing on Verdict: DRIFT. Watch the demo
Repo: https://github.com/jasveerai88/spec-receipt
Feedback I'd most value: spec shapes the extractor misses, and verdicts you'd judge differently. Both are exactly what the dogfood round improved — once I'd stopped believing my own tool's first output.
Top comments (0)