DEV Community

Jasveer Singh
Jasveer Singh

Posted on

My spec said POST. The code said GET.

The spec for a small test project of mine says the items endpoint supports POST /items — create an item. The code registers the path with GET only. Nothing crashed. No test failed. The spec just quietly became fiction, while remaining the document anyone new — human or agent — would trust first.

That gap has a name (drift) and, I eventually decided, it deserved a receipt. So I built spec-receipt: a deterministic, stdlib-only checker that extracts what a Markdown spec declares, extracts what the code declares, and prints a verdict per item, with file-and-line evidence on both sides.

pipx install spec-receipt
spec-receipt check path/to/repo
Enter fullscreen mode Exit fullscreen mode

What it checks

Five surface classes, in both directions:

  • CLI flags (with stated defaults and choice sets) — from argparse and click, via AST
  • CLI commands — subparsers and click groups
  • HTTP routes — Flask, FastAPI, Django URL declarations; an endpoint's identity is its (method, path) pair, which is how POST /items gets caught
  • Config keys — example configs and config[...] / config.get(...) reads, presence only
  • Environment variables — constant-name reads, including negations ("never reads AWS_*")

Spec items come back IMPLEMENTED, MISSING, CONTRADICTED or — for prose requirements, which get traced to anchors in code/tests rather than judged — TRACED / UNTRACED. Code surfaces the spec never mentions come back UNDOCUMENTED.

The verdicts are stingy on purpose. MISSING is issued only when a surface class was completely enumerated and the item is absent from it. CONTRADICTED needs hard evidence on both sides. A prose requirement is never declared absent: natural language can't prove a behaviour absent, so it's UNTRACED, and the receipt says why.

Then I ran it on real repos, and it lied to me

Before releasing, I dogfooded it on three real spec'd projects — its own design document, a sibling tool (skill-claims), and a public spec-kit-layout project — hand-checking every hard verdict.

Coverage passed. Precision didn't. All three CONTRADICTED verdicts were false.

The tool hadn't misjudged any real declaration. It had read the examples inside my design document — the section that illustrates what a spec item looks like, an illustrative --mode with a stated default, a quoted "no --legacy" — as if they were declarations, and formed verdicts against pairs that should never have existed. It also traced one feature's FR-001 to another feature's tests, and two requirement IDs to a comment block that explicitly lists them as having no anchor. A drift checker that cries wolf is worse than no checker, so v1.0.0 waited.

The fix was one narrow class — extraction scoping, not the verdict engine:

  • Sections and lines that present examples or define the notation are illustrative, never declarative
  • Template placeholders (/path, SCREAMING_CASE) name the notation, not a surface
  • Quoted text is the document talking about wording, not wording that declares
  • Sample output in a spec is not configuration
  • Requirement anchors are scoped to the ID's own spec, and a comment that denies anchoring is never an anchor

The rerun, same repos, same commands: zero false CONTRADICTED anywhere, every trace correctly scoped or honestly UNTRACED, mean coverage 92.6%. Two requirements are now honestly UNTRACED where the first run had traced them to that denial comment — lower numbers, correct receipts. Every failure shape became a regression test; the suite is 87 tests.

Honest limits

Static evidence only — the tool executes nothing, so runtime-built surfaces can hide. Config values are never compared, only presence. Requirements are traced, never semantically judged. Python implementations only. Each disclosed trade-off of the scoping rules (a genuine declaration sharing a line with "e.g." is missed, for instance) is printed in the README and in every receipt's Limits section.

It's MIT, has zero dependencies, and prints no score — a receipt, not a rating. The two fixtures in the repo show both directions verbatim: one repo in exact agreement with its spec, one drifted in six planted ways.

Watch it run (27 seconds): the demo video runs the checker on the repo's own drifted fixture — 17 spec items receipted: 11 IMPLEMENTED, 1 MISSING, 3 CONTRADICTED (the --mode default, the --legacy negation, and POST /items), closing on Verdict: DRIFT. Watch the demo

Repo: https://github.com/jasveerai88/spec-receipt

Feedback I'd most value: spec shapes the extractor misses, and verdicts you'd judge differently. Both are exactly what the dogfood round improved — once I'd stopped believing my own tool's first output.

Top comments (0)