DEV Community

Cover image for Spec-driven development has a verification hole
Emeka Adirije
Emeka Adirije

Posted on

Spec-driven development has a verification hole

In part one I argued that markdown won the specification layer without anyone holding a vote, and that we carried version control and code review across to it and left the rest of our discipline behind. This is the specific thing that absence lets through.

Spec-driven development is mostly right, and I say that as someone who has reorganised how my team works around it.

The argument is sound. When generating an implementation becomes cheap, the quality of the specification becomes the constraint. Sean Grove's framing in The New Code is the clearest statement of it: prompts are transient, while a versioned spec aligns humans and models over time. GitHub's Spec Kit, AWS Kiro and Claude Code have all built real tooling on that premise. At the radical end of the movement, humans only ever edit the spec and treat the code as a compiled artifact.

I think that direction is correct. I also think the formats the movement has standardised on, including the ones I use, are all missing the same section.

The thing that keeps going wrong downstream

The delivery numbers are strange in a specific way.

Sonar's analysis of LinearB's 2026 Engineering Benchmarks reports that AI-assisted pull requests merge at roughly half the rate of human-authored ones. They also run about 2.5 times larger and wait roughly five times longer for a reviewer.

Happy Bhati's paper on the agentic software development lifecycle puts the same thing structurally. Agents change the bottleneck in software delivery, and "review, integration, testing, security, deployment, and production operations remain constraining stages." Field-study gains, the paper says, "attenuate sharply between writing code and shipping reliable software."

So code gets produced faster and gets accepted at half the rate. The failure is happening between produced and accepted, which is the part of the pipeline the spec movement has spent the least time on.

Specifications point one way

A spec describes a destination. That is its whole job, and it is good at it.

Accepting a result requires something else. You have to compare what you have now against what you had before, and decide whether the difference is the one you asked for. That comparison needs an origin.

Almost no spec format carries the origin. The person who wrote the spec held it, in their head, at the moment they decided the work was worth doing. They knew what was broken, they had probably seen it happen, and then they wrote down what they wanted instead and threw the rest away.

Which means the author is the only person in the building who can tell whether the work is done.

That was survivable when a human engineer did the implementing, because a human reads a spec, notices the gaps, and silently fills them from context. They have seen the product. They know what the user was complaining about. They supply the missing half without anyone asking.

An agent reads it literally and builds exactly what the words describe. The missing half stays missing, and it surfaces at review, which is where the merge-rate numbers are coming from.

A spec is an assertion

I wrote recently about tests that have never been watched failing. A test written against code that already passes was true the moment it was written, by construction, and has never once distinguished a working system from a broken one. It raises coverage and verifies nothing.

A spec has the same shape one level up. It asserts that the product should behave a certain way. If nobody has observed the product failing to behave that way, and nobody wrote down what that failure looked like, then the assertion cannot be checked by anyone other than the person who made it.

Same disease, different artifact.

The missing section

This is not a claim about people writing lazy specs. It is structural, and you can check it yourself in about ten seconds.

Here is the specification template GitHub's Spec Kit generates, headings verbatim:

# Feature Specification: [FEATURE NAME]
  Feature Branch | Created | Status | Input

## User Scenarios & Testing (mandatory)
### User Story 1 - [Brief Title] (Priority: P1)
### User Story 2 - [Brief Title] (Priority: P2)
### User Story 3 - [Brief Title] (Priority: P3)
### Edge Cases

## Requirements (mandatory)
### Functional Requirements
### Key Entities

## Success Criteria (mandatory)
### Measurable Outcomes

## Assumptions
Enter fullscreen mode Exit fullscreen mode

Twelve headings, three of them marked mandatory, on a toolkit with around 141,000 stars.

Read down that list and look for the place where you write what is happening today. Success Criteria describes the destination. Edge Cases describes situations the new behaviour should survive. "Why this priority", inside each user story, argues for sequencing. Assumptions records what you are taking on faith.

Nowhere does the format ask what a person currently experiences, how you came to know it, or how somebody else could go and see it for themselves.

Kiro's requirements phase produces the same shape, an introduction followed by user stories and EARS-style acceptance criteria. Forward, all of it.

So when work specified this way comes back implemented, the only person who can say whether it worked is the person who was holding the before-state in their head on the day they wrote it. Everyone else is reading a description of a destination and taking the author's word for the journey.

What I would add

One section, placed before the user stories.

## Current behaviour (mandatory)

Problem
What a person experiences today, in a sentence or two.

Evidence
How you know. Tickets, a number, a recording, a complaint.

Steps to reproduce
What someone else does to see it for themselves.
Enter fullscreen mode Exit fullscreen mode

Filled in, that looks like this.

## Current behaviour

Problem
The weekly digest arrives with at most 20 items for every account,
regardless of how much activity there was that week.

Evidence
Four accounts raised it this month. The largest had 300 qualifying
events in the week and received a digest listing 20 of them, with no
indication that anything had been left out.

Steps to reproduce
1. An account with more than 20 qualifying events in the period
2. Trigger the weekly digest
3. Count the items in the delivered email. It is 20, and the job
   reported success.
Enter fullscreen mode Exit fullscreen mode

Three sentences and a numbered list. Notice that none of it tells the agent how to build anything, which is deliberate. It specifies the destination by specifying the gap.

Three things become true that were not true before.

A reviewer who has never spoken to the author can verify the work, because the reproduction steps are sitting right there and they either still produce 20 items or they do not.

The agent can check its own output before handing it over, against a stated observable rather than against its own reading of what it was asked to do.

And the acceptance criterion becomes a comparison between two numbers that already exist in the product, which is considerably harder to satisfy by accident than "pages through the full result set".

The whole thing, assembled

One added section, in a complete spec, in Spec Kit's own structure.

# Feature Specification: Complete Weekly Digest

Feature Branch: 014-complete-weekly-digest
Created: 2026-10-15
Status: Draft
Input: "The weekly digest is cutting off activity for our bigger accounts."

## Current behaviour (mandatory)

Problem
The weekly digest arrives with at most 20 items for every account,
regardless of how much activity there was that week. Accounts with
more activity than that receive a digest that silently omits the rest.

Evidence
Four accounts raised it this month. The largest had 300 qualifying
events in the week and received a digest listing 20, with nothing in
the email indicating anything had been left out. The job reported
success in all four cases.

Steps to reproduce
1. Use an account with more than 20 qualifying events in the period
2. Trigger the weekly digest for that period
3. Count the items in the delivered email. It is 20, while the activity
   view for the same period shows 300.

## User Scenarios & Testing (mandatory)

### User Story 1 - See the full week of activity (Priority: P1)

An account owner opens the weekly digest and finds every qualifying
event from that week, so they can review what happened without logging
in to check whether the email left anything out.

Why this priority: the digest exists to be a complete weekly record,
and an account owner who has to verify it elsewhere has no reason to
open it at all.

Independent Test: send a digest to an account holding more qualifying
events than the current cap, and confirm the delivered email lists all
of them.

Acceptance Scenarios:
1. Given an account with 300 qualifying events in the period, When the
   weekly digest is generated, Then the delivered email lists 300 items.
2. Given an account with 3 qualifying events in the period, When the
   weekly digest is generated, Then the delivered email lists 3 items.

### User Story 2 - Know when a digest has been shortened (Priority: P2)

An account owner receiving a digest that could not carry every event
sees how many were left out, so they know to go and look.

Why this priority: it only matters once the cap is raised, since at
that point omission becomes rare rather than routine.

Independent Test: force a digest past the message size limit and
confirm the email states the number omitted.

Acceptance Scenarios:
1. Given a digest whose full contents exceed the message size limit,
   When it is delivered, Then it states how many events were omitted.

### Edge Cases
- An account with zero qualifying events in the period
- An account whose event count changes while the digest is being built
- A digest whose full contents exceed the provider's message size limit

## Requirements (mandatory)

### Functional Requirements
- FR-001: The digest MUST include every qualifying event in the period
  for the account it addresses.
- FR-002: The digest MUST preserve chronological ordering across the
  entire set.
- FR-003: Where a size limit prevents including every event, the digest
  MUST state how many were omitted.

### Key Entities
- Digest: a periodic summary addressed to one account, covering a
  fixed window, containing an ordered list of qualifying events.
- Qualifying event: an account activity eligible for inclusion,
  carrying a timestamp and a type.

## Success Criteria (mandatory)

- SC-001: The item count in a delivered digest equals the item count
  shown in the activity view, for the same account and period, across
  100% of digests sampled over four weeks.
- SC-002: Digest-related support contacts fall from four a month to
  zero.
- SC-003: No delivered digest omits an event without saying so.

## Assumptions
- The activity view is the authority on what qualifies for a period.
- Message size limits are a provider constraint rather than a product
  decision, so FR-003 handles them rather than removing them.
Enter fullscreen mode Exit fullscreen mode

"But my spec is for a new feature"

This is the strongest objection to everything above, and it deserves a real answer rather than a dodge.

Nothing is broken when you are building something that does not exist. There is no bug report, no reproduction, no before.

There is, though, a current behaviour. Someone wants an outcome today and does something else to get it. They export to a spreadsheet and sort it by hand. They ask a colleague in chat. They open four tabs. They give up and the outcome never happens at all.

That is observable, reproducible, and writable in the same section:

Problem
There is no way to see which customers have lapsed. Account managers
currently export the full customer list and sort it in a spreadsheet
by last order date.

Evidence
Three of five account managers do this weekly. It takes about 40
minutes each and the result is stale by the time it is used.

Steps to reproduce
1. Open the customer list
2. Attempt to filter by last activity date
3. There is no such filter, and the list view offers no ordering by it
Enter fullscreen mode Exit fullscreen mode

If you cannot write that section for a feature, that is useful information. It usually means you have not yet established that anyone does the thing you are about to build a better way of doing.

Why adding more implementation detail makes it worse

Over-specification is already a named failure mode in spec-driven development, alongside spec-rot and slow starts. I want to be more specific about the mechanism, because "do not over-specify" is advice nobody can act on.

The detail people add under pressure is forward detail. More bullets about what to build, which classes to touch, which approach to take. Every one of those lines does two things. It removes the agent's freedom to find a route you had not considered, and it adds nothing you can check when the work comes back.

You are spending your specification budget on the half of the document that was never the problem. The same ten lines spent on what is currently happening, how to see it, and what number should change would make the result verifiable by someone who has never met you.

What this changes on a team

The format my team uses has seven fields. Problem, Evidence, Steps to reproduce, Expected result, Scope, Acceptance criteria, and Out of scope. Six of them are ordinary. The load-bearing one is the observable before-state, because a ticket with no reproducible symptom cannot be signed off by anybody.

Three things fell out of that once it was enforced.

Work became portable. A spec with a reproduction can be picked up by whoever is free, or by an agent, without a conversation to recover the context that was never written down.

Review got cheaper. The reviewer is not reconstructing intent from a diff. They are running three steps and reading a number.

And a fair number of proposed tickets died at the writing stage, because the person writing them could not produce evidence that the problem existed. That is the part I did not expect, and it has saved more engineering time than anything else on this list.

The claim

We are blaming the models for something we built into the process. A specification that was never checkable produces work that cannot be accepted, and the whole industry is discovering this at once because the humans who used to quietly supply the missing half have been taken out of that step.

Specs are the new code. Fine. Then they need the thing code has always needed, which is a stated expectation that something other than their author can evaluate.

Top comments (0)